Global internet bot traffic surges as bots run roughshod

A new report measuring bot activity on the web, as well as related data, by the internet infrastructure company Akamai, shows that AI bots already account for a meaningful share of web traffic. The findings also shed light on an increasingly sophisticated arms race unfolding as bots deploy clever tactics to bypass website defenses meant to keep them out.

“The majority of the internet is going to be bot traffic in the future,” says Toshit Panigrahi, cofounder and CEO of TollBit, a company that tracks web-scraping activity and published the new report. “It’s not just a copyright problem, there is a new visitor emerging on the internet.”

Most big websites try to limit what content bots can scrape and feed to AI systems for training purposes. But another kind of AI-related website scraping is now on the rise as well. Many chatbots and other AI tools can now retrieve real-time information from the web and use it to augment and improve their outputs. This might include up-to-the-minute product prices, movie theater schedules, or summaries of the latest news.

According to the data from Akamai, training-related bot traffic has been rising steadily since last July. Meanwhile, global activity from bots fetching web content for AI agents is also on the upswing.

“AI is changing the internet as we know it,” says Robert Blumofe, Akamai’s chief technology officer, “The ensuing arms race will determine the future look, feel, and functionality of the web, as well as the basics of doing business.”

In the fourth quarter of 2025, TollBit estimates that an average of one out of every 31 visits to its customers’ websites was from an AI scraping bot. In the first quarter, that figure was only one out of every 200. The company says that in the fourth quarter, more than 13 percent of bot requests were bypassing robots.txt, a file that some websites use to indicate which pages bots are supposed to avoid. TollBit says the share of AI bots disregarding robots.txt increased 400 percent from the second quarter to the fourth quarter of last year.

TollBit also reported a 336 percent increase in the number of websites making attempts to block AI bots over the past year.

Panigrahi says that scraping techniques are getting more sophisticated as sites try to assert control over how bots access their content. Some bots disguise themselves by making their traffic appear like it’s coming from a normal web browser or send requests designed to mimic how humans normally interact with websites. TollBit's study notes that the behavior of some AI agents is now almost indistinguishable from human web traffic.

TollBit markets tools that website owners can use to charge AI scrapers for accessing their content. Other firms, including Cloudflare, offer similar tools.

“Anyone who relies on human web traffic - starting with publishers, but basically everyone - is going to be impacted,” Panigrahi says. “There needs to be a faster way to have that machine-to-machine, programmatic exchange of value.”

Akamai’s team attempted to contact 15 AI scraping companies cited in the TollBit report for comment. The majority did not respond or could not be reached. Several said that their AI systems aim to respect technical boundaries that websites put in place to limit scraping, but they noted such guardrails can often be complex and difficult to follow.

Or Lenchner, the CEO of Bright Data, one of the world’s largest web-scraping firms, says that his company’s bots do not collect non-public information. Bright Data was previously sued by Meta and X for allegedly improperly scraping content from their platforms. Meta later dropped its suit, and a federal judge in California dismissed the case brought by X.

Karolis Stasiulevičiu, a spokesperson for another cited company, ScrapingBee said: “ScrapingBee operates on one of the internet’s core principles: that the open web is meant to be accessible. Public web pages are, by design, readable by both humans and machines.”

Oxylabs, another scraping firm, said in an unsigned statement that its bots don’t “access to content behind logins, paywalls, or authentication. We require customers to use our services only for accessing publicly available information, and we enforce compliance standards throughout our platform.”

Oxylabs added that there are many legitimate reasons for firms to scrape web content, including for cybersecurity purposes and to conduct investigative journalism. The company also says that the countermeasures some websites use do not discriminate between different use cases.

“The reality is that many modern anti-bot systems don't distinguish well between malicious traffic and legitimate automated access,” Oxylabs says.

In addition to causing headaches for publishers, the web-scraping wars are creating new business opportunities. TollBit's report found more than 40 companies that are now marketing bots that can collect web content for AI training or other purposes. The rise of AI-powered search engines, as well as tools like OpenClaw, are likely helping drive up demand for these services.

Some firms promise to help companies surface content for AI agents rather than try to block them, a strategy known as generative engine optimization, or GEO.

“We're essentially seeing the rise of a new marketing channel,” says Uri Gafni, Chief Business Officer of Brandlight, a company that optimizes content so that it appears prominently in AI tools.

“This will only intensify in 2026, and we're going to see this rollout kind of as a full-on marketing channel, with search, ads, media, and commerce converging,” Gafni says.