Key Points
- AI bots now represent a substantial share of web traffic, with a notable rise in the fourth quarter of 2025.
- More than 13% of bot requests bypass robots.txt, a 400% increase from earlier quarters.
- Websites are intensifying efforts to block AI bots, with a 336% rise in blocking attempts over the past year.
- Companies are offering tools to detect, block, or monetize AI bot access to online content.
- A new market is emerging around generative engine optimization, positioning AI bot traffic as a marketing channel.
Rising AI Bot Traffic
Recent measurements from an internet infrastructure firm reveal that AI‑driven bots now account for a notable portion of overall web visits. In the fourth quarter of 2025, an average of one out of every 31 visits to participating sites originated from an AI scraping bot, a sharp rise from the earlier first‑quarter figure of one out of every 200. The same data show a steady increase in traffic generated by bots that fetch real‑time information for AI agents, indicating a broader shift in how automated systems interact with online content.
Bots Bypassing Controls
Analysis indicates that more than 13 percent of bot requests ignored the robots.txt file, a standard used by websites to signal which pages should not be accessed by automated agents. The proportion of AI bots disregarding robots.txt climbed 400 percent from the second quarter to the fourth quarter of the previous year. These bots increasingly mimic normal browser behavior, making their traffic nearly indistinguishable from human visitors and complicating detection efforts.
Industry Response
Website owners are reacting with a surge in defensive measures. Over the past year, the number of sites attempting to block AI bots grew by 336 percent, according to the reporting firm. Tools are being deployed to identify and filter out bot traffic, while some providers are developing solutions that allow sites to charge automated scrapers for access to their content. Major security firms, including Cloudflare, are offering similar capabilities, aiming to protect the value of human‑generated traffic.
Publishers, who rely heavily on human visitors, are particularly concerned about the impact of automated traffic on advertising revenue and content integrity. The ongoing “arms race” between bot developers and site operators is shaping the future look, feel, and functionality of the web, as highlighted by a chief technology officer from the infrastructure firm.
Emerging Business Opportunities
The escalation of AI‑powered scraping has spawned a new market segment. More than 40 companies now market bots that collect web content for AI training and other purposes. Some firms focus on helping content appear prominently in AI tools, a practice referred to as generative engine optimization. Executives from a content‑optimization company describe this as the rise of a new marketing channel that blends search, advertising, media, and commerce.
While many scraping services assert they only target publicly available information, they acknowledge the difficulty of adhering to complex technical boundaries set by websites. The tension between legitimate automated access—such as for cybersecurity or investigative journalism—and malicious scraping continues to drive innovation on both sides of the equation.
Source: wired.com