OpenClaw and the rising tide of AI scraping bots
OpenClaw, the viral virtual assistant once called Moltbot and Clawdbot, has put a spotlight on a shift in how the internet functions. What started as a novelty is now a marker of a broader change: AI agents are increasingly acting on the web without direct human control. That shift is forcing publishers, platforms, and security teams to rethink traffic and value.
Recent industry reports and infrastructure data show a measurable uptick in bot-driven visits. Training-related scraping and real-time fetching by AI agents are both on the rise, pushing overall bot traffic higher across major sites. The tactics bots use are getting more sophisticated, and defenses are struggling to keep pace.
“The majority of the internet is going to be bot traffic in the future,” says Toshit Pangrahi, cofounder and CEO of TollBit. His company tracks scraping behavior and this projection comes from months of traffic analysis. If accurate, that changes how we imagine online audiences and who actually consumes content.
Sites try to use robots.txt and other signals to limit scraping, and some publishers have pursued legal avenues when they suspect training data was taken without permission. At the same time, AI tools are increasingly designed to pull fresh, real-time details—prices, schedules, headlines—to improve their answers. That makes blocking both harder and more consequential.
Data from major web infrastructure providers points to a clear trend: the share of scraping aimed at AI training has been rising, and the proportion of bots fetching live content has climbed too. TollBit reported a jump in bots ignoring robots.txt and in overall scraping attempts. The arms race between scrapers and site defenses is escalating fast.
“AI is changing the web as we know it,” Robert Blumofe, Akamai’s chief technology officer, tells WIRED. What follows is likely to reshape user experience, site architecture, and business models. The stakes are not just technical; they affect who gets credit and revenue for content online.
TollBit’s monitoring found that bot visits went from about one in 200 visits earlier in the year to about one in 50 by the fourth quarter of 2025 for some customers. The company also reported that more than 13 percent of bot requests were bypassing robots.txt in that quarter. Those kinds of numbers explain why site owners are scrambling.
Blocking attempts have surged: TollBit recorded a 336 percent increase in websites trying to stop AI scrapers over a year. Scrapers have adapted, mimicking human browsers and interaction patterns to slip past defenses. For defenders, telling the difference between a legitimate automation and a persistent scraper is now a major headache.
Some firms sell tools that let publishers charge scrapers for access, while others promote ways to cooperate with AI systems so content surfaces rather than gets blocked. This has spawned new services around optimization for AI tools, sometimes called generative engine optimization, or GEO. Companies see these moves as an emerging marketing channel tied to AI discovery.
Not all scraping is framed as malicious. “ScrapingBee operates on one of the internet’s core principles: that the open web is meant to be accessible. Public web pages are, by design, readable by both humans and machines.” And some providers insist their bots don’t “access to content behind logins, paywalls, or authentication. We require customers to use our services only for accessing publicly available information, and we enforce compliance standards throughout our platform.”
The conflict over scraping raises tough questions about value, access, and control on the web. As AI agents multiply, businesses will have to decide whether to block, monetize, or optimize for them. The near-term result will be more complexity and new vendor opportunities as the web adapts to a growing population of machine visitors.