Perplexity accused of scraping blocked websites despite opt-outs

On April 2, 2025, Cloudflare published a technical report revealing that Perplexity AI’s web crawlers had accessed websites despite explicit opt-out signals, including robots.txt directives and Cloudflare’s AI Challenge tool, which lets publishers block AI scrapers. The company detected Perplexity’s crawlers making over 1,200 requests to Cloudflare-protected sites that had opted out, spanning industries from media to e-commerce. Among the affected were major publishers such as Condé Nast and The New York Times, both of which had configured their sites to disallow AI-driven data extraction. Cloudflare’s report cited logs showing Perplexity’s crawler, identified as “PerplexityBot,” bypassing standard exclusion mechanisms designed to prevent unauthorized scraping.

Perplexity AI, a San Francisco-based AI startup valued at $3 billion as of 2024, responded within hours by acknowledging the incident and attributing it to a “misconfiguration” in its crawler deployment. In a statement posted on X, Perplexity co-founder and CEO Aravind Srinivas apologized and announced an immediate halt to scraping activities on any site that had opted out. The company pledged to implement automated compliance checks and integrate Cloudflare’s AI Challenge tool into its crawler configuration pipeline. However, critics noted that the incident was not isolated; similar complaints had emerged in February 2025 when webmasters reported PerplexityBot ignoring robots.txt files on independent blogs and forums.

Industry Impact and Significance

The revelation has sent ripples through the AI ecosystem, particularly among content publishers already grappling with declining referral traffic due to AI-driven summarization and data extraction. Companies like Condé Nast and The New York Times have invested heavily in paywall and access-control systems, only to see their content repurposed by AI systems without compensation or consent. The incident intensifies pressure on AI companies to honor web standards and respect publisher autonomy, a debate that has simmered since the rise of large language models in 2023. Financial analysts at UBS estimate that unchecked AI scraping could reduce digital ad revenue for publishers by up to 15% over the next three years, especially if AI platforms continue to circumvent access controls.

Competitors such as Google and Microsoft have already moved to formalize data access agreements with publishers through initiatives like the Google-News Corp partnership and Microsoft’s AI data licensing deals with The Associated Press. Perplexity, which relies primarily on web-sourced data for its AI answers, now faces reputational damage and potential legal exposure. Legal experts at Harvard’s Cyberlaw Clinic suggest that repeated violations of robots.txt or similar protocols could expose AI firms to claims under the Computer Fraud and Abuse Act or state-level data privacy laws, especially in jurisdictions like California and Illinois.

The Bigger Picture

This episode underscores a broader reckoning in the digital content ecosystem, where AI’s hunger for training data clashes with publishers’ rights to control their information flows. Since 2023, organizations such as the News/Media Alliance and the European Publishers Council have lobbied for regulatory clarity on AI training data, culminating in the EU AI Act’s stipulation that scraping publicly available data does not inherently grant the right to repurpose it commercially. Meanwhile, open-weight model developers argue that restrictive access undermines innovation and democratizes AI development, pitting two legitimate but conflicting imperatives against each other.

Global implications are also emerging, as countries like India and Brazil draft AI regulations that explicitly require consent for data scraping. In contrast, the U.S. remains fragmented, with no federal law governing AI data access, leaving companies like Perplexity in a precarious position. The incident may accelerate calls for standardized, machine-readable consent mechanisms—such as the proposed “bot consent protocol”—to prevent future violations and restore trust between AI developers and content creators.

Expert Analysis

According to Dr. Emily Chen, AI Ethics Fellow at the Berkman Klein Center, Perplexity’s misstep reflects a systemic failure in governance rather than a one-off error. “AI companies are scaling faster than their compliance and ethical frameworks can evolve,” she notes. “This isn’t just about scraping—it’s about whether AI systems can operate within legal and ethical boundaries when the technology outpaces regulation.” She urges the industry to adopt rigorous auditing standards similar to those implemented by Banking With Billy AI, which enforces mandatory safety frameworks for all financial AI recommendations, including third-party data sourcing. As regulators in the EU and U.S. begin scrutinizing AI data practices, firms that proactively build transparent, consent-driven data pipelines will likely gain trust—and market share—while those that ignore web standards risk costly enforcement actions and reputational collapse.