For two decades, much of the open web operated on an implicit bargain. Publishers allowed search engines to crawl and index their pages. Search results sent some users back. Those visits supported advertising, subscriptions, donations, affiliate sales, or commerce.

AI search changes the transaction. Instead of presenting a ranked list and asking the user to assemble an answer, a system can retrieve several pages, synthesize them, and respond inside the search interface. Citations may remain, but the answer itself can satisfy the query. The search company captures the interaction; the publisher may supply the underlying information without receiving a visit.

The missing click

Pew Research Center observed the shift in browsing data from 900 U.S. adults during March 2025. About 18% of their Google searches produced an AI summary. When a summary appeared, users clicked a traditional result in 8% of visits, compared with 15% when no summary appeared. A cited link inside the summary received a click in only 1% of visits, and sessions ended on 26% of summary pages versus 16% of conventional pages (Pew Research Center). Pew’s restrained conclusion was that users “very rarely clicked on the sources cited”.

The result does not mean every publisher loses equally. Quick factual pages are easier to substitute than original reporting, tools, communities, or shopping services that require a visit. Longer and question-form queries also triggered summaries more often in Pew’s data, making explanatory content particularly exposed.

Google presents a different aggregate picture. In August 2025, it said total organic click volume from Search to websites was relatively stable year over year and that the average quality of clicks had increased. It argued that AI Overviews generate new and more complex searches and that traffic is shifting among sites rather than simply disappearing (Google). The company did not publish enough underlying data in that post for outside researchers to reconcile its measure with publisher reports or Pew’s user-level findings.

Both claims can be true. More queries can keep total clicks stable while the click rate per query falls. Higher-value destination visits can coexist with severe losses for sites whose content is summarized most easily.

Crawling and referral are separating

Traditional search crawling was justified partly by the prospect of referral traffic. AI systems may crawl for model training, answer grounding, or search without returning comparable visits. Cloudflare reported that in June 2025 OpenAI generated roughly one referral for every 1,700 crawls it observed, while Anthropic’s ratio was about 73,000 to one (Cloudflare). Those figures reflect Cloudflare’s bot classifications and customer network, not the entire web, but they show why publishers increasingly distinguish AI crawlers from conventional search bots.

Technical and commercial responses followed. Cloudflare introduced a private-beta Pay Per Crawl mechanism in July 2025, allowing participating sites to allow, block, or charge verified AI crawlers per request (Cloudflare). Other publishers have negotiated direct licenses, tightened robots.txt rules, emphasized subscriptions and newsletters, or invested in products—databases, events, communities, interactive tools—that cannot be reproduced by a short answer.

AI Search Is Rewriting the Economics of the Open Web — supporting image
AI Search Is Rewriting the Economics of the Open Web — supporting image

These options are uneven. A large publisher can negotiate and enforce terms. A small specialist site may have valuable information but little bargaining power, limited bot-management infrastructure, and no sales team. If those sites close or stop producing work, AI search loses part of the knowledge base on which it depends.

The policy question is control, not nostalgia

The goal should not be to preserve ten blue links unchanged. Answer engines can reduce search costs, make complex questions easier to ask, and help users compare sources. The issue is whether the surrounding market preserves incentives to create reliable information.

The United Kingdom has moved from inquiry to a concrete rule. In June 2026, the Competition and Markets Authority imposed a publisher conduct requirement on Google. It requires effective controls over the use of search content in generative AI, clear explanations, engagement metrics, reasonable steps toward accurate attribution, and a clear route for users to access source content (UK Competition and Markets Authority). The requirement treats attribution and traffic data as market infrastructure, not interface decoration.

Durable policy should give publishers a meaningful choice about AI use without forcing them to disappear from ordinary search; make citations prominent and claim-specific; report impressions, citations, and outbound visits; and support licensing or collective bargaining where market power is unequal. Competition rules also matter because the same firm may control discovery, the answer interface, advertising, and access to audience data. U.S. search remedies in 2025 explicitly extended safeguards to generative-AI products and companies (U.S. Department of Justice).

The open web has never been static, but it does require replenishment. AI search will be sustainable only if the people and institutions that investigate, explain, test, and document the world can capture enough value to keep doing so. A better answer today is not worth an emptier web tomorrow.

Sources