🔴 Which of my pages does AI visit most? Where does it send visitors?
AI learns about brands by reading websites. And AI sends visitors to webpages through citations in responses. Marketers have adapted their strategies (and expectations) but mostly, we do this blind. We’re missing key information. We haven’t answered these key questions:
- What pages does AI read the most? What pages does AI rarely read?
- What kinds of pages does AI send visitors to?
- How deep do AI crawlers go into websites?
The answers to these questions are in the reports from bot-tracking services like Cloudflare. Cloudflare tracks AI requests in the “Most crawled paths” report and visits in the “AI referral traffic” report. We looked at these reports across 74 accounts to discover what AI is doing with marketing websites.
After many hours of research and analysis, we’re ready to share the insights. Here is The AI Crawl vs. Traffic Report.
1. AIs crawl homepages 15x more than other types of pages
Here’s the first insight: AIs visit homepages a lot. Homepages get roughly 15 times more AI attention than other types of pages. Nothing else comes close.
If AI requests were evenly distributed, a page type that makes up 5% of a site’s total URLs would get roughly 5% of AI crawls. This is definitely not the case. Adjusting for the fact that the homepage is just one page competing against dozens or hundreds of others, the homepage requests are literally off the chart.
The homepage completely shatters the baseline. We carefully categorized the other URLs for comparison: service and product pages, articles, about pages, contact pages, pricing, and case studies. And even when combined, they get much less proportional attention from AI. AI crawlers don’t disproportionately favor any particular type of page. AIs disproportionately favor the homepage itself.
Marketers everywhere are asking “what kind of content does AI prefer?” But the data shows that ‘where to publish’ may be more important than ‘what to publish.’ If you really want AI to know something about your brand, put it on your homepage.
 |
Cyrus Shepard, Founder at Zyppy“Incredible. This data suggests that many publishers may be sleeping on the potential of their homepages. Makes sense, but you have to wonder how many brands really pay attention to what their homepage says about them, or if they are offloading this valuable information space to about, support, and article pages.” |
Takeaway for marketers
If you only fix one page, fix this one. To train AI to be a sales rep for your brand, confirm that your homepage includes all of the following:
- A short, sharp sentence that summarizes your core positioning
- Details about what you do, how you do it, whom you do it for and your key differentiators
- Answers to the most common sales questions
- Strong evidence that supports your specific claims (data, awards, testimonials, etc.)
- Add case studies to the homepage, but don’t just link to them. Add the most impactful statement from each case study to the homepage.
That chart above controlled for website size (number of URLs) because the dataset includes sites large and small. Next we’ll look at how website size correlates with AI attention.
2. Larger websites command more attention from AIs
The size of a site (total number of URLs) has a strong correlation with the amount of attention.
Bigger sites pull in more total AI attention. The relationship is almost perfectly proportional (0.86 correlation). AI requests grow proportionally to page count. There isn’t a drop-off or compounding advantage with scale.
As with traditional search, sites with more URLs generally attract more bot requests than sites with fewer URLs. More pages mean more entry points to match against more queries and prompts. More surface area means more requests.
But it’s not a perfect correlation. Some small sites in the dataset attract much more attention from AI than expected. Look closely at the dots on that chart. Notice the outliers. Sites with 50 pages may get as many requests at sites with 1000 pages.
This could be for many reasons. They may have better content on better optimized pages. They may have stronger brands that are more prominent in AI’s background training. Or they may just be better at marketing.
Takeaways for marketers
- Build out your service page sections. You should have a URL for each combination of service, target industry, geography, etc.
- Add pages that frame your brand within AI models: comparison pages, case studies, team pages, an awards page, service-specific FAQ pages and a dedicated pricing page if you don’t already have one.
- Keep your content program turned on. Even if it doesn’t drive traffic from search or from AI, it’s likely still feeding what AI models know (and say) about your brand’s expertise
Now let’s look closer at the types of pages, not just how many, and we’ll discover what kinds of pages AI sends traffic to.
3. Which types of pages drive the most AI referral traffic?
Next we look at which pages attract referral traffic from AI sources.
These are not just citations. These are actual visits. A citation is just a link in an AI response. Here we’re measuring actual clicks on those links. These are real humans visiting websites from AI sources. Most citations are “AI visibility” without traffic. And because Cloudflare sits upstream, the data is more accurate than Google Analytics.
When we categorized the pages into groups, we put all of the “decision-shaping” pages into one category called “service/product.” These pages directly promote the offerings of a brand. Other special page types (home, articles, about, case studies) all got their own categories.
As a ratio of crawls-to-referrals, a few page types outperform others by a mile.
Note: The dataset here is smaller. Only 27 of the websites have pro-tier Cloudflare accounts. But you can easily see this pattern on your own website. Look at your traffic from AI sources in GA4 and you’ll notice the homepage is your most visited from AI sources by a large margin.
While the home and service pages that attract the most visitors from AI sources, articles underperform homepages by nearly 20%. They get read by AI, but it doesn’t send them a lot of traffic. We call this the “Dark Library Effect.” Articles get summarized and may be cited, but not clicked.
This phenomenon is familiar to SEO-focused content marketers. AI systems are simply absorbing article content and answering users’ questions directly. It’s the same zero-click dynamic that’s reshaped traditional search for many years.
- For information-intent users, AI does a wonderful job of satisfying without sending them to websites. Got your answer? No need to click.
- For commercial-intent users, who are making an actual decision, AI may provide the short list, but the users still want to visit the website (especially the homepage and service pages) before making a final decision. So they click.
We see this pattern again in the data showing total visits from AI sources, removing the AI crawls per page data and just exposes median total referral visits. Service and product pages earn roughly three times more total AI-referral traffic per page than a typical article.
When we look at the rate at which AI sends visitors to various types of pages (totals divided by how many pages that exist in that category), the picture sharpens dramatically.
You have just one homepage, but many service pages. So in the rate-per-page-type data, we see the relative importance of page types for AI referrals.
Takeaway for marketers
- To drive AI-referral traffic that may convert into leads, audit your homepage and service/product pages before investing in more top-of-funnel content.
- To shape how AIs understand your brand’s expertise, publish highly relevant and differentiated content (reports like this are an example) but don’t expect much traffic from AI sources.
Content marketing is valuable even if there is no referral traffic. Zero search referrals doesn’t mean zero brand value. We shouldn’t apply search traffic goals to articles that were never going to win those clicks in the first place.
And remember the many and powerful benefits of content marketing beyond clicks from search and AI: sales support, word of mouth, email list growth, social engagement, thought leadership, etc.
4. Page depth and URL structure
Our Cloudflare “most crawled paths” report shows the URL structure of every page that the AIs visited, giving us an opportunity to look at how deep the AI crawlers go. We can answer a few more interesting questions:
- Do AI bots go deep into the site structure?
- Are pages in sub-sub-directories visited less often?
- How does page depth correlate with AI crawl intensity and AI referral traffic?
The data shows that AIs crawl pretty much everything. AI bots are perfectly willing to find a page three or four folders down. But AIs are much less likely to send visitors to the pages buried deep within a folder structure. And the drop-off is steep…
There’s a well-understood technical reason to expect this kind of pattern: crawl budget. All bots operate with finite resources per site and allocate them accordingly. The willingness to go deeper trails off. We checked to see if this was related to site size or types of pages. It’s not. The pattern holds for big and small websites, for service pages and articles.
A page three folders deep earns about a quarter of the AI traffic its footprint predicts. At four folders deep, that number is close to zero. The deeper you bury a page in a folder, the less likely AI is to send it visitors. AI systems will find your deep pages, but they don’t often recommend them.
This suggests an “architecture tax” for site structure decisions. But this is correlation, not causation. Moving a page to a shallower location won’t necessarily improve its AI traffic performance. We don’t recommend moving pages around just for this reason, but if you’re planning a new website or reorganizing your pages, this data may help guide your decisions…
Takeaways for marketers
- Flatten the architecture if possible. This is good UX and it may help drive AI traffic to key pages.
- Put key services pages within a click or two of the homepage.
- Redesign your website. Be honest. When it’s time, it’s time.
Got your own Cloudflare data? Run this diagnostic prompt…
If your site is on Cloudflare, you can run a lightweight version of this analysis yourself. Go to AI Crawl Control → Metrics → Paths and download the “Most Crawled Paths” CSV. If you have a paid account, download the Referrals report too. The screen will look something like this:
Then upload to your favorite AI along with the following prompt…
Cloudflare “AI Crawl Control” Report Analysis Prompt
I run a B2B site and want to know if AI bots are wasting attention on the wrong pages, or if my best pages are getting seen. Analyze the attached Cloudflare AI crawl data. Check the actual columns first — don’t assume a format.
1. Categorize: Group pages into homepage, service/product, blog/articles, case studies, contact/about, everything else. Show each category’s share of crawl requests, and referrals per page (not just raw totals — a category with 300 pages will look big even if each one is barely read).
2. Benchmark: Try fetching this report for context: https://www.orbitmedia.com/blog/ai-crawl-vs-traffic-report/ — it studied 74 B2B/services sites and found homepages get ~15x more attention than their page count predicts, service/product pages earn ~3-4x more referral traffic per page than articles, and roughly 1 in 5 pages get crawled but never sent anyone a visit. If you can’t fetch the link, use those numbers anyway and say so. Tell me how my site compares.
3. Flag data problems: duplicate homepage rows (www/non-www, port variants, secondary subdomains), non-content files inflating the count (CSS/JS/images, cache-busting assets, login pages), any single page eating an outsized share of requests.
4. If I gave you referral data too: which page types get crawled a lot but rarely convert to a visit? Flag anything like a case-studies or portfolio section that’s heavily crawled but almost never referred.
5. Give me a one-paragraph verdict: is my site structured to turn AI attention into leads, or is my best content buried where AI can’t easily point to it? Say plainly what’s solid versus a rough guess from limited data.
This won’t have the rigor of our 74-site study (no cross-site comparison, just one snapshot of your site) but it references this study. This will quickly show you the same kinds of problems and patterns on your own data. It’s a diagnostic and benchmark analysis.
Give it your brand styles and it will make a nice little report for you.
Train the AIs to recommend your brand
AI reads a huge percentage of the public internet. And it may learn about your brand from many sources. But your website is the corner of the internet that you control. Your site is your best (and most direct) hope of training an AI to recommend your brand.
AI bots are likely visiting your website today. And now that we have a clear picture of what those bots visit, you can look at those pages and see what you’re teaching them. We also learned where AI tends to send visitors, which shows the same pattern.
47% of all pages in this dataset generated zero referrals. That’s fine. There are actually two very different reasons an AI bot visits your site: to fill in background knowledge (training bots) and to answer someone’s question right now (search agent bots). Either way, we now know where that attention actually goes.
Have a conversation with your favorite AI about your brand. Share this article with it and ask it for insights. Few marketers have had a serious conversation with an AI about why and when it would (and wouldn’t) recommend their brand.
Now go fill those gaps. Train the AIs on your difference, your offers and the impact of your work. AI needs us marketers to keep improving the web, so it can efficiently connect our best-fit prospects with the value we bring to the world.
About the dataset
This analysis covers a wide range of marketing websites, covering professional services, B2B tech, industrial, nonprofit, education and even a few ecommerce websites. Every page was classified by type and by how many folders deep it sits, then measured against its own site’s baseline so no single large site could skew the results.
- 74 sites, ranging from 45 to 95,522 crawl requests each
- 560,695 AI crawl requests from all 74 websites
- 446,267 AI referrals from the 27 sites with Cloudflare Pro-tier referral tracking
- 21,212 individual paths
The analysis was tricky because URL structures vary. Websites have all kinds of pages in all kinds of places. Some sites have all of their blog posts in a common directory, some do not. We had to carefully review the sites to categorize pages accurately. Data was processed and analyzed using Claude Opus with a careful review of all major judgement calls.
PDFs were handled separately. Accurate classification of PDFs is tricky. Some are newsletters, some are product manuals, some are case studies. In the end, we just gave them their own category.
AI bots are weird. They repeatedly crawl pages and files that have no practical value at all. AI is perfectly happy to crawl a useless backend file (such as /login/ajax.php) thousands of times. There were so many of these zero-value requests that we had to carefully exclude them using a pre-processing data cleanup step.
Is your website blocking AI?
Is your website blocking AI bots? How many? Which ones? There’s an easy way to answer this question. Just plug your domain into our AI Bot Access Checker, and you’ll instantly know.
As a marketer, you probably don’t want to block bots from training on your content. AI discovery is a goal, after all. How can AI cite and recommend you if it can’t get in the front door?
