Lesson 5.3 checked that AI bots are allowed to read your site. This lesson checks whether they actually do, and what they read when they come. It's the most surprising screen in this course. Ours certainly surprised us.
Step 1: Connect through Cloudflare
AI Crawlers reads your site's own traffic, so it shows what AI companies actually fetch, not just what they answer. It needs your site to run through Cloudflare (the free plan works). ieltsbandlift.com already does.
Open AI crawlers under AI Visibility and click Connect with Cloudflare.
You sign in to Cloudflare and give SEODojo read-only access to your analytics. Nothing on your site changes. (If you'd rather not use sign-in, Or paste an API token instead works too.) SEODojo then loads up to 30 days of history and updates once a day, a few hours after midnight UTC.
Note: Our first load hit Cloudflare's analytics rate limit ("Rate limiter budget depleted"), so only about the first nine days of our 30 had loaded when we took these screenshots. SEODojo retries on its own. The numbers below come from those nine days, and the patterns in them were already clear.
Step 2: Read the summary, and don't believe it yet
1,741 AI bot requests. For a site with 8 Google clicks in two months, that looks amazing.
Now look at the next number: 8% verified by Cloudflare.
Any program can call itself "GPTBot" or "PerplexityBot". It's just a line of text in the request. Cloudflare checks whether a request really came from the company it claims to be, and for ieltsbandlift.com, 92% of them didn't.
The Per day chart shows it too. Almost nothing for a week, then a sudden spike of more than 1,200 requests in one day. Real crawlers don't usually behave like that. A scanner sweeping the internet does.
Step 3: See what the fakes wanted
Scroll to Most-read pages.
Two rows are real crawling: /sitemap.xml and /robots.txt, read 69 times each by Claude-SearchBot and ClaudeBot, Anthropic's bots, which Cloudflare verified.
Almost everything else is someone looking for secrets: /.ssh/id_ecdsa, /.ssh/id_rsa, /.env, /config.env, /.git/HEAD, /gcp-service.json, /.aws/…. These are private keys, passwords and config files that a badly set-up server might expose by accident. Each of those requests claimed to be an AI crawler ("PerplexityBot", "ChatGPT-User", "OAI-SearchBot") to slip past filters.
That's a security lesson as much as an SEO one. Check that every one of those paths returns "not found" on your site. They should on any normal setup, but it takes two minutes to be sure, and it matters far more than any ranking.
Step 4: Read only the verified bots
Filter the By bot table in your head to what's verified, and the real picture for ieltsbandlift.com was:
- Anthropic came: Claude-SearchBot (87 requests, 100% verified) builds Claude's search index, and ClaudeBot (17% of its requests verified) collects for training.
- They read robots.txt and the sitemap, and in these first nine days we saw no visits to actual pages. They found the map, but hadn't started on the content yet.
- No verified visits from OpenAI, Perplexity, Amazon or Apple in that period. Every request in their names was unverified.
- "241 fetched to answer a person" sounds like people asking AI about us. All of it came from unverified "ChatGPT-User", "DuckAssistBot" and "MistralAI-User" requests, so we don't count any of it.
That explains a lot about lesson 7.1's result: 0 of 34 AI answers named us. AI search engines can only cite pages they've read, and as far as we can see, none of them had read ours yet.
Note: Google's AI Overviews and AI Mode aren't in this list. Google reads pages for them with its ordinary search crawler, so your Google rankings (Module 6) are how you measure that side.
What to do about it
- Security first. Make sure the secret-file paths return "not found". If you're worried by the volume, Cloudflare's security settings can block unverified bots that claim to be AI crawlers, while letting the verified ones through.
- Make your pages easy to find from the sitemap. Real bots start where Anthropic's did. Make sure your sitemap lists every page you want read, and that your important pages are linked from others (lesson 5.2's orphans).
- Get linked from pages AI bots already read. Crawlers follow links. A mention on a list article that ChatGPT already cites (lesson 7.2) is a path for its bot to your site.
- Check again in a month. Watch for the first verified visits from OpenAI and Perplexity, and the first verified "answering a person" request. That's the moment an AI assistant opened your page because someone asked about you.
Done when
- Your site is connected through Cloudflare
- You've looked at the verified percentage before believing any total
- You've checked that the secret-file paths in Most-read pages return "not found" on your site
- You know which real AI bots visit you, and which pages they actually read
- You have a date in a month to look again