How to Read Your Server Logs for AI-Bot Traffic (An AEO Log-File Playbook)

The 30-second answer

If you want to know whether AI answer engines are visiting your site, stop guessing from a probability-based AEO tool and go read your own logs. Three numbers to anchor this: AI assistants send visitors to 404 pages at 2.87 times the rate Google Search does (Ahrefs, 2025). Cloudflare has reported bot requests overtaking human requests for HTML pages on its network. And a large citation study found that blocking a training crawler in robots.txt barely changes whether you get cited at all. Your server already recorded the evidence. This is how to read it.

What counts as "AI-bot traffic" in your logs

Most people mean one of three different things when they say "AI bot" and that confusion is why the advice online is so scattered. There are training crawlers that harvest pages to improve a future model. There are answer-time crawlers that fire the moment a real person asks a real question. And there's everything else pretending to be one of the above, because user-agent strings are just text anyone can type into a request header. Here's the reference table we wish existed when we started pulling this data for our own site. Grep for the exact token in the "UA token" column. A fuzzy match on the bot's name will pull in noise.

Two of these rows carry an honest asterisk. Applebot-Extended's exact UA string wasn't confirmed on a direct read of Apple's own support page during our research (the page truncated on fetch), so don't quote a literal string for it until you've checked it yourself. And Bytespider, ByteDance's crawler, has no official vendor documentation at all. Every UA string for it floating around the internet, including ours if we'd printed one, traces back to third-party crawler-directory sites. ByteDance itself hasn't published any of it. Treat anything you read about Bytespider as unverified until ByteDance says otherwise.

Training crawler or answer-time crawler? The distinction changes what you do

This is the part most "block the AI bots" advice skips, and it's the single most useful mental model in this whole piece. Every major vendor with a training crawler draws the same line in its own documentation: one bot harvests content for a future model version, and a separate, differently-named bot fires live when someone actually asks a question. OpenAI splits GPTBot (training) from ChatGPT-User (a live fetch OpenAI says "may not apply" robots.txt to, because a person triggered it) and OAI-SearchBot (search indexing). Anthropic splits ClaudeBot (training) from Claude-User (a live fetch when someone asks Claude a question) and Claude-SearchBot (search quality). Meta splits meta-externalagent (training and indexing) from meta-externalfetcher, which Meta's own docs say "may bypass robots.txt" outright. Apple splits the base Applebot, which powers Siri and Spotlight and Search, from Applebot-Extended, a training-only opt-out where disallowing one has zero effect on the other. Perplexity is the outlier worth remembering. PerplexityBot isn't a training crawler by Perplexity's own claim, it's search indexing only, and Perplexity-User is the live fetcher that "generally ignores robots.txt rules." Why this matters for your logs: if you only grep for "GPTBot" you're measuring how much OpenAI's model-training pipeline touched your site, which tells you nothing about whether ChatGPT is actually citing you to a live user right now. The traffic you actually care about for AEO is the answer-time bots (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher), because that's the fetch that happens the moment your page might get quoted back to a real person.

Method 1: grep your own access logs (if you control the server)

If you're running on your own VPS or a server where you can read the raw Apache or Nginx access log, this is the fastest path and it costs nothing.

grep -i "GPTBot" access.log | wc -l
grep -i "GPTBot" access.log | awk '{print $9}' | sort | uniq -c
grep -i "GPTBot" access.log | awk '$9 == 404'

The first line counts total hits. The second breaks hits down by status code, so you can see how many landed on a success, a redirect, or a 404. The third isolates just the 404s, which is the single most useful filter in this whole piece (more on why below). Run the same three lines for ClaudeBot, PerplexityBot, ChatGPT-User, Claude-User, and Perplexity-User, and you have a real, first-party picture of who's actually showing up. The catch: this only works if you have filesystem access to the origin server's raw log, and if the request actually reached the origin instead of getting served from a cache layer in front of it. Most B2B SaaS marketing sites today sit behind a CDN, which means the origin log is missing most of the traffic.

Method 2: read your CDN's edge analytics (Cloudflare, Vercel, CloudFront)

If you're behind a CDN, the CDN's edge is what actually saw the request, and each platform exposes that data differently. Cloudflare's classification system changed in 2026. As of July 1, 2026, Cloudflare retired "AI Crawler" as its own category and folded it into a broader "Training" behavior classification, paired with an "Operational Label" of Direct versus Intermediary access. Cloudflare identifies a verified bot three ways: a cryptographic signature (Web Bot Auth), a published IP range paired with a stable user-agent, or reverse-DNS validation. The programmatic path is Cloudflare's GraphQL Analytics API, specifically the httpRequestsAdaptiveGroups dataset, queried by user-agent and grouped by status code. We built exactly this for our own site, a script we call /ai-crawl-report that pulls Cloudflare zone analytics and classifies AI bot traffic by user-agent, because the AI Crawl Control dashboard on Cloudflare's free plan is locked to 1-hour and 24-hour windows with no way to query it programmatically. One thing we learned building it that we couldn't find written down anywhere in Cloudflare's own docs: the free plan caps that dataset at roughly one day per query, and overshooting by even a few milliseconds gets rejected with a quota error. That's an operational finding from our own build. Cloudflare hasn't documented that particular ceiling anywhere we could find, so budget for it if you're building something similar. Here's the shape of the output. This is a format example only, walking through what the report structure looks like; it isn't a live pull from our zone.

Two reads matter more than the raw counts. A bot that drops to zero requests in a window it normally shows up in is a signal something broke (a robots.txt change, a firewall rule, a bot-management toggle), worth checking before you assume it's just quiet. And a rising unsuccessful count for one specific bot is worth a look at which URLs it's hitting, because that's usually either a redirect you should fix or a genuine server error. If you're on Vercel, the equivalent is log drains or Runtime Logs streamed through the platform rather than a flat file. If you're behind CloudFront, access logs land in an S3 bucket on a delay, in W3C-extended format rather than the Apache combined format Method 1 assumes. None of these are a live grep target the way an origin log is. Match the method to your actual stack instead of copying a snippet built for someone else's.

The mistake we see most often is someone grepping their origin log, finding almost nothing, and concluding AI bots aren't visiting. Half the time the real answer is that the CDN in front of the origin already served the request from cache and the origin never saw it. Check which layer actually terminated the request before you trust a low number.

Method 3: verify a bot isn't spoofed before you trust the count

User-agent strings are just text. Anyone's scraper can claim to be GPTBot. Before you build a strategy on a number, verify at least a sample of it. Most vendors publish a machine-readable IP list built for exactly this. OpenAI publishes three separate JSON files, one per bot. Anthropic publishes one combined file covering all three of its crawlers, with an explicit warning that IP-only blocking "may not work correctly" as a long-term strategy since ranges change. Perplexity publishes per-bot JSON lists and explicitly tells you to pull the current version rather than hardcode it. Google offers both a reverse-DNS method (run host on the IP, confirm it resolves to googlebot.com, google.com, or googleusercontent.com, then confirm the forward lookup matches) and five separate IP-range JSON files for different crawler types. Two vendors have a real gap here, and it's worth knowing before you rely on their UA string alone. Meta publishes no IP list for either of its crawlers, user-agent matching only, by its own documentation. ByteDance's Bytespider has no official verification path at all because ByteDance doesn't publish crawler documentation in the first place. If a request claiming to be one of those two matters to a decision you're making, check the IP against what you know of that operator's infrastructure manually, or don't lean on that single data point.

What the numbers actually mean once you have them

Two ideas do most of the heavy lifting once you're looking at real log data. The crawl-to-refer ratio. Cloudflare Radar defines this as total crawler requests for HTML content from a given AI platform, divided by total HTML requests whose referer header names that platform. It's a live, constantly-updating number on Cloudflare's own dashboard rather than a fixed report, and Cloudflare's own methodology note flags that native in-app referrals (which carry no referer header) can overstate the ratio by an amount they say is unclear. Don't quote a specific ratio as evergreen fact in a strategy deck. Pull it live from Radar the day you need it, and read the trend rather than the snapshot. AI-bot 404s are a demand signal, not just an error to fix. When an AI assistant's fetcher requests a URL on your domain that doesn't exist, the model pattern-matched your URL structure and inferred a page should be there. Ahrefs' 2025 study, built from over 8 million clicked AI-referred URLs plus roughly 9 million cited URLs benchmarked against Google's equivalents, found AI assistants send visitors to 404 pages at 2.87 times the rate Google Search does. Per-assistant, ChatGPT's clicked-URL 404 rate came in at 1.01% against Google's 0.15% baseline. That gap is either a stale, once-valid URL the model remembers from training data, or a fully invented URL that only sounds plausible. Either way, every 404 pattern you see repeating is a content brief someone else already asked for. We wrote a full separate guide, The AI Demand Engine, on turning that specific signal into an automated Cloudflare-to-Notion pipeline.

The myth that won't die: does blocking robots.txt actually stop citations?

No, and the data on this is more direct than most people assume. A citation study from BuzzStream, built from 4 million citations across 3,600 prompts spanning ChatGPT, Gemini, AI Overviews, and AI Mode across 10 industries, found 88.2% of sites blocking GPTBot in robots.txt got cited anyway, and 92.3% of sites blocking Google-Extended got cited anyway. On the answer-time side, 70.6% of sites blocking ChatGPT-User still showed up in citations, and roughly 70% of all ChatGPT citations in that dataset came from sites that block ChatGPT's own retrieval bot. I don't think this means robots.txt is pointless. It's still the correct signal for "don't use my content to train your model," and several vendors honor that distinction cleanly. What it means is that robots.txt functions as a content-use preference. It isn't a citation lever. If you're blocking bots hoping it'll change your visibility one way or the other, the evidence says it won't, and you should stop treating that toggle as a growth decision.

A worked example: running this on our own site

We run Cloudflare on loudface.co, so we built /ai-crawl-report against our own zone rather than wait for a vendor dashboard to catch up. The mechanism is the same three things this piece just walked through: query httpRequestsAdaptiveGroups scoped to a 24-hour window (the free-plan ceiling we mentioned above), classify each request's user-agent against a maintained list of known AI bots, and flag anything that looks like a new, unclassified crawler showing up at real volume so we can decide whether to add it to the list. What we actually use it for: confirming a newly published piece got picked up by Bing within minutes and by the AI answer-time bots within a few days, and catching the moment a bot that normally shows up every day goes quiet, which is usually the first sign something in a Cloudflare config or a robots.txt edit broke access we didn't mean to break. That's the whole point of reading your own logs instead of waiting on a third-party dashboard. You see the change the day it happens, not the week a client asks why traffic looks different. Our own AEO work moved LoudFace from 0.18% to 10.35% of AI answers on tracked prompts in 90 days (full case study here, the play-by-play here), and log-level visibility into what's actually crawling us is part of how we caught the moves that worked early enough to double down on them. If you'd rather have a team build and run this end to end, that's the same work our SEO and AEO service does for clients.

How often should you actually check this?

Not daily, and definitely not manually every day. Set up a weekly pull at minimum, and add an ad-hoc check any time you ship a piece you specifically want AI engines to pick up. Bing typically shows up within minutes of a new URL going live. The AI answer-time bots take anywhere from a few days to a couple of weeks, depending on how often that specific topic gets asked about. If you're checking hourly hoping to catch the exact moment ChatGPT-User shows up, you're spending effort on a cadence the data doesn't reward. Weekly is enough to catch a real trend, and an alert on a sudden zero for a bot that normally shows up is worth more than any dashboard you'd stare at daily.