Server logs are the only crawl data that isn't sampled

Why the log, not the console

The Crawl Stats report is useful for a trend and useless for a decision. It aggregates, it samples, and it expires. It won't tell you which template got 61% of Googlebot's requests last week while your new service pages got four hits between them, or that the CDN served a 503 to Googlebot for eleven minutes on Tuesday, or that half the "crawl" is a scraper wearing a Googlebot user agent.

The access log tells you all of that, per request, with a timestamp. It's raw, it's large, and it's the closest thing to seeing the site the way the crawler sees it. On a healthcare site with a few hundred URLs it fits in a spreadsheet. On anything larger it goes into BigQuery and stays there.

Verify Googlebot before you trust a line

Anyone can send a request with Googlebot/2.1 in the user agent, and a surprising amount of "Googlebot" traffic in a raw log is not Google. Before any of the cuts below mean anything, verify the bot by reverse DNS: resolve the IP to a hostname, confirm it ends in googlebot.com or google.com, then resolve that hostname forward and confirm it returns the original IP. Google also publishes its crawler IP ranges as JSON, which is faster at scale, but the reverse lookup is the check that catches spoofing.

import socket

def is_googlebot(ip):
    try:
        host = socket.gethostbyaddr(ip)[0]
    except socket.herror:
        return False
    if not (host.endswith(".googlebot.com") or host.endswith(".google.com")):
        return False
    return ip in socket.gethostbyname_ex(host)[2]

Cache the results. Reverse lookups are slow and the same IPs recur; a small dictionary keyed by IP turns a multi-hour pass into minutes.

The standard cuts

Every log analysis starts with the same five views. They answer the "is it being crawled" layer of the diagnostic stack and usually point at the fix by themselves.

cut	what it shows	what it usually finds
hits by user agent	share of requests per crawler, verified	spoofed bots inflating crawl estimates
status code over time	2xx, 3xx, 4xx, 5xx served to Googlebot per day	a 5xx spike that explains a coverage drop
frequency by URL template	crawl budget distribution across page types	budget spent on tags and archives, not services
wasted crawl	parameters, faceted URLs, redirected and 404 URLs	a filter plugin generating thousands of near-duplicate URLs
discovery lag	publish time versus first Googlebot hit	new pages found only via sitemap, days late

The template cut is the one most people skip and the one that most often changes the recommendation. Group URLs by pattern (/blog/, /services/, /?s=, anything with a parameter) and count verified Googlebot hits per group. If the important templates are starved, the fix is crawl control and internal linking, and no amount of new content will help until it's done.

A pipeline that doesn't expire

For a one-off, pandas is enough and the script can be handed to the client to re-run. For anything ongoing, the pipeline is logs into BigQuery, with Looker Studio on top. The reasons are practical: BigQuery handles a year of logs without complaint, the SQL is readable by the next person, and the same project can hold the Search Console bulk export so crawl, indexation and clicks sit side by side without sampling.

SELECT
  DATE(ts) AS day,
  REGEXP_EXTRACT(path, r'^/([^/?]+)') AS template,
  COUNTIF(status BETWEEN 200 AND 299) AS ok,
  COUNTIF(status BETWEEN 300 AND 399) AS redirect,
  COUNTIF(status >= 400) AS error
FROM `project.logs.access`
WHERE verified_googlebot
GROUP BY day, template
ORDER BY day DESC, ok DESC;

Keep the log-level detail in internal views and put the aggregates on the client dashboard. A practice manager doesn't need to see request lines; they need to see that Googlebot hit the new physiotherapy page 40 times this week and got a 200 every time.

The rule this enforces

Never recommend a content fix for what the logs show is a crawl problem.

It sounds obvious, and it's the single most common mistake in SEO recommendations. A page that Googlebot requested twice in a quarter isn't underperforming because of its copy. Read the log first; the copy can wait.