Log File Analysis
SEOAlso: Server Log Analysis · Crawl Log Analysis
Quick definition
Log file analysis is the process of reviewing a website's raw server logs to see exactly which pages search engine crawlers, like Googlebot, requested and when. Unlike crawling tools that simulate a bot, log files show what actually happened, which pages got crawled, ignored or hit repeatedly.
How it varies across Australia
Larger Australian ecommerce and marketplace sites tend to have the most to gain from log file analysis, since crawl budget is genuinely finite once a catalogue passes a few thousand pages. Smaller brochure sites usually get crawled fully regardless, so the payoff is lower.
See technical SEO patterns across Australian industries →What it actually means
Every request to your server gets recorded somewhere, whether it's a customer loading a page or Googlebot checking if anything changed. A log file is that record. Log file analysis means pulling those records and asking specific questions. Which pages does Googlebot visit most. Which pages has it never touched. Is it wasting time on parameters, duplicate URLs or pages that 404.
This is different from a standard technical SEO crawl. A tool like Screaming Frog simulates a crawler and tells you what's crawlable in theory. Log files tell you what actually happened, no simulation involved. The gap between the two is usually where the interesting problems live.
Crawl budget is the concept that makes this matter. Google allocates a rough amount of attention to each site based on size and authority. On a large ecommerce catalogue or a site with heavy pagination, that budget gets burned on filtered category pages and thin variants instead of the pages that actually drive organic traffic. Log files are the only way to prove that's happening rather than guess at it.
A crawling tool shows you what a bot could see. Log files show you what a bot actually did.
How it shows up
Log file analysis shows up as a spreadsheet or dashboard pairing every crawled URL with hit count, response code and crawler identity. It surfaces things standard tools can't. Pages Googlebot hits daily that generate no organic traffic. Sections of the site it hasn't crawled in months. Spikes in crawl activity right after a sitemap update, or the absence of any spike at all, which tells you the sitemap was ignored.
The Australian context
Australian sites hosted overseas sometimes show up in log files with slower Googlebot response times purely because of the distance the requests have to travel. That's worth separating from genuine crawl inefficiency before you start blaming your architecture.
Where people get this wrong
Log File Analysis vs Technical SEO
| Log File Analysis | Technical SEO | |
|---|---|---|
| What it examines | Actual crawler requests recorded by the server | Site structure, code and configuration broadly |
| Data source | Raw server logs | Crawling tools, code review, page speed tests |
| Answers | What did Googlebot actually do | What could Googlebot theoretically do |
| Best suited to | Large sites with real crawl budget constraints | Any site regardless of size |
Related terms
Common questions
Do I need log file analysis for a small website?
Probably not. Crawl budget only becomes a real constraint once you have thousands of URLs or heavy pagination and faceted navigation. A site with a few hundred pages usually gets crawled fully without any intervention.
What tools are used for log file analysis?
Screaming Frog Log File Analyser, JetOctopus and Botify are common choices. Some larger sites pull raw logs into BigQuery or similar for custom querying. The right tool depends on log volume and how technical the team is.
How is this different from checking Search Console's crawl stats?
Search Console gives you a summarised, sampled view of crawl activity. Raw log files give you every single request, unsampled, which lets you drill into specific URL patterns Search Console won't break out for you.
How often should log files be reviewed?
Quarterly for large or fast-changing sites is a reasonable baseline. Beyond that, review immediately after a migration, a major redirect rollout, or a big change to site architecture, since that's when crawl behaviour is most likely to break.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →