Crawl Budget
SEOAlso: Crawl Rate · Crawl Allocation
Quick definition
Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. Google allocates each site a crawl budget based on how fast your server responds and how valuable it judges your content to be. If your budget is spent on low-value pages, your important pages may not get crawled or indexed.
How it varies across Australia
Most small to mid-sized Australian sites never hit crawl budget limits. The constraint shows up mainly on large ecommerce sites with thousands of filtered URLs, news publishers with deep archives, or sites that have accumulated years of redirect chains and duplicate content. If your site is under a few thousand pages and technically clean, crawl budget is rarely the bottleneck.
See technical SEO scores across Australian industries →The two forces that shape your budget
How fast Googlebot can crawl without overwhelming your server. Improved by faster server response times.
How much Google values checking your site frequently. Driven by links, freshness and past ranking performance.
What it actually means
Think of crawl budget like a visiting inspector with a fixed number of hours. They will look at as many rooms as they can in the time given. If you fill the building with empty storage closets, they spend half their visit on rooms you never intended them to see, and some of the important ones don't get checked.
Google's crawl budget is shaped by two things it calls crawl capacity and crawl demand. Crawl capacity is how often Googlebot is willing to visit without overloading your server. Crawl demand is how much Google thinks your site is worth checking frequently, which is driven by how many pages have inbound links, how often content changes, and how valuable the indexed pages have proven to be.
For most sites, crawl budget is not the limiting factor for rankings. The limiting factor is content quality, backlinks and Core Web Vitals. But for larger sites, particularly ecommerce platforms with faceted navigation generating thousands of near-duplicate URLs, crawl budget becomes the reason why new product pages take weeks to appear in search results even when the rest of the technical SEO is clean.
The fix is less about adding resources and more about subtraction. Blocking junk URLs via robots.txt or noindex, consolidating duplicate pages, fixing redirect chains, and improving server response times all redirect Googlebot's attention toward the pages you actually want ranked.
Googlebot is not patient. Waste its time on junk pages and it leaves before reaching the ones that matter.
How it shows up
Crawl budget problems show up in a few places. Search Console's crawl stats report shows how many pages Googlebot requested each day and what response codes it saw. A high volume of 404s, redirect chains or soft-404s is a signal that budget is being spent poorly.
The index coverage report shows pages that are discovered but not indexed, which is often a symptom of crawl budget exhaustion on larger sites. If you're publishing new pages that take more than a few days to appear in search results on an otherwise healthy domain, crawl budget is one of three likely explanations, alongside thin content and a weak internal link structure.
The Australian context
Australian ecommerce sites running on platforms like Shopify or Magento often generate large volumes of filtered and parameterised URLs through category navigation, colour and size filters, and search result pages. These are common sources of crawl waste that eat into the budget available for product pages.
Sites hosted on servers based overseas can also see reduced crawl rates for Australian visitors if server response times are slow from Google's Australian crawl infrastructure. Pairing crawl budget work with a content delivery network or Australian-based hosting can improve both crawl rate and Core Web Vitals at the same time.
Where people get this wrong
Related terms
Common questions
Does crawl budget affect my rankings directly?
Not directly, but indirectly. If important pages are not crawled they cannot be indexed, and pages that are not indexed cannot rank. For most sites the ranking ceiling is content quality and links, not crawl budget. For large sites with crawl waste, fixing crawl budget unlocks indexation that was already being blocked.
How do I check my crawl budget in Search Console?
Go to Settings in Search Console and find Crawl Stats under the Crawl section. You'll see daily crawl volume, response codes, and what types of files Googlebot is spending time on. A high share of 404s or redirects is the clearest signal that budget is being wasted.
Should I use noindex or robots.txt to control crawl budget?
Use robots.txt to stop Googlebot crawling pages you do not want it to visit at all, such as staging environments or internal search result pages. Use noindex for pages you want crawled but not indexed, such as thank-you pages or legal pages. Do not use robots.txt on pages that are already indexed and that you want deindexed.
How long does it take for crawl budget improvements to show up?
Changes to robots.txt and noindex tags are usually picked up within a few days of Googlebot's next visit. The downstream effect on indexation for previously neglected pages can take weeks, because Googlebot needs to return and re-evaluate them. Search Console crawl stats will reflect the change faster than the index coverage report.
Debrief
Get the next one
No spam. No fluff. Just the next article, straight to your inbox.
Keep exploring
About New Rebellion
New Rebellion is a marketing intelligence consultancy. We build tools, score Australian businesses on how their marketing actually performs, and publish Debrief every day. This dictionary is part of how we work in the open.
How we think →