Understand how Google allocates crawl budget to your site - and how to make every crawl count.
Check your DA in Ahrefs, SEMrush or Moz
Faceted nav, paginated pages, URL params, etc.
Est. daily crawl rate
~120
Pages/day Googlebot visits
Days to crawl site
4.2
Full site crawl cycle
Wasted crawl budget
100
Pages/day on thin content
Crawl efficiency score
64%
% of crawl on quality pages
Crawl budget factorImpactYour score
Domain authorityHigher DA = more crawl allocation–
Page speedSlow servers reduce crawl rate–
Duplicate contentWastes budget on low-value pages–
Internal linkingHelps Googlebot discover pages–
Server reliabilityErrors reduce crawl frequency–
Adjust your site metrics to estimate your crawl budget health.
Crawl budget. The resource most sites waste without realising it
Google doesn't crawl every page on your site every day. It allocates a crawl budget, a finite number of page visits per day, based on your site's authority, server speed, and content quality. For smaller sites this rarely matters. But for sites with hundreds or thousands of pages, wasting crawl budget on low-quality URLs means your important pages get crawled less frequently, indexed more slowly, and updated less often in Google's index.
What wastes crawl budget
The most common crawl budget wasters are: faceted navigation URLs (e.g. /shoes?colour=red&size=8), pagination pages with thin content, near-duplicate pages across international subdirectories, session IDs in URLs, redirect chains, and parameter-based URLs that generate thousands of variations of the same content.
How to reclaim it
Use noindex tags on low-value pages (zero-result pages, thin filtered views, print versions). Use canonical tags to consolidate duplicate content. Block parameter-based URLs in robots.txt or via Google Search Console's URL parameters tool. Consolidate redirect chains into direct 301s. Each fix returns that crawl budget allocation to your important pages.
Speed and authority matter too
Googlebot respects your server's capacity, slow server response times cause it to crawl less aggressively to avoid overloading your server. Higher domain authority tells Google your site is trusted and worth frequent crawling. A fast, authoritative site with clean crawl paths will always be crawled more thoroughly than a slow, low-authority site with crawl traps.
What I found in a real international site audit
Vuly Play had 720 pages "Crawled, not indexed" and 33 "Discovered, not indexed" across their 8-country site. Investigation found spare parts pages showing "no spare parts available" (zero-result pages, noindex fix), blog posts duplicated across country sites without hreflang (canonicalisation fix), and pages blocked by robots.txt that should have been indexed for E-E-A-T signals, including the /accessories/all page that was actually linked in the main navigation.
How do I check my current crawl rate?
Google Search Console shows crawl data under Settings → Crawl Stats. This gives you the average number of pages Googlebot crawled per day over the past 90 days, broken down by response type. You can also use Screaming Frog with a log file analyser to see exactly which pages are being crawled and how frequently.
Should I use robots.txt or noindex to manage crawl budget?
These do different things. Robots.txt prevents Googlebot from crawling a page but does not prevent it from being indexed, Google can still index a disallowed page if other sites link to it. Noindex allows crawling but prevents indexing. For crawl budget management, use both: robots.txt for pages you never want crawled (admin pages, login areas), noindex for pages you want crawled but not indexed (thin filtered views, print pages).
Does crawl budget matter for small sites?
For sites under 1,000 pages with good server speed, crawl budget is rarely a limiting factor, Google will typically crawl the whole site regularly regardless. Crawl budget becomes critical for large eCommerce sites (10,000+ product pages), news sites with high publication frequency, and international sites where the same content exists across multiple locale subdirectories.
Want a full SEO audit for your site?
Grant Hilton, 25+ years experience, Brisbane-based, Australia-wide