| SUMMARY: Google’s own documentation says crawl budget only matters for sites with 1 million+ pages, or 10,000+ pages with daily updates. That’s it. John Mueller calls crawl budget “over-rated” for most sites. Most guides ignore this. Crawling is roughly proportional to your authority, not to how clever your sitemap is. Pruning pages and faking lastmod dates won’t fix it. If you have under 10,000 pages, crawl budget isn’t the problem. It’s almost certainly authority, internal linking, or content quality. Real fixes for normal sites: build genuine backlinks, sort internal linking, speed up your server, fix actual technical errors. |
Most of the advice you’ve read about crawl budget is either out of date, hopelessly oversimplified or just made up. Here’s what actually drives whether Google crawls your pages. And what doesn’t.
If you’ve ever Googled “how to optimise your crawl budget,” you’ve probably been served a buffet of nonsense.
Update your sitemap. Prune your old pages. Submit your URLs. Speed up your server. Add structured data. Sacrifice a goat.
It all sounds technical. It all sounds plausible. And for 99% of websites, almost none of it matters.
Sorry.
I’ve been doing SEO for over 15 years, and the obsession with crawl budget is one of the most persistent myths in the industry, mostly kept alive by guides written for big sites that get copy-pasted at small sites that don’t have the problem in the first place. So before you spend another afternoon “optimising” something Google has openly told you to ignore, let’s talk about what crawl budget actually is, when it matters (rarely), and what to do instead.
What Google actually says about crawl budget
Let’s start with the source. Google’s own documentation on crawl budget management is unambiguous about who needs to care:
- Large sites with more than 1 million unique pages updated weekly.
- Medium sites with more than 10,000 pages updated daily.
- Sites with a high proportion of “Discovered – currently not indexed” in Search Console.
If your site doesn’t fall into one of those three buckets, Google itself says: “merely keeping your sitemap up to date and checking your index coverage regularly is adequate.” That’s the official guidance. Direct from the team that built the crawler.
Gary Illyes from Google confirmed in 2025 that the 1-million-page threshold still stands. And John Mueller, Google’s Search Advocate, has been even blunter:
IMO crawl-budget is over-rated. Most sites never need to worry about this.”
John Mueller, Google
That tweet is from 2018. It’s still pinned in the back of every honest SEO’s brain because it’s still true.
Mueller has since added that “crawling is independent of website size”. Some massive sites have, in his words, a “gazillion useless URLs” that Google barely crawls, while small sites with strong signals get crawled efficiently.
So, where does all the BS comes from?
The crawl-budget-optimisation-guide industrial complex didn’t appear from nowhere. Three things keep it alive.
1. Old advice, infinite shelf life
A lot of crawl budget content is genuinely 10+ years old, written when Google’s crawler was less sophisticated and shared hosting was wheezing. The advice gets quoted, paraphrased, and republished, year after year, with the publish date refreshed. Search results don’t really tell you which guide was written for a 2014 e-commerce site running on a Pentium and which one was written last week. They all look equally authoritative.
2. The “guide for big sites” problem
Google’s documentation is genuinely written for very large enterprise sites. The advice in it is solid. If you have a million pages. The problem is that most people reading it don’t. They’re running a 200-page service business website and applying enterprise-level advice to a site that Google can crawl in its sleep. It’s like reading a manual on managing a fleet of HGVs and applying it to your Ford Focus.
3. The desire for a magic switch
If your pages aren’t ranking, the comforting thing is to believe there’s a technical fix. A button you haven’t pressed yet, a sitemap you haven’t optimised, a parameter you haven’t blocked. Crawl budget gives people that hope. It’s much harder to accept the actual answer, which is usually “you need more authority and better content,” because that takes years and there isn’t a checklist for it.
What actually drives crawl: authority
Here’s the bit nobody wants to hear. Crawl frequency is driven by authority, not by technical fiddling.
Back in 2010, Matt Cutts (then head of Google’s webspam team) told Eric Enge in a now-famous interview:
“The number of pages that we crawl is roughly proportional to your PageRank.”
Matt Cutts, ex-Google
PageRank as a public metric is dead. The principle isn’t. Google still allocates crawl resources based on what it thinks of your site: popularity, link signals, freshness, user value.
Call it authority. Call it whatever you like. The point is: it’s earned, not configured.
Ross Tavendale of Type A put it neatly in Conductor’s crawl budget guide: “the largest spikes in crawled pages we see in Google Search Console directly relate to when we win big links for our clients.”
Not when they restructure their sitemap. Not when they prune 30 old blog posts. When they get links.
Lily Ray, who has presented on enterprise crawl budget at BrightonSEO, makes a related point: the goal isn’t to game the crawler, it’s to make sure everything Google does crawl is worth indexing. Trim the dead weight, sure. But don’t expect Google to suddenly hand out more crawl just because you’ve tidied up.
The myths the guides keep selling you
Here are the worst offenders…
Myth 1: “Removing pages frees up crawl budget”
This one is the zombie of crawl budget myths. The logic seems intuitive: fewer pages, more attention to the ones that remain. The reality is that pages with no traffic and no links were barely getting crawled in the first place. Google had already deprioritised them. Removing them doesn’t redistribute anything. It just removes them.
If you want to prune pages because they’re bad, fine. Prune them. But don’t expect crawling of your homepage to magically go up. It won’t.
Myth 2: “More crawling = better rankings”
Even Google’s official documentation flatly states that crawl rate is not a ranking signal. Getting Google to fetch your page more often doesn’t make it rank higher. It just makes it fetch your page more often. Which, if the page is already crap, is just faster delivery of the same crap.
Myth 3: “Update your sitemap dates and Google will recrawl”
Tempting. Lazy. Doesn’t really work for most sites. Authoritative sites with high-priority crawl pools (think CNN, Reuters) genuinely benefit from accurate <lastmod> tags because Google’s crawlers are practically watching their sitemaps in real time. For everyone else, faking the dates teaches Google’s systems to start ignoring your <lastmod> tags entirely. There’s no going back from that.
Myth 4: “You can throttle Googlebot to optimise crawl”
You used to be able to. Google removed the manual crawl rate setting from Search Console. The crawl-delay directive in robots.txt? Googlebot ignores it entirely. Always has. If your server is healthy, Googlebot adjusts its rate automatically. If it isn’t, fixing the server is the answer, not faking it with a directive Google doesn’t read.
When does crawl budget actually matter?
To be fair, there are real cases where crawl budget is a serious concern. None of them apply to most readers of this blog, but here they are:
- E-commerce sites with faceted navigation that generate hundreds of thousands of filter-combination URLs (size + colour + price + brand). This is where Google can genuinely get lost in URL bloat.
- Enterprise publishers with millions of pages such as large news sites, marketplaces, real estate portals.
- Sites with serious server performance issues. Gary Illyes was clear in 2025 that “if you are making expensive database calls, that’s going to cost the server a lot.” A 500,000-page site with slow SQL queries can have worse crawl issues than a 2-million-page site that’s fast.
- Sites with a chunk of their URLs stuck in “Discovered – currently not indexed.” That’s often a quality/authority issue rather than a crawl issue, but it’s the symptom Google explicitly flags.
There’s also a newer wrinkle worth noting: Cloudflare reported that AI and search crawler traffic grew 18% between May 2024 and May 2025. With GPTBot up 305% and Googlebot up 96% in that period alone. Servers that were fine two years ago are now juggling more bot traffic than ever. If your hosting was always borderline, this is the year it might tip over.
If your site has fewer than 10,000 pages and decent server speed, crawl budget is not your problem. It is genuinely, mathematically not the bottleneck. Stop chasing it.
What should I do instead?
If you’re a normal-sized business site (such as a service company, B2B SaaS, established blog), here’s where your time is better spent:
1. Build genuine authority
Backlinks from real, relevant sites. Mentions from people who matter in your space. The boring stuff that compounds over time. As Ross Tavendale said: this is what produces the crawl spikes. This is what increases the size of your “pool” in Google’s allocation.
2. Sort your internal linking
Internal links pull pages out of low-priority crawl queues and into higher ones. They distribute authority around your site. They help Google understand which pages you think matter. If you have orphan pages (no internal links pointing to them), that’s a crawl problem you can actually fix.
3. Make your important pages obviously important
Strong slugs. Clear page titles. Genuine content depth. Don’t bury your money pages five clicks from the homepage. A flat-ish architecture means link equity flows where you want it to flow.
4. Sort the actual technical errors
Broken internal links. 5xx server errors. Soft 404s. Redirect chains. These genuinely waste Googlebot’s time. Fix them. But fix them because they’re errors, not because you’ve convinced yourself they’ll “unlock” crawl budget.
5. Speed your server up
Aim for a TTFB (time to first byte) under 400ms. This is one of the few areas where technical work translates directly to better crawling. A faster server means Google can fetch more pages in the same window.
Crawl before you walk
Crawl budget optimisation guides exist because they’re easy to write, easy to rank, and easy to sell. The actual answer for 99% of sites is “make your site genuinely good and authoritative, and Google will crawl it fine”. But this is harder, slower, and doesn’t make for a snappy 12-step listicle.
This is the kind of thing I find myself talking about with clients all the time. They’ve been told by previous agencies that they have a “crawling problem,” when what they actually have is a topical authority problem, or a content depth problem, or a backlinks problem. Once we stop chasing crawl optimisation ghosts and focus on the real levers, the rankings start to move. Funny, that.
Don’t let Google crawl past you
If you’ve been told you have a crawl budget problem and you’ve got fewer than 10,000 pages, you almost certainly don’t. What you probably do have is an authority problem, and that’s something we can actually do something about.
At Wrise, we work with businesses to build the kind of expertise-led content and link profiles that Google rewards with rankings, not just crawls. SEO packages start from £99/month, no contracts, no setup fees.
Get a free local review of your website! 🧐
Want to boost your website’s position in the local map pack? Tell us about your project to get started. We’ll provide a free preliminary review and local SEO action plan.
Affordable SEO Services for Small Businesses 🚀
Outsmart your competition with a low-cost SEO package that delivers real results. Stop dreaming. Start ranking!
✅ No Contracts
✅ No Set Up Fees
✅ Free Review
Affordable SEO Services for Small Businesses 🚀
Outsmart your competition with a low-cost SEO package that delivers real results. Stop dreaming. Start ranking!
Written by Rob Truslove
Rob is the founder of Wrise and a specialist in SEO and search strategy. He has over 15 years’ experience helping more than 500 brands grow their online visibility and performance.
With an academic background in creative writing and storytelling, Rob combines narrative-led content with data-driven search strategy to help websites rank, engage and convert. Over his career, he has worked with household brands including P&O, The AA, Trevor Sorbie and Purple Bricks, as well as ambitious SMEs looking to scale quickly.
