A practical crawl budget checklist for large sites
2026-09-07
Crawl budget is the number of URLs a search engine is willing to fetch from a site over a period. For a few hundred pages it does not matter. For tens or hundreds of thousands, it decides how quickly new and updated pages are seen, and most of it is usually being spent on URLs that will never rank.
Find where it leaks
The usual culprits are faceted navigation, where every combination of filters is a crawlable URL, session and tracking parameters that create infinite variants of the same page, paginated archives that go on for hundreds of pages, and thin or duplicate pages that exist but add nothing.
Pull a server log sample and look at what the crawler actually fetched. If a large share of crawls are hitting parameter URLs and filter combinations, that is budget not being spent on your content.
Concentrate the crawl
Tighten internal linking so the pages that matter are the easiest to reach in the fewest clicks. A search engine crawls what is linked, so link structure is the main lever.
Control parameter and filter URLs. Use canonical tags to point variants at the clean URL, robots rules to stop crawling of obviously infinite spaces, and noindex where a page should exist for users but not for search.
Keep the sitemap limited to canonical, indexable pages. A sitemap full of redirects, noindex pages and 404s wastes the crawler time and lowers trust in the file.
Fix the health signals
A slow site gets crawled cautiously. So does one that returns a lot of 5xx errors or long chains of redirects. Server response time and error rate directly affect how aggressively a site is crawled.
Prune or consolidate thin pages. Every page that will never rank is a page the crawler still has to consider.
Measure it
Track discovered pages against pages worth indexing over time. If the gap is large and growing, crawl budget is being wasted. Semantyra reports both figures from a crawl, along with the orphan and thin-page issues that usually explain the difference.
The takeaway
Crawl budget leaks into faceted URLs, parameters, endless pagination and thin pages. Concentrate it with tighter internal linking, canonical and robots control of variant URLs, a clean sitemap, and by fixing the speed and error signals that make a site get crawled cautiously.
Related
See this measured on your own site.
Start free scan