Skip to content
SEOWofSEOWof
Book an inspection
Browse the standard

Standard SW-1:2026.2 · Category 1 of 9

Crawlability & Access

Everything else on the checksheet is irrelevant if a crawler cannot get in. This category inspects the front door: directives, response codes, redirect behaviour, sitemap integrity and server reliability under crawl load. It carries the heaviest weight in the standard because a single fault here can suppress an entire site regardless of how good the content is.

Weight
16 / 100
Checks
7
Pass-blocking
3
WOF analogue
Doors & access
CR-01Critical failweight 4measured

robots.txt does not block indexable revenue paths

Does robots.txt block any URL path that contains pages you need indexed?

Pass, advisory and fail criteria for CR-01
PassNo URL returning HTTP 200 with an indexable meta directive is disallowed for Googlebot.
AdvisoryDisallow rules exist that block only non-indexable or non-canonical URLs (e.g. faceted parameters, cart, internal search).
FailOne or more indexable pages are blocked, or the file returns 5xx, or a blanket "Disallow: /" is present on the production host.

Why it matters

This is the single most destructive fault a website can have and one of the most common, because a staging-environment robots.txt shipped to production is a routine deployment accident. A disallowed page cannot be crawled, so its content is never assessed, its internal links are never followed, and its existing rankings decay as the cached version ages. Unlike a noindex directive, the page may still appear in results as a bare URL with no snippet, which is worse than absence.

How we inspect it

We fetch /robots.txt, parse every group, and test every discovered URL against the applicable Disallow rules using the same longest-match precedence Googlebot applies. Any indexable URL that resolves to Disallow is recorded with the exact rule line that blocked it.

Source: Live fetch of /robots.txt + path matching against discovered URLs

Remedy

  1. 1Retrieve the production robots.txt and diff it against your intended source-controlled version.
  2. 2Remove or narrow any Disallow rule that matches an indexable path; never use robots.txt to keep a page out of the index — use a noindex directive, which requires the page to be crawlable.
  3. 3Keep the file under 500 KiB, return HTTP 200, and serve it as text/plain from the root of every host and protocol variant.
  4. 4Add the absolute URL of your XML sitemap index as a Sitemap: directive.
  5. 5Move robots.txt into version control with a deployment test that fails the build if "Disallow: /" appears in a production artefact.

Verification

Recheck refetches the file, re-runs path matching across the full URL set, and confirms zero indexable URLs match a Disallow rule.

Usually fixed by
Developer
Effort
Small · 0.5–2 hrs
Est. impact
high
Typical cost
NZ$80–160
CR-02Major failweight 3measured

Server responds reliably under crawl load

Do any pages return 5xx errors or time out when crawled?

Pass, advisory and fail criteria for CR-02
Pass5xx rate below 0.5% of crawled URLs and p95 TTFB under 800 ms with no concurrency-correlated degradation.
Advisory5xx rate below 0.5% but p95 TTFB between 800 ms and 1,500 ms.
Fail5xx rate at or above 0.5%, any timeout on a template-level URL, or TTFB degrading materially as concurrency rises.

Why it matters

Search engines reduce crawl rate when a host returns server errors, and they do so conservatively — recovery of crawl budget takes far longer than the outage that caused it. Intermittent 5xx responses are especially damaging because they are invisible in normal browsing and rarely reproduce on demand, so they persist for months. A site that fails only under crawl load will silently lose index coverage while every manual spot-check passes.

How we inspect it

Every URL in the crawl set is recorded with its status code and time-to-first-byte. We compute the 5xx rate, the timeout rate, and the p95 TTFB, and we flag any degradation correlated with request concurrency — the signature of a server that fails specifically under crawl conditions.

Source: Inspection crawl response codes + timing distribution

Remedy

  1. 1Pull the last 30 days of server logs and isolate 5xx responses by user agent, URL pattern and hour.
  2. 2Separate genuine application errors from rate limiting, WAF blocks and CDN origin timeouts — the remedies are completely different.
  3. 3If a WAF or bot-mitigation layer is responsible, allow verified search-engine crawlers by reverse-DNS verification rather than by user-agent string.
  4. 4Add response caching at the edge for anonymous requests so crawl traffic does not reach the origin application.
  5. 5Instrument alerting on 5xx rate by user agent so crawler-specific failures surface immediately.

Verification

Recheck re-crawls the full URL set at the same concurrency and confirms the 5xx rate and p95 TTFB are inside the pass criteria.

Usually fixed by
Developer
Effort
Medium · 2–8 hrs
Est. impact
high
Typical cost
NZ$240–800
CR-03Minor failweight 2measured

No redirect chains, loops or soft-404 redirects

Do any internal links or redirects pass through more than one hop?

Pass, advisory and fail criteria for CR-03
PassAll redirects resolve in a single hop to a relevant HTTP 200 URL. No loops.
AdvisoryTwo-hop chains exist but only on external inbound paths, not on internal links.
FailAny loop, any chain of three or more hops, any internal link pointing at a redirect, or any redirect to an irrelevant fallback page.

Why it matters

Each hop adds latency for the user and dilutes the crawl. The more serious fault is the soft-404 redirect: sending a removed product to the homepage tells the search engine the homepage is the answer to that query, so the redirect is treated as a 404 anyway and the equity is discarded. Chains also mask migration errors — a three-hop chain usually means three separate migrations were each done half-way.

How we inspect it

We follow every 3xx response to its terminus, recording hop count, status code at each hop, protocol and host changes, and whether the final destination is a relevant page or a generic homepage/category fallback (a soft-404 redirect).

Source: Redirect graph traversal from the inspection crawl

Remedy

  1. 1Export the full redirect graph and collapse every chain so each source points directly at its final destination.
  2. 2Update internal links, canonical tags, sitemap entries and hreflang annotations to reference the final URL, not the redirecting one.
  3. 3Replace redirects-to-homepage with either a 410 Gone, or a redirect to the genuinely equivalent page.
  4. 4Consolidate protocol and host normalisation (http→https, www→apex) into one edge rule so it never stacks with an application-level redirect.
  5. 5Retain migration redirects for at least twelve months before pruning.

Verification

Recheck retraverses the redirect graph and confirms maximum hop count of one and zero internal links to 3xx URLs.

Usually fixed by
Developer
Effort
Medium · 2–8 hrs
Est. impact
medium
Typical cost
NZ$240–600
CR-05Minor failweight 2measured

XML sitemap is valid, current and clean

Does your sitemap list only canonical, indexable, HTTP 200 URLs?

Pass, advisory and fail criteria for CR-05
PassValid XML, under 50,000 URLs and 50 MB uncompressed per file, referenced from robots.txt, and every URL is a self-canonical indexable 200.
AdvisoryValid and clean but lastmod values are absent or clearly untrustworthy (e.g. all identical to the build date).
FailInvalid XML, contains non-canonical, redirecting, 4xx or noindex URLs, or is missing entirely on a site above 100 pages.

Why it matters

A sitemap is a machine-readable declaration of what you consider canonical and current. When it lists redirects, dead URLs or noindex pages, it contradicts your other signals and the whole file loses credibility as a discovery source. Accurate lastmod values are the strongest lever you have over recrawl scheduling, but only if they are genuinely maintained — fabricated timestamps are quickly discounted.

How we inspect it

We fetch the sitemap index and every child sitemap, validate against the sitemaps.org schema, check file and URL count limits, and cross-reference every listed URL against its live status code, canonical target and index directive.

Source: Sitemap fetch, XSD validation, cross-reference against crawl

Remedy

  1. 1Generate the sitemap from the same source of truth that determines canonical status, so the two can never diverge.
  2. 2Exclude any URL that is noindex, redirected, parameterised, paginated beyond page one, or non-canonical.
  3. 3Split by content type into a sitemap index — it makes index-coverage debugging in Search Console dramatically faster.
  4. 4Emit lastmod only when content genuinely changed, in W3C datetime format.
  5. 5Reference the sitemap index from robots.txt and submit it in Search Console.

Verification

Recheck refetches and revalidates every child sitemap and confirms 100% of listed URLs are self-canonical indexable 200s.

Usually fixed by
Developer
Effort
Small · 0.5–2 hrs
Est. impact
medium
Typical cost
NZ$120–320
CR-06Minor failweight 2derived

Crawl allocation is not consumed by low-value URLs

Are parameter, faceted or session URLs consuming your crawl allocation?

Pass, advisory and fail criteria for CR-06
PassUnder 15% of crawlable URLs are parameter or facet variants, and these are excluded from discovery.
Advisory15–30% of the crawlable surface is combinatorial but correctly canonicalised.
FailOver 30% of the crawlable surface is combinatorial, or facet combinations are linkable and uncontrolled.

Why it matters

Crawl allocation is finite and roughly proportional to a site's standing. Faceted navigation can generate millions of URL permutations from a few hundred products, and if those are discoverable the crawler will spend its allocation enumerating filter combinations instead of finding your new content. The symptom is deceptive: new pages take weeks to be indexed while the crawl stats report high activity.

How we inspect it

We cluster discovered URLs by parameter signature and path pattern, then compute what proportion of the crawlable surface is generated combinatorially rather than editorially. Where server logs are supplied we compare that proportion against actual crawler request distribution.

Source: Crawl URL-pattern clustering + server log sample where supplied

Remedy

  1. 1Enumerate every parameter the application accepts and classify each as content-defining, filtering, tracking or session.
  2. 2Render non-content-defining facet links without crawlable hrefs, or route them through a POST or fragment.
  3. 3Canonicalise every permitted variant to the unfiltered parent.
  4. 4Disallow tracking and session parameters in robots.txt once they are already non-canonical.
  5. 5Choose deliberately which high-demand facet combinations deserve indexable landing pages, and make only those linkable.

Verification

Recheck re-clusters the discovered URL set and confirms combinatorial URLs are under threshold and non-discoverable.

Usually fixed by
Developer
Effort
Large · 1–3 days
Est. impact
medium
Typical cost
NZ$600–2,400
CR-07Major failweight 1measured

HTTPS enforced and host variants canonicalised

Do http, https, www and apex variants all resolve to one canonical host?

Pass, advisory and fail criteria for CR-07
PassAll variants 301 to a single canonical https host in one hop; certificate valid, trusted and covering all served hostnames; HSTS present.
AdvisoryCanonicalisation correct but HSTS absent, or certificate expiring within 30 days.
FailAny variant serves HTTP 200 content, mixed content is present, or the certificate is invalid, expired or hostname-mismatched.

Why it matters

Four addressable variants of the same site is four copies competing with each other and splitting inbound link equity between them. It is the oldest fault in technical SEO and it still appears constantly after platform migrations, CDN changes and certificate renewals. An invalid certificate additionally triggers a browser interstitial that ends the session before the page is ever seen.

How we inspect it

We request the homepage and three deep URLs across http/https and www/apex, record the redirect behaviour of each, and inspect the TLS certificate chain, expiry and hostname coverage.

Source: Direct requests to all four host/protocol permutations + certificate inspection

Remedy

  1. 1Choose one canonical host and enforce it with a single edge-level 301 covering every other permutation.
  2. 2Ensure the redirect happens in one hop — never http→https then www→apex as two separate rules.
  3. 3Fix mixed content by serving all subresources over https with protocol-relative or absolute https URLs.
  4. 4Enable HSTS with a sensible max-age once you are confident https is stable everywhere.
  5. 5Set certificate expiry monitoring with alerting at 30 days.

Verification

Recheck re-requests all host and protocol permutations and confirms single-hop convergence with a valid certificate.

Usually fixed by
Developer
Effort
Small · 0.5–2 hrs
Est. impact
high
Typical cost
NZ$120–320