Implementation worksheet · 5 min read
A Crawlability Acceptance Test for an AI-Built Marketing Site
Run eight checks before launch: fetch the page with JavaScript disabled and confirm the copy is in the HTML; confirm one canonical tag per page pointing at the URL you want indexed; confirm robots.txt does not block what you want crawled and that no page carries an accidental noindex; confirm the sitemap exists, lists only 200-returning canonical URLs, and is referenced from robots.txt; confirm every internal link resolves without a redirect chain; confirm titles and descriptions are unique per page; confirm the site answers on one hostname with the others redirecting; and confirm a 404 returns 404 rather than 200 with a friendly page. Generated sites fail the first and the last most often, and both are invisible in a browser.
An AI-built site looks correct because a human loads it in a browser with JavaScript enabled. Crawlers are less forgiving and quieter about it: nothing errors, the pages simply never rank, and the cause is usually decided in the first hour of the build.
Put it into practice
1. Fetch without JavaScript
`curl -s <url> | grep` for a sentence you can see on screen. If it is missing, your copy lives in client-side rendering and you are relying on a rendering budget you do not control. This is the single highest-value check on the list.
2. Check canonical and robots directives per template
One canonical per page, absolute, self-referencing unless you mean otherwise. Then grep the built output for `noindex` — a staging directive left in a shared layout is the classic way to launch invisible.
3. Validate the sitemap against reality
Every URL in it returns 200 and is canonical; nothing redirected, 404ing or noindexed appears. A sitemap listing redirects teaches crawlers to distrust it, and it is trivially checkable with a loop.
4. Walk the internal links
Every link resolves in one hop. Redirect chains inside your own site are pure waste, and generated sites accumulate them when a route is renamed and old links are left behind.
5. Confirm one hostname and a real 404
www and apex must not both serve 200. A missing page must return status 404, not a styled page with status 200 — soft 404s let dead URLs accumulate in the index indefinitely.
Pre-launch crawl checks
Copy this structure into your review document and record your observed result for each row.
| Check | How | Pass criteria | Verified |
|---|---|---|---|
| Copy in HTML | curl + grep | visible text present | |
| Canonical | view source | one, absolute, correct | |
| No stray noindex | grep built output | none unintended | |
| Sitemap accurate | loop over URLs | all 200 + canonical | |
| Internal links | crawl | no redirect chains | |
| Unique titles/descriptions | crawl | no duplicates | |
| Single hostname | curl both | one 200, one 301 | |
| Real 404 | curl a missing path | status 404 |
A failure worth checking
The invisible launch: the site looks perfect, the team celebrates, and six weeks later nothing is indexed because a `noindex` from the staging template shipped in the shared layout. Nothing broke, no error was raised, and the only symptom was absence — which is why this belongs in a launch checklist rather than in monitoring.
Common questions
Do I need a crawler tool for this?
No. All eight checks are curl, grep and view-source. A crawler makes repeat runs easier on a large site, but the first pass on a marketing site takes about twenty minutes by hand.
What about AI crawlers specifically?
Same fundamentals, plus confirming your robots policy does not block the assistants you want citing you. Retrievability is upstream of everything else — a page no crawler can fetch cannot be cited by any of them.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.