Implementation worksheet · 5 min read
A Preview-Versus-Production Indexing Policy for App Builders
Send `X-Robots-Tag: noindex, nofollow` on every response from any hostname that is not production, driven by an environment check rather than a file someone must remember to edit — and **allow crawling of those hosts in robots.txt**, because a crawler that obeys a disallow never fetches the page and therefore never sees the noindex. Disallow plus noindex is the combination that fails: the crawler honours the block and can still index the URL it was told about from a link. If you want the content genuinely unreachable, authenticate it — that is an access-control decision, and then robots.txt does not matter. Add three supports: canonical tags on preview pages pointing at production, no sitemap served from preview hostnames, and a launch check confirming production does not carry the same noindex. That last one is the failure that costs the most.
Every modern build platform gives each branch a public URL. They are convenient, they are crawlable, and they contain either duplicates of your production content or work you have not announced.
Put it into practice
1. Detect environment in code, not configuration
One check — deployment context or hostname — that decides whether the noindex header is sent. Anything depending on a person remembering to edit a file will eventually ship wrong in one direction or the other.
2. Allow crawling where you rely on the header
This is the step most policies get backwards. `X-Robots-Tag: noindex` only works if a crawler fetches the response and reads it, so a public preview host must **not** be disallowed in robots.txt. Disallow prevents crawling, not indexing: a disallowed URL linked from anywhere can still appear in results, described only by its anchor text. Google documents the prerequisite directly — a noindex rule cannot be honoured on a page the crawler was told not to fetch.
3. Use authentication when the content must not be reachable at all
Headers are an indexing control, not an access control. If a preview contains unreleased work that genuinely must not be readable, put it behind a login. Then disallow-all in robots.txt is fine, because you are no longer relying on a header being read. Pick one posture per host and write it in the table rather than mixing them.
4. Point preview canonicals at production
If a preview page is fetched, its canonical should name the production URL. This is a second line of defence, not a replacement for the header.
5. Do not serve a sitemap from preview
A preview sitemap actively advertises the duplicate set. Gate sitemap generation on the same environment check.
6. Verify production is NOT noindexed at every launch
The inverse failure is worse than the one you are preventing: the environment check misfires, production ships noindex, and the site quietly leaves the index. One curl of the production headers per deploy catches it.
Environment policy
Copy this structure into your review document and record your observed result for each row.
| Environment | X-Robots-Tag | robots.txt | Sitemap | Canonical |
|---|---|---|---|---|
| Production | none | allow | served | self |
| Preview / branch (public) | noindex, nofollow | allow — so the header can be read | not served | production URL |
| Staging (public) | noindex, nofollow | allow — so the header can be read | not served | production URL |
| Preview / staging (authenticated) | n/a — not reachable | disallow all | not served | n/a |
| Local | n/a | n/a | n/a | n/a |
A failure worth checking
The self-cancelling policy: disallow-all in robots.txt plus `X-Robots-Tag: noindex` on the same public preview host. It reads as belt and braces and is neither. The crawler obeys the disallow, never fetches the response, never reads the noindex — and then indexes the URL anyway because a ticket or a Slack unfurl linked to it, showing it with no description. Two of your URLs now compete for the same query and the wrong one sometimes wins, with unreleased copy on it. Either allow crawling so the header is read, or authenticate the host so there is nothing to index.
Common questions
Is password-protecting previews better?
Yes, where previews contain unreleased work — it is the only option that makes the content genuinely unreachable rather than merely unlisted. It also simplifies the robots question: an authenticated host can be disallowed without consequence, because there is no header you need a crawler to read.
What about AI crawlers on preview URLs?
Same header, same effect — `noindex` is honoured by the major AI crawlers that respect robots directives. The wider point stands either way: a public URL is public, and the only reliable control is not serving the content unauthenticated.
Why not disallow and noindex together?
Because they cancel. The disallow stops the fetch, so the noindex is never read, and a linked URL can still be indexed with no description. Choose per host: allow plus noindex for public previews, or authentication plus disallow for private ones.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.