Implementation worksheet · 6 min read
A Pricing Experiment Specification for an AI-Built App
Specify seven things before changing a price, starting with the randomisation unit — account or company for B2B, person for B2C — because that one choice decides whether two colleagues can see different numbers. Then: who is eligible (almost always new visitors only — never existing customers mid-term), what each variant actually shows including on every surface where the price appears, how long a shown price is honoured once someone has seen it, the primary metric and the guardrail metrics, the duration and minimum sample, and the grandfathering rule for what happens to test participants when the test ends. That last field is the one teams skip and the one that produces a support queue: someone signed up at a tested price and now needs to be told, in writing, what they will pay at renewal.
A pricing test looks like any other experiment and is not. Feature variants are reversible and invisible to people outside the test; prices are quoted, screenshotted, shared in Slack and compared between colleagues at the same company. Two people on the same team seeing different numbers for the same plan is a trust problem no significance calculation offsets, and it is the default outcome unless you design against it.
Put it into practice
1. Choose the randomisation unit before anything else, and match it to who compares prices
Existing customers keep their price, full stop — changing what someone pays mid-term to satisfy a test is not an experiment, it is a contract change. For everyone else the first decision is the unit you randomise on. **B2B: randomise by account or company domain**, not by person. Per-person assignment is stable for each individual and still puts two colleagues at the same company in opposite variants, because they are two different people — that is the failure, not a recognition problem. Keep every member of an account on the same variant and analyse at the account level. **B2C: person-level assignment** is the right unit, since the people who compare prices are not systematically grouped. Either way, assign deterministically from a stable identifier so the variant does not change between visits or devices.
2. Enumerate every surface the price appears on
Pricing page, in-app upgrade prompts, checkout, plan comparison, marketing site, emails, the FAQ, help documentation and any PDF quote. A variant that updates the pricing page but not the checkout produces a mismatch at the moment of payment, which is the single worst place to be inconsistent. List the surfaces, then check each one in the variant.
3. Decide how long a shown price is honoured
Someone sees $29, leaves, and returns a week later. Do they still get $29? Write the answer down — a session, thirty days, or indefinitely for anyone who reached checkout — and make sure support can look it up. Without this, the answer becomes whatever the agent on the ticket decides, which is neither consistent nor defensible.
4. Name one primary metric and at least two guardrails
Primary is usually revenue per visitor rather than conversion rate, because a lower price converts better and earns less. Guardrails: refund rate, support contacts about pricing, and downgrade or cancellation rate within the first period. A pricing test that wins on conversion while tripling refunds has not won.
5. Set duration and minimum sample before you start
Pricing effects are slow — the interesting outcomes are renewal and refund, which arrive a billing period later. Decide the end date in advance and resist reading the result early, because early reads on pricing are how teams ship a price change on noise and discover it at the next renewal cycle.
6. Write the grandfathering rule before launch, not after
When the test ends, what happens to people who bought at the losing price? Honour it indefinitely, honour it for a term, or migrate with notice. Any of the three is defensible; deciding after the fact is not, because by then you are making a policy under pressure from whichever customer complained loudest.
The pricing experiment specification
Copy this structure into your review document and record your observed result for each row.
| Field | Your answer | Checked |
|---|---|---|
| Eligible population | new visitors only | |
| Assignment key | stable identifier for that unit | |
| Randomisation unit | account/company (B2B) or person (B2C) | |
| Surfaces updated | full list, each verified | |
| Price-honouring window | ||
| Support can look up a shown price | yes/no | |
| Primary metric | revenue per visitor | |
| Guardrail: refund rate | ||
| Guardrail: pricing support contacts | ||
| Guardrail: early cancellation | ||
| Duration and minimum sample | decided before start | |
| Grandfathering rule | written before launch | |
| Who approves ending the test | named |
A failure worth checking
Two colleagues, one company, two prices. They compare screenshots, conclude the pricing is arbitrary or that one of them is being overcharged, and raise it publicly. No statistical result is worth that. The trap is that person-level assignment looks like the fix and is not: Alice and Bob are two distinct people, so a perfectly stable per-person hash still places them in opposite variants. Only randomising at the account or company level keeps a buying group consistent, which is why the unit is the first row of the specification rather than an implementation detail.
Common questions
Is it legal to show different prices to different people?
It varies by jurisdiction and by what the difference is based on, and this is a question for someone qualified rather than a field guide. What is clear regardless: publishing a price and then not honouring it for someone who saw it causes problems in every jurisdiction, which is why the honouring window is on the specification.
How long does a pricing test need to run?
Longer than a feature test, because the outcomes that matter — renewal, refund, expansion — arrive a billing period later. A monthly plan needs at least two cycles before the result means anything. Reading conversion at week one measures curiosity, not willingness to pay.
Person-level or account-level assignment?
Account or company for B2B, person for B2C. The question to ask is who might compare prices with each other: if that is colleagues at one company, the randomisation unit has to be the company, and analysis has to happen at that unit too. Randomising by account and analysing by person understates the uncertainty, because people within an account are not independent observations.
Basis and scope
This is a proposed implementation method using illustrative examples, not a measured benchmark or a customer case study. Prepared with AI assistance. Validate product-specific behavior against current documentation and your own test environment.