Experiment Results Guardrails
Say whether a test can be trusted yet, before anyone declares a winner.
What it adds
A results view that reports sample sufficiency, exposure quality, and effect size alongside the numbers — and refuses to crown a winner early.
What your agent is told to do
5
What your agent is told to do
5-
1
Compute results from exposure events — users who actually saw the variant — not from assignment counts. Assigned-but-never-exposed users dilute every rate.
-
2
Fix the primary metric and the stopping rule when the experiment is created, and version them. A metric changed after the data arrives is not a result.
-
3
Show a confidence interval and the absolute effect next to every relative lift. A 40% improvement on a 0.1% baseline is noise dressed as a win.
-
4
Compute the sample size needed for the stated minimum detectable effect, and show progress towards it prominently.
-
5
Do NOT display a winner, a green badge, or a significance verdict before the stopping rule is met. Show the guardrail instead and say why.
Edge cases it handles
6
Edge cases it handles
6- Sample ratio mismatch — buckets arriving at meaningfully different sizes — means assignment is broken. Detect it and invalidate the readout rather than interpreting the numbers.
- Repeated checking inflates false positives. Either record how many times results were viewed and warn, or use a method that tolerates peeking.
- The current day and the first hours of an experiment are partial. Exclude or mark them rather than letting them swing the totals.
- Weekday and weekend traffic behave differently. Warn when an experiment has not covered whole weekly cycles.
- Testing many secondary metrics will find something significant by chance. Label secondary metrics as exploratory and adjust or say you have not.
- A guardrail metric moving the wrong way — errors, latency, refunds — must be surfaced even when the primary metric wins.
Definition of done
9
Definition of done
9- Rates are computed from exposure events, not assignment records.
- The primary metric and stopping rule are fixed at creation and versioned on change.
- Required sample size and progress towards it are shown on the results view.
- Absolute effect and a confidence interval accompany every relative lift.
- Sample ratio mismatch is detected and blocks the readout.
- No winner is declared before the stopping rule is satisfied.
- Guardrail metrics are shown regardless of the primary result.
- The feature matches the existing design system.
- No existing functionality is broken.
Related features
UI Event Naming Schema
UI Event Naming Schema
Give every tracked interaction a name that still means the same thing in a year.
What it does
A documented vocabulary and structure for event names and properties, applied across the app's existing tracking.
How it works
- 1 Audit the events the app already sends and list them. The duplicates, the near-synonyms, and the ones nobody can explain are the reason the schema is needed, and they are also the migration list.
- 2 Fix a structure of object, action, and context: what was acted on, what happened to it, and where. Fix a closed set of actions — viewed, clicked, opened, submitted, selected, failed — and require every new event to reuse one of them.
- 3 Keep variable detail in properties, never in the name. A record identifier, a plan name, or a workspace slug baked into an event name produces thousands of unaggregatable one-off events.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/ui-event-naming-schema
Amplitude Event Tracking
Amplitude Event Tracking
Send behavioural events to Amplitude with stable users, devices, and account grouping.
What it does
A server-side pipeline that forwards behavioural events, user properties, and account-level groupings to Amplitude.
How it works
- 1 Identify the behaviours worth measuring and emit them from one place in the application layer, so that the set of events is readable in a single file rather than inferred by grepping the codebase.
- 2 Establish the identity model before writing any send code: what identifies a signed-in user, what identifies a device or anonymous session, and how the two are stitched at sign-up and sign-in.
- 3 Map the app's own tenancy — workspace, organisation, or team — onto Amplitude's group concept, and derive it from the record the event belongs to rather than from whatever context the request happened to carry.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/amplitude-event-tracking
Component Analytics Hook
Component Analytics Hook
Instrument shared components once instead of hand-writing events on every screen.
What it does
Interaction tracking attached to the app's shared components, emitting named events carrying the context of where they were used.
How it works
- 1 Identify the shared components that account for most interaction — buttons, links, menus, dialogs, tabs, table rows, form submits — and attach tracking inside them rather than at each of the hundreds of call sites.
- 2 Have each component emit its own identity and the context it was rendered in: the screen, the surface, and a stable identifier supplied by the caller. Position on the page and visible label are not identity.
- 3 Emit exactly one event per user action. A menu item inside a card inside a table row must not produce three overlapping records of the same click.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/component-analytics-hook
How it works
-
1
Copy the link
Grab the Markdown instruction URL for this feature.
-
2
Give it to your AI
Paste it into Claude Code, Cursor, v0, Lovable — whatever you build with.
-
3
It inspects, then implements
Your agent reads your existing app first, then adds the feature to fit it.
Works with your stack
These instructions are written to adapt. They tell the agent to detect your framework, match your existing design system, and reuse what you already have — rather than assuming a particular stack.
Need it tighter than that? Customize the feature and tell it exactly what you're running.