Experiment Results Guardrails
Say whether a test can be trusted yet, before anyone declares a winner.
What it adds
A results view that reports sample sufficiency, exposure quality, and effect size alongside the numbers — and refuses to crown a winner early.
What your agent is told to do
5
What your agent is told to do
5-
1
Compute results from exposure events — users who actually saw the variant — not from assignment counts. Assigned-but-never-exposed users dilute every rate.
-
2
Fix the primary metric and the stopping rule when the experiment is created, and version them. A metric changed after the data arrives is not a result.
-
3
Show a confidence interval and the absolute effect next to every relative lift. A 40% improvement on a 0.1% baseline is noise dressed as a win.
-
4
Compute the sample size needed for the stated minimum detectable effect, and show progress towards it prominently.
-
5
Do NOT display a winner, a green badge, or a significance verdict before the stopping rule is met. Show the guardrail instead and say why.
Edge cases it handles
6
Edge cases it handles
6- Sample ratio mismatch — buckets arriving at meaningfully different sizes — means assignment is broken. Detect it and invalidate the readout rather than interpreting the numbers.
- Repeated checking inflates false positives. Either record how many times results were viewed and warn, or use a method that tolerates peeking.
- The current day and the first hours of an experiment are partial. Exclude or mark them rather than letting them swing the totals.
- Weekday and weekend traffic behave differently. Warn when an experiment has not covered whole weekly cycles.
- Testing many secondary metrics will find something significant by chance. Label secondary metrics as exploratory and adjust or say you have not.
- A guardrail metric moving the wrong way — errors, latency, refunds — must be surfaced even when the primary metric wins.
Definition of done
9
Definition of done
9- Rates are computed from exposure events, not assignment records.
- The primary metric and stopping rule are fixed at creation and versioned on change.
- Required sample size and progress towards it are shown on the results view.
- Absolute effect and a confidence interval accompany every relative lift.
- Sample ratio mismatch is detected and blocks the readout.
- No winner is declared before the stopping rule is satisfied.
- Guardrail metrics are shown regardless of the primary result.
- The feature matches the existing design system.
- No existing functionality is broken.
Related features
UI Event Naming Schema
UI Event Naming Schema
Give every tracked interaction a name that still means the same thing in a year.
What it does
A documented vocabulary and structure for event names and properties, applied across the app's existing tracking.
How it works
- 1 Audit the events the app already sends and list them. The duplicates, the near-synonyms, and the ones nobody can explain are the reason the schema is needed, and they are also the migration list.
- 2 Fix a structure of object, action, and context: what was acted on, what happened to it, and where. Fix a closed set of actions — viewed, clicked, opened, submitted, selected, failed — and require every new event to reuse one of them.
- 3 Keep variable detail in properties, never in the name. A record identifier, a plan name, or a workspace slug baked into an event name produces thousands of unaggregatable one-off events.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/ui-event-naming-schema
Data Table Cell Primitives
Data Table Cell Primitives
Render every kind of table cell the same way, wherever the table happens to be.
What it does
A shared set of cell renderers for numbers, dates, people, statuses, links, and row actions, used by every table in the app.
How it works
- 1 Catalogue the cell types the app's tables already render and reduce them to a small set: number and currency, date and relative time, person or avatar, status or badge, link or identifier, and an actions cell.
- 2 Separate each cell's underlying value from its presentation, so sorting, filtering, grouping, and export operate on the raw value while the user sees the formatted one.
- 3 Align and format numerically consistent cells the same way everywhere — figures right-aligned with tabular figures and a fixed precision per column, dates in the user's locale and time zone, currency with its code where more than one is possible.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/data-table-cell-primitives
Segment Event Forwarding
Segment Event Forwarding
Send one clean stream of product events to Segment and let it fan out downstream.
What it does
A documented event schema and a single forwarding path emitting track, identify, group, and page calls with consistent identity.
How it works
- 1 Write the event schema before writing any code: the exact set of event names, the properties each carries with their types, and the traits attached to a user and to an account. Every downstream tool inherits this schema, so a name chosen carelessly is expensive to change later.
- 2 Establish one identity model and apply it everywhere. Use a stable internal user identifier that never changes, associate the anonymous identifier from the first visit with it at signup, and attach the account or workspace so downstream tools can roll events up by customer.
- 3 Route every event through one internal emitter rather than calling the provider from feature code. That emitter validates the event against the schema, drops or flags anything unrecognised, and is the single place where identity and default properties are attached.
Copy the prompt
No account needed
Add this feature to my app:
https://addthisfeature.com/x/segment-event-forwarding
How it works
-
1
Copy the link
Grab the Markdown instruction URL for this feature.
-
2
Give it to your AI
Paste it into Claude Code, Cursor, v0, Lovable — whatever you build with.
-
3
It inspects, then implements
Your agent reads your existing app first, then adds the feature to fit it.
Works with your stack
These instructions are written to adapt. They tell the agent to detect your framework, match your existing design system, and reuse what you already have — rather than assuming a particular stack.
Need it tighter than that? Customize the feature and tell it exactly what you're running.