AV

The problem: experiments that never end

Teams say they want to “run more experiments.” What they often mean is: “Let’s start more things.” The hard part is not starting. The hard part is stopping—especially when the work looks almost promising, or when someone senior sponsored it, or when the team already invested time.

That’s how you get zombie experiments: they keep consuming attention, pulling meetings onto calendars, and stealing cycles from work that’s already proven. They don’t produce learning fast, and they don’t ship outcomes. They just stay alive.

A simple kill criteria sheet solves a specific failure: the absence of an agreed stopping point.

What “kill criteria” means

Kill criteria are the pre-committed conditions under which you will stop an experiment and reallocate resources.

Think of it like this: in many fields, people define stopping rules before they start—because continuing after certain signals is wasteful or risky. In clinical trials, for example, sponsors plan interim analyses and may stop early for futility (it’s unlikely to meet the endpoint) or for safety/benefit reasons. The FDA discusses this logic in guidance around trial monitoring and interim analyses.

Your product, growth, or ops experiment is not a clinical trial—but the principle transfers: decide what would make continuing irrational before you fall in love with the work.

Kill criteria vs. guardrails vs. success criteria

These three get mixed up. Keep them separate:

• Success criteria (scale): “If we hit X, we roll out / invest more.”
• Kill criteria (stop): “If we see Y, we end it.”
• Guardrails (protect): “Even if X looks good, we stop if Z gets worse.”

Guardrails matter because an experiment can “win” on a local metric while harming the system (revenue up, churn up; signups up, support tickets up). In online controlled experiments, practitioners emphasize trustworthy measurement and guardrails to avoid misleading wins.

Why teams avoid killing (and why it’s predictable)

Two forces show up repeatedly:

1. Sunk cost bias: People keep investing because they already invested. Behavioral economics describes this as the sunk cost fallacy—continuing an endeavor because of resources already spent.
2. Initiative overload: Organizations layer on projects faster than they remove them. HBR has written about how hard it is to kill existing initiatives—and how overload spreads below leadership.

A kill criteria sheet doesn’t make you “more disciplined” through willpower. It makes stopping the default behavior by design.

Why now: experimentation is cheaper, attention is not

Tools made experimentation cheap: feature flags, no-code landing pages, email platforms, rapid prototyping. But cheap starts create expensive portfolios.

The scarcest resource is not “ideas.” It’s focused time. Economists use opportunity cost to describe the value of the best alternative you give up when you choose something else.

Every week a zombie experiment stays alive, it quietly taxes your best alternative: fixing onboarding, improving reliability, closing deals, shipping the next release.

Kill criteria are how you pay that opportunity cost consciously, not accidentally.

How the kill criteria sheet works

You don’t need a complex framework. You need one page that forces clarity.

Here’s the template I recommend. You can copy this into a doc, a spreadsheet, or your experiment tracker.

The one-page template (fields that matter)

Field	What you write	Why it matters
Experiment name	Short, specific	Avoids vague “test stuff” projects
Hypothesis	“If we do A, then B because C”	Makes learning explicit
Primary metric	One outcome metric	Stops metric shopping
Guardrails	1–3 “must not worsen” metrics	Prevents local wins
Minimum meaningful effect	“Worth it if ≥ X”	Defines “signal” vs noise
Budget cap	Time + $ + people	Limits blast radius
Run length	Start date → decision date	Ensures an ending
Kill criteria	2–3 conditions to stop	Prevents zombie drift
Pivot criteria	When to change approach	Adds a middle path
Success criteria	When to scale	Prevents “analysis forever”
Owner + approver	Names	Ensures accountability
Next step	Kill / Pivot / Scale	Forces a decision

If you want one small addition that pays off: add a “Premortem risks” box. A premortem is a structured technique where you imagine the project failed and list plausible reasons—so dissent and concerns surface early.

The three checkpoints: start, mid-run, end

Checkpoint 1 (Start):

• Confirm instrumentation works.
• Confirm the decision date is on the calendar.
• Confirm owner + approver.

Checkpoint 2 (Mid-run):

• Check guardrails first (protect).
• Check whether the kill criteria are already met (stop early if yes).
• Only then look at the primary metric (learn).

Checkpoint 3 (End):

• Decide: kill, pivot, or scale.
• Write one paragraph: “What did we learn?”
• Close the loop: update the playbook, pricing, onboarding, or roadmap.

A worked example (with simple numbers)

Below are three examples. The numbers are illustrative—use what fits your business.

Example 1: Marketing test (landing page)

• Hypothesis: If we position the offer around “resource allocation,” then qualified leads increase because visitors self-select faster.
• Primary metric: Qualified lead rate (QLR)
• Guardrails: Cost per lead must not rise > 20%; spam lead rate must not rise > 10%
• Budget cap: $800 ads + 12 hours design/copy
• Run length: 10 days

Kill criteria (pre-commit):

1. If QLR is flat or down after 300 clicks, stop.
2. If spam lead rate rises >10%, stop.
3. If tracking is broken for >24 hours, pause and restart (don’t “guess”).

Success criteria: QLR up ≥ 25% with stable guardrails → scale spend and test variant copy.

Why this works: it defines enough data to decide, and it protects you from “we got leads” when they’re not qualified.

Example 2: Product/UX experiment (onboarding step)

• Hypothesis: Removing one onboarding step increases activation because fewer users drop before value.
• Primary metric: Activation rate (defined event within 24 hours)
• Guardrails: Refund requests must not rise; support tickets per user must not rise > 15%
• Budget cap: 2 engineer-days + 1 designer-day
• Run length: 14 days

Kill criteria:

1. If activation doesn’t improve by your minimum meaningful effect (say ≥ 3 percentage points) by day 14, kill.
2. If support tickets rise > 15% for 3 consecutive days, kill or revert.
3. If a key cohort (e.g., paid users) performs worse, revert for that cohort.

Success criteria: Activation improves and guardrails stay stable → roll out to 100%.

This aligns with the emphasis in experimentation practice: define metrics and guardrails up front so you don’t “win” by breaking the experience.

Example 3: Ops/process experiment (handoff change)

• Hypothesis: A structured handoff checklist reduces cycle time because fewer items bounce back.
• Primary metric: Cycle time (request → completion)
• Guardrails: Rework rate must not rise; customer satisfaction must not drop
• Budget cap: 1 week trial in one team
• Run length: 4 weeks (process signals take longer)

Kill criteria:

1. If cycle time doesn’t improve by week 4, kill.
2. If rework rate rises > X (you define), kill.
3. If adoption stays below 60% by week 2, pause and fix the workflow—don’t blame results.

This keeps you honest about change management: if the team can’t adopt it, the “result” is not interpretable.

CTA (mid): If you want these fields pre-formatted, download the kill criteria sheet and use it as your default experiment intake.

Trade-offs and failure modes

A kill sheet is a tool. Like any tool, you can misuse it.

Killing too early

Some experiments need time because effects lag (retention, trust, long sales cycles). If you force a 7-day decision on a 90-day effect, you’ll kill good ideas.

Fix: make the run length match the feedback loop. If signals are slow, define leading indicators that correlate with the outcome, and treat them as provisional.

Gaming metrics

If teams fear killing, they will optimize the metric rather than the outcome. That’s not a measurement problem; it’s an incentive problem.

Fix: require a one-paragraph “what we changed” and “what we learned,” and review it. Use the premortem to surface how the metric could lie.

Measuring the wrong thing

If your primary metric is fuzzy (“engagement”), you will debate forever.

Fix: pick one crisp metric and define it operationally. The book Trustworthy Online Controlled Experiments repeatedly stresses the importance of clear metrics and interpretation discipline in experimentation programs.

What to do next

Here’s the rollout that works without ceremony.

1. Make the sheet mandatory for new experiments. No sheet, no start.
2. Set defaults: budget cap (time/$), 1–2 guardrails, a decision date, and 2 kill criteria.
3. Hold a 15-minute “decision review” weekly: kill/pivot/scale only. No redesign debates.

Safety/limitations (and a quick disclaimer)

• Not financial advice: use your own judgment on budgets and thresholds.
• Context matters: regulated domains, safety-critical systems, and medical claims require formal methods and compliance review.
• Correlation risk: leading indicators can mislead; document uncertainty and rerun tests when stakes rise.
• Human impact: “kill” should apply to experiments, not people—be explicit about that.

If you want a related read on making hard calls under constraints, see: The Founder’s Anti-Framework: When NOT to Scale and What to Measure in Week 1 vs Week 8.

Sources (external, footnoted):
[1] HBR on initiative overload and killing projects.
[2] Premortems for surfacing risks early (Klein; Kahneman).
[3] Trustworthy experimentation discipline (Kohavi et al.).
[4] Opportunity cost as a decision lens.
[5] Sunk cost fallacy and why teams persist.
[6] Predefined stopping logic in formal experiments (FDA guidance).

FAQ

1) What are kill criteria in experimentation?
Kill criteria are pre-defined conditions that end an experiment when results are unlikely to be worth further time or risk. They reduce indecision and prevent sunk-cost persistence.

2) How do kill criteria differ from success criteria?
Success criteria define when you scale; kill criteria define when you stop; guardrails define what must not break while you test. This separation is a core discipline in trustworthy experimentation.

3) How many kill criteria should an experiment have?
Usually 2–3: one tied to the primary metric (no meaningful movement), one tied to a guardrail (harm), and one tied to execution quality (instrumentation/adoption failure).

4) When should you decide to kill an experiment?
Set a decision date aligned to the feedback loop. Fast loops (ads, UX) can be days to weeks; slow loops (retention, ops change) may need longer plus leading indicators.

5) How do you avoid killing experiments too early?
Use a minimum meaningful effect, ensure instrumentation works, and choose a run length that matches the metric’s natural lag. Add a premortem to surface hidden risks and false signals.