Product & Systems

On Experimentation: Key Ideas

Premise

Centralize the Method.
Decentralize the Judgment.

Every experimentation platform must balance authority against agility. These ideas isolate the forces, systems, and interfaces that let evidence-driven decisions scale.

The Tension

Every platform is choosing how much to control.

Control too little and trust collapses into ad-hoc analysis. Control too much and the platform becomes a review queue staffed by people without local product context.

  • Control establishes standards, consistent checks, and certifies evidence.
  • Trust is the reason to own any layer in the first place.
  • Enablement is the counterweight: product teams must retain the speed to innovate.
Drag the slider between Trust and Enablement.
CONTROL TRUST ENABLE PLATFORM OPERATING REGIME

Increasing control pulls the operating point up, choking enablement.

The Principle

Centralize the method. Decentralize the judgment.

The platform owns methodology to standardize truth, but leaves product decisions to the teams who carry the context and own the outcomes.

  • Methodology includes randomization, significance filters, and calculations.
  • Judgment evaluates tradeoffs, feature complexity, and strategic alignment.
  • Context does not scale or centralize. Evidence can.
Pick a regime. See who owns what.
CENTRALIZED DECENTRALIZED Method Stats, Engine, Holdouts Judgment Taste, Context, Decisions

Evidence requires uniform machinery. Decisions require local taste.

Controlled Layers

Own the layers where being trusted matters more than being flexible.

A shared language for truth requires standardizing the calculations beneath it. You control the layers that certify results, and stay out of the way elsewhere.

  • All teams share the same statistical assumptions and metrics infrastructure.
  • Randomization units must be rigid to prevent experimental leakages.
  • Certification turns raw metrics into trusted organizational currency.
Hover a layer. See why the platform must own it.
Certified Experiment Result 4. Automated Guardrail Checks 3. Statistical Significance Engine 2. Metric Aggregation Engine 1. Randomization & Assignment

Each layer provides stability to the metrics resting on top.

The Surface

The Experiment UI is a trust boundary.

The interface is where methodology is enforced. It steers developers from config errors, checks statistical validity, and stops flawed tests from looking authoritative.

  • Pre-configured guardrails cannot be dropped to make a bad test look green.
  • The UI warns users when a proposed setup lacks statistical power.
  • It hides results entirely when core data sanity checks fail.
Idle
Set SRM and Power, then run the pipeline.
1. Setup Config 2. Checks Validation 3. Readout Reporting

The UI is method in screen form; an SRM check blocks bad data.

The System

Holdouts measure what isolated wins cannot.

Twenty teams can each claim a win while the top metric stays flat. An org-level holdout is the clean baseline that measures the cumulative truth.

  • Shields a small user population (e.g. 1-2%) from all changes for months.
  • The baseline reveals features that cannibalize one another or have transient gains.
  • Only the platform can run this. No single feature team has the context or isolation capability.
Comparison Mode:
Hover the points for local wins. Show the cumulative gap for the real total.
HOLDOUT BASELINE (1%) SHIPPED USERS (99%) MONTHS OF SHIPPING TOP-LINE VALUE

The gap is the real gain, net of cannibalization.

The Structure

Representatives keep judgment local without losing the method.

The platform team lacks the bandwidth and context to sign off on every feature. Embedding trained reps inside product teams keeps judgment local.

  • Representatives are product team data scientists or developers trained by the platform.
  • They review complex experimental setups prior to launch.
  • They prevent product teams from queuing up to wait for platform support.
Pick a node. See how it localizes the platform.
Platform METHOD RIDER REP DRIVER REP FREIGHT REP DELIVERY REP

Embedded experts prevent the platform from acting as a centralized bottleneck.

The Future

When agents run experiments, trust moves into the protocol.

Visual interfaces build trust through prompts, confirmations, and warnings. With agents, trust must move from the page to the API contract itself.

  • Agents configure and read experiments programmatically through backend APIs.
  • The validation checks cannot live as UI alerts—they must be embedded in the query logic.
  • Cryptographically signed results become the basis of trust, not a dashboard layout.
Toggle the mode. Human: checks on screen. Agent: checks in the payload.
Dashboard UI SRM Checked HUMAN READS SCREEN GET /api/v2/result SRM-Audit-Token: Ok { "srm_status": 200 } AGENT VERIFIES PROTOCOL

When code decides, the checks must ride inside the payload.

Operating Model

The method is universal. The application is local.

Standardizing truth requires keeping the calculations centralized. Adapting features requires keeping the judgment local. A balanced platform integrates both regimes.

  • The central methodology maintains statistical reliability.
  • The local teams evaluate business viability.
  • The platform supports the evidence, the team authors the release.
Hover a segment. See where method ends and judgment begins.
CENTRALISE DECENTRALISE METHOD JUDGMENT OWN TRUST CHAOS (Avoid) QUEUE (Avoid)

The ideal regime: own the method centrally, trust local judgment.

Conclusion

Standardize the Rail.
Protect the Agility.

Experimentation is the scientific method applied to product features. Standardize the statistical rails to keep data valid. Beyond that, step back, trust the developers, and keep the org moving.

01 / 10