Every experimentation platform eventually faces the same question: how much should we control?
Control too little and it's the wild west — ad-hoc analyses, inconsistent standards, leaky holdouts and leaky metrics. Control too much and the platform becomes a bottleneck, a queue of review requests staffed by people who don't have the context to review well. Both failure modes are common. Most platforms oscillate between them, never quite settling.
The answer, I think, is a separation that sounds simple but is surprisingly hard to maintain: centralize the method, decentralize the judgment.
Underneath it sit three forces in tension: control, trust, and enablement.
Control and trust travel together. The reason to own a layer is that you can establish trust on it: if the platform controls how metrics are computed and how users are randomized, a result that rides on that layer can be certified, not merely reported. You control the layers where being trusted matters more than being flexible.
Enablement pulls the other way. The platform also has to let other teams innovate without becoming the blocker that signs off on every move. Each layer you control is a layer someone else cannot change — so you control deliberately, and everywhere else you get out of the way. The whole craft is choosing which layers belong on which side.
Three things make this concrete: the Experiment UI, org-level Holdouts, and Experiment Representatives — a tightly controlled surface, a controlled system, and a deliberately decentralized structure.
The platform owns the methodology: how metrics are computed, how users are randomized, how statistical significance is determined, what a valid holdout looks like. These things should be standardized. A Rider experiment and a Freight experiment should use the same statistical engine, the same variance reduction techniques, the same guardrails. You don't want five teams independently deciding what p-value threshold to use.
But the judgment — should we ship this feature given these results? — has to live with the people who understand the product. The platform can tell you the treatment moved conversion by 0.3% with a confidence interval of [0.1%, 0.5%]. It cannot tell you whether that tradeoff is worth the engineering maintenance cost, whether the metric masks a worse experience for a subset of users, or whether the feature conflicts with something else shipping next quarter. That's context. Context doesn't centralize.
This separation is easy to state and hard to hold. Three things I've seen work well in practice — a surface, a system, and a structure.
The Experiment UI
The most heavily controlled layer is the one people actually touch: the interface where experiments are set up and read. Control here isn't bureaucracy — it's the thing that makes the surface worth trusting.
So the UI controls how an experiment comes into being. It owns the flows and the constraints on setting them up. It nudges toward correct configurations and steers away from the risky and the likely-to-be-wrong: a randomization unit you can't quietly change, a guardrail metric you can't drop, a warning before you launch something underpowered. It refuses to show results when treatment-distribution checks fail, rather than letting someone read a broken test. And it corrects for multiple comparisons on its own, scaling the threshold to the cardinality of the report — how many treatments, metrics, and segments are being looked at at once.
Because the layer is controlled, everything it surfaces can be trusted. A number that reaches the UI has already cleared the checks; it is certified, not merely displayed. That is the entire return on control: the interface becomes something the organization can believe without re-deriving it.
A digression on what's coming. This trust is anchored to the interface — a human reads a vetted screen and believes it. In a post-agentic world that anchor loosens. When agents, not people, set up and read experiments, the UI stops being the place where trust is established; the screen is no longer the chokepoint. The checks have to migrate into the protocol the agents speak. The trusted surface was always a proxy for a trusted method — and agents make us pay that debt directly.
Org-Level Holdouts
Individual experiments are noisy. A team ships a feature, sees a +0.2% metric win, declares victory. Another team does the same. Over a quarter, twenty teams each claim small wins. But when you look at the top-line metric, it hasn't moved — or worse, it's down. The wins were real in isolation; they interfered with each other, or the measurement was subtly wrong, or the gains were transient.
Org-level holdouts are a centralized system that solves this honestly. You maintain a population of users — typically 1-2% — who are shielded from all changes for months at a time. The holdout group becomes a living baseline. You compare the holdout against everyone else, and the gap tells you the cumulative, real impact of everything the organization shipped.
It changes what teams optimize for: not the local win, but whether the features we release compound or cannibalize one another.
Centralization belongs here. No individual team could maintain a cross-org holdout — it requires platform-level coordination, consistent randomization, careful metric computation. But the insight it produces flows back to the teams, who use it to calibrate their own experiment results. The method is centralized; the learning is distributed.
Experimentation Representatives
The context gap between the platform team and its consumers grows with both sides. The platform doesn't know the business logic of a product team. The product team doesn't fully understand the platform's statistical assumptions or how to correct for peeking.
The temptation is to grow the platform team into every org. That doesn't scale, and it centralizes judgment in the wrong place. The better structure is to invert: place a representative inside each major org — a data scientist, engineer, or PM fluent in experimentation — who carries a local mandate. They review complex setups before launch, translate platform updates into local context, and surface patterns the platform team would only see as incidents.
Judgment stays decentralized; training stays centralized. The platform equips the representatives with the methodology; the representatives apply it with the context only they possess. The round-trip from idea to valid experiment gets shorter, and the platform team stops being a queue.
What the representative gets is real: a defensible mandate, cross-cutting visibility, and work that compounds. What the platform gets is a network of people who can answer questions it would otherwise have to triage cold.
Experimentation is the scientific method at company scale. The method is universal. The application is always local. Build the platform to standardize the former, and design the structure to protect the latter.