Source-linked AI summary
Designing and Deploying Online Field Experiments
Eytan Bakshy, Dean Eckles, Michael S. Bernstein
TL;DR
Online experiments are widely used for product decisions, but producing generalizable knowledge requires sophisticated designs and careful iterative management that existing tools only partly support. The paper introduces PlanOut to separate experimental design from application code and adds infrastructure for managing, logging, and analyzing iterative experiments. It presents PlanOut as a way to specify complex randomized parameterizations and support experiments aimed at causal mechanisms and general design principles, while recognizing limits to generalization and to designs requiring coordinated assignments.
Problem
Online experiments are widely used, but tools provide limited support for sophisticated designs, iterative deployment, and the logging and analysis needed for generalizable knowledge.
Method
The paper introduces PlanOut, a domain-specific language that separates experimental design from application logic, together with infrastructure for managing, logging, and analyzing iterative and distributed experiments.
Results
PlanOut specifies simple and complex randomized parameterizations, including designs involving multiple units and continuous treatments, and supports concise experiment descriptions and exposure logging.
Takeaways & Limitations
Parameter-based scripts and supporting infrastructure can help online experiments investigate causal mechanisms and general design principles beyond choosing among existing alternatives.
Takeaways & Limitations
Field-experiment results may not generalize across times and populations, while designs requiring coordinated assignments across many units can require offline computation.
Abstract
from arXiv · showhide
Online experiments are widely used to compare specific design alternatives, but they can also be used to produce generalizable knowledge and inform strategic decision making. Doing so often requires sophisticated experimental designs, iterative refinement, and careful logging and analysis. Few tools exist that support these needs. We thus introduce a language for online field experiments called PlanOut. PlanOut separates experimental design from application code, allowing the experimenter to concisely describe experimental designs, whether common "A/B tests" and factorial designs, or more complex designs involving conditional logic or multiple experimental units. These latter designs are often useful for understanding causal mechanisms involved in user behaviors. We demonstrate how experiments from the literature can be implemented in PlanOut, and describe two large field experiments conducted on Facebook with PlanOut. For common scenarios in which experiments are run iteratively and in parallel, we introduce a namespaced management system that encourages sound experimental practice.
1. INTRODUCTION
Online field experiments can support generalizable knowledge and strategic decisions, but iterative, distributed practice creates design, deployment, logging, and analysis challenges. The paper introduces PlanOut and management guidelines to make complex experimentation more systematic.
- Online experiments can explore design spaces, attribute outcomes to causes, and estimate effects for understanding user responses rather than only picking winners.
- Iterative and distributed experimentation complicates evaluation because changing live experiments can produce incorrect statistical inferences and follow-up work can be error prone.
- PlanOut separates experimental design from application code by representing service features as parameters assigned through concise scripts.
- PlanOut combines random-assignment primitives to specify complex designs involving conditional logic, multiple experimental units, multiple treatments, and continuous values.
- The proposed management system supports iterative and parallel experiments through experiment management, logging, interference prevention, and sound analysis practices.
- Short, centralized scripts summarize assignment procedures, improving communication and replication while encouraging more agile and generalizable experimental practice.
2. RELATED WORK
Existing experiment-design tools support small, preplanned studies and online experimentation instrumentation, but online settings require real-time assignment and can involve unit characteristics unavailable in advance. Related systems also address exclusivity, configuration, or outcome optimization.
- Design-matrix packages support factorial and fractional factorial studies, but their a priori designs are poorly suited to real-time online assignment.
- Online experiment frameworks and recommendations address crowdsourcing deployment, instrumentation, and common analysis pitfalls, although Internet-industry experimentation is only partially represented in scholarly literature.
- Google, Microsoft, and Facebook systems use layers or universes to make conditions mutually exclusive, while some tools link condition identifiers to parameter configurations.
- Multi-armed bandit methods optimize stochastic outcome functions, whereas the paper focuses on experiments intended to inform product decision-making.
3. THE PLANOUT LANGUAGE
PlanOut separates experimental design from application logic by expressing parameter assignments in concise scripts that can be serialized and executed across platforms. Its operators support standard, factorial, conditional, and multi-unit experiments, enabling designs that target different causal effects.
- PlanOut separates experimental design from application logic by representing experiments as parameter assignments.Scripts specify assignment inputs, randomization procedures, and parameter names accessible through an API and logging system.
- Serialized PlanOut definitions can be stored and executed across backend, frontend, and mobile platforms.Definitions can be generated or edited through the DSL or graphical interfaces and represented as JSON.
- Uniform and weighted assignment operators support factorial designs, including six conditions from two independently assigned parameters.The example assigns three button colors uniformly and button text with 80% “Sign up” and 20% “Join now” weighting.
- Conditional control-flow operators handle experiments in which valid values or assignment probabilities depend on other variables.Supported operations include if / else, boolean and comparison operators, and array indexing.
- Tuple-based experimental units enable within-subjects designs that independently randomize treatment across viewer–story combinations.A [viewerid, storyid] unit assigns each viewer an expected independent 5% of stories with collapsed comment boxes, estimating story-level effects on responses.
- Changing the randomized unit changes the estimand: viewer-level assignment affects all stories for selected users, while story-level assignment affects all viewers of selected stories.The paper contrasts these designs with tuple-based assignment, which randomizes individual viewer–story exposures.
4. EXAMPLE EXPERIMENTS
PlanOut can express both basic and complex online field experiments, including published social-computing studies and designs involving conditional logic or multiple experimental units.
- PlanOut's design range: PlanOut concisely specifies complex scientific experiments while remaining accessible for basic experiments.The paper presents it as a low-threshold language with a high ceiling.
- Published experiments: The MovieLens study varies group versus individual goals and specific versus nonspecific goals through randomized email assignments.The example shows how experimental logic can be separated from application code.
- Published experiments: For social cues, each user–page pair is assigned a cue count between one and the maximum displayable number, then specific friends are sampled.Inputs are userid, pageid, and the array of friends associated with the advertised page.
- Facebook experiments: The voter-turnout experiment jointly manipulates banner exposure, eligibility for social voting stories, and conditional call-to-action text.The banner is assigned to 97% of users, while feed-story eligibility depends on banner assignment.
- Facebook experiments: A continuous-treatment design assigns each source user a collapse probability and applies it across source–story–viewer assignments.The design supports analyses of feedback received, engagement, and instrumental-variable relationships.
5. RUNNING EXPERIMENTS
Running online experiments requires infrastructure beyond randomized assignment. PlanOut's supporting system manages experiment allocation, parameter defaults, parameter retrieval, exposure logging, and analysis support.
- Experiment management: An experiment combines a PlanOut script with a target population and a launch time, while different experiments remain separate to experimenters and logging infrastructure.This definition establishes the unit managed by the surrounding system.
- Experiment management: The supporting infrastructure includes segmentation, launch defaults, a parameter API, and logging architecture.Together these components support management and analysis of deployed experiments.
- Experiment management: Automatic exposure logging records experiment context, inputs, and script-assigned variables when get() is called.The paper notes that exposure records can simplify analysis and distinguish affected from unaffected units.
5.1 Namespace model of site parameters
PlanOut uses parameter namespaces to support iterative and parallel experiments that manipulate persistent service parameters across teams and launch times.
- Motivation: Persistent parameters may be manipulated by multiple experiments conducted serially or in parallel, motivating a namespace model.The setting includes experiments run by different teams with varying degrees of coordination.
- Motivation: Iterative experiments may repeat designs for greater precision or introduce variations based on earlier results and redesigns.Continual development can change parameter effects, creating reasons for additional experiments.
- Namespace model: Because experiments can involve multiple teams and different service layers, experimentation tools must allocate units across experiments launched at different times.The paper identifies cross-platform support as part of this requirement.
- Namespace model: A namespace centers on a primary unit and allocates part of the population to a PlanOut script within a new or existing namespace.This organizes successive experiments around enduring service parameters.
5.2 Experiment management
The management system organizes experiments through namespaces and randomized segment allocation, supporting iterative launches while reducing interference and preserving valid comparisons.
- Segment allocation: Each experiment receives a random set of segments, which are deallocated after completion and can be reused by later experiments.New experiment versions can receive new segments during iterative experimentation.
- Namespace organization: Namespaces group experiments that set overlapping parameters, requiring experiments in the same connected component to be mutually exclusive.This prevents conflicting assignments when experiments affect the same parameters.
- Assignment independence: Assignment within an experiment is independent of segmentation, reducing carryover risks when segments are reassigned across experiments.The approach corresponds to local randomization and requires each experiment to have its own control group.
- Parameter defaults: Units outside an active experiment receive launch values or application defaults when no experiment sets the requested parameter.These values often represent the status quo or currently preferred settings.
- Iterative workflow: Iterative experiments are generally analyzed separately, and a later backtest can compare a new default with the original parameterization.Separate analysis avoids problems from pooling experiments with changed designs; the backtest evaluates the launch decision after a longer period.
- Statistical boundary: Excluding units outside an experiment from the control group can reduce analytical efficiency unless additional historical-value conditions are satisfied.An analyst may use those data when the required conditions hold.
5.3 Integration with application code
PlanOut integrates with application code by retrieving parameter values for specified units at runtime, while supporting overrides that help developers test and debug experimental behavior.
- Runtime retrieval: Application code requests namespace parameters for particular experimental units through an experiment object.For example, a viewer-story pair can be used to retrieve the collapse_story parameter.
- Execution pipeline: The management system maps units to segments and experiments, executes the corresponding PlanOut script, and returns either an assigned value or a parameter default.Assignment events are logged when a requested parameter is set.
- Testing and debugging: Developers can freeze parameters at prespecified values during script execution to test service behavior across parameter ranges.This functionality helps test and debug PlanOut scripts in situ.
- Override mechanisms: Frozen overrides can be supplied through URL query parameters for web applications or server-side management tools for mobile and backend services.Multiple parameters can be frozen together to test downstream effects of combined inputs.
5.4 Logging
PlanOut automatically records experiment exposure when application code retrieves parameters, creating data that supports more targeted and statistically efficient analysis.
- Automatic exposure logging: Exposure logging occurs automatically when get() is called, recording the units’ exposure to an experiment.By default, the system logs the namespace, experiment name, input data, and variables set by the script.
- Analytical value: Exposure records distinguish units affected by an assignment from assigned units that never reached the manipulated part of the service.This distinction can simplify downstream analysis and increase statistical power.
6. ANALYZING EXPERIMENTS
PlanOut supports automated analysis for many online experiments by exposing parameter assignments and experimental units, while retaining custom methods for complex designs and cautioning against indiscriminate pooling.
- Automated analysis: Much routine online-experiment analysis can be automated because PlanOut scripts encode assignment procedures and parameter structure.Automation can reduce analysis errors, improve comparability, and lower the burden of running additional experiments.
- Exposure-based analysis: Logging only exposed users can reduce estimated treatment-effect variance when relatively few assigned users encounter the manipulation.It also focuses analysis on a subpopulation whose outcomes are plausibly affected by treatment.
- Factorial designs: Parameter logs support summary statistics for unique parameter combinations and automation of main-effect analyses in full or fractional factorial designs.The approach uses labeled input units and assigned parameter values as the basis for analysis selection.
- Iterative experiments: Iterative experiment variants are normally analyzed separately because changed conditions and exposure timing can bias pooled comparisons through novelty or cumulative treatment effects.Adding users to an existing experiment while assigning new conditions is one example of this problem.
- Pooling experiments: Identical scripts can sometimes be pooled to increase power, but the resulting estimate is a weighted average of potentially time-varying effects.Pooling may understate or overstate treatment effects, and automated cross-experiment selection remains future work.
- Complex experimental units: Experiments assigning multiple unit types require analyses that account for dependence among repeated observations of the same user or advertisement.Inspecting the script can identify randomized units and guide subsequent inference-method selection.
7. DISCUSSION
The discussion presents PlanOut as a way to make online field experiments more expressive, iterative, collaborative, and oriented toward generalizable behavioral knowledge. It also emphasizes that online experiments and PlanOut retain important limits concerning generalization, interference, assignment complexity, and analysis.
- Implications: PlanOut supports parameterized experiments that target causal mechanisms and general design principles, rather than only choosing among existing alternatives.The paper frames this as a shift from short-term pick-the-winner testing toward scientific uses of randomized experiments.
- Implications: PlanOut’s short scripts help describe and collaborate on complex experiment designs before deployment, while its infrastructure supports Internet-scale execution.The authors report using PlanOut both as a deployment tool and as a notation for design collaboration.
- Limitations: Online experiment results from one time and population may not generalize to new times and populations because effects can be heterogeneous and dynamic.The authors hope more sophisticated behavioral experiments and easier replication can improve generalizability, but do not eliminate this limitation.
- Limitations: Interference occurs when one unit’s outcome is affected by other units’ assignments, including through peer influence and network effects.Connected users can substantially change post-launch behavior as they interact with one another.
- Limitations: PlanOut is less direct for designs whose assignments depend on many other units, including block randomization, graph cluster randomization, and sequential experimentation.Such designs may require offline computation and custom operators rather than only PlanOut’s built-in operators.
- Limitations: The paper only briefly addresses experiment analysis, while evaluating experimental tools can depend on organizational context and difficult-to-trace long-term consequences.The authors identify automated analysis and better evaluation methods as areas for further research.