Steps in running a Geo Experiment on Lifesight

Hypothesis

Hypotheses are generated by using the following approaches:

1. Recommendations from the Attribute Quality Scores (AQS) from an existing MMM (model).
2. Testing commonly tested tactics (e.g. BOF, Video, etc.).
3. Marketer-driven exploratory tests, based on strategic or operational initiatives, innovative channel opportunities, or market-specific hypotheses. This includes measuring the incrementality of a newly-launched channel.

Specify secondary hypotheses, to capture potential halo effects (e.g. Impact to marketplace sales like Amazon from scaling up Meta spend).

Experiment Design

Lifesight supports single and multi-cell experiment designs.

Single cell experiments - These involve one test group and one control group, with all test regions receiving the same marketing treatment (where is a treatment is scaling up or cutting spend). These tests are useful to determine overall incrementality of a channel.

Multi-cell experiments - These involve multiple distinct test groups, each receiving different treatments, along with associated control groups receiving no additional treatment. These tests are useful for testing multiple treatments or strategies simultaneously.

Decide and define the following parameters for an experiment design:

• Cost Per Incremental Conversion (CPIC) - Decide the maximum Cost Per Incremental Conversion that you're willing to pay for the channel being tested.
• Effect size - The magnitude of the incremental impact you’re aiming to detect through your experiment - start with a value of ~15-20% if unsure.

Once the channels and tactics to experiment on are decided, a decision to scale up or hold out spend is made.

Data Preparation

Collect historical data for selected geographic regions covering the pre-experiment baseline period.

Available options:

• Robust Multiple Objective Experiments: 2 years
• Multiple Objective Experiments: 1 year
• Single Objective Experiments (recommended for Non-Seasonal Businesses): 3-6 months

Lifesight supports the following methods of data input to run an experiment:

• Upload (csv file)
• Integrated (using a Shopify connection)
• Google Sheets

A sample file for csv and Google sheets uploads can be found here.

Power Analysis and Statistical Validation

• Conduct power analysis to determine the required duration and detectable effect size.
• Select test parameters (e.g., experiment duration, budget allocation) based on the power analysis results.

Experiment Deployment

• Launch marketing interventions exclusively within test regions
  • Lifesight supports auto-deployment of experiments to Meta and Google.
  • For other channels, a deployment guide is provided.
• Maintain consistent marketing activities in control regions throughout the experiment period.

Post-Experiment Analysis

• Evaluate incremental impact by comparing performance in test vs control regions.
• Include a defined post-treatment window to capture any lagged effects of the marketing intervention.
• Interpret statistical outputs, confidence intervals, and significance levels to quantify the experiment's effectiveness.

Measurement of Secondary KPIs (Halo Effects)

Analyze the impact on secondary KPIs, such as marketplace sales or other indirect effects influenced by the primary marketing intervention.

Quantify and interpret halo effects to provide comprehensive insights into overall marketing effectiveness.

Reporting and Insights

Summarize experiment results, highlighting incremental impact, statistical significance, and actionable insights.

Document recommendations and potential optimizations based on experiment outcomes for future marketing initiatives.