Run a campaign holdout that answers a real business question

How do retailers run a reliable campaign holdout experiment?

Define one campaign change, establish eligibility and randomly assign customers to treatment or an appropriate comparison before activation. Keep assignment stable, protect the comparison from the tested campaign and measure outcomes for all assigned customers over equal periods. Choose sample size around the effect worth detecting, then inspect data quality and uncertainty before interpreting lift or making a rollout decision.

Write the experiment contract before the campaign brief

A holdout is useful when it changes a business decision. Start with a question such as whether an additional replenishment message produces enough incremental contribution to justify sending it. State the treatment, the existing experience, the eligible population and the outcome window. Decide what result would justify rollout, revision or stopping.

Choose a primary outcome before launch. Qualifying purchase rate or contribution per assigned customer may be more useful than opens for a commercial campaign. Keep diagnostic measures, such as delivery and clicks, alongside the primary measure. They explain execution, but should not replace the original objective because they happen to look positive.

Assign customers once and preserve the assignment

Create the eligible population using only information available before assignment. Resolve duplicates and apply the relevant permission rules first. Then randomise at the appropriate unit: often the customer, sometimes the household, store or catchment when spillover is material. The analysis must respect that unit rather than pretending many transactions are independent people.

Store an immutable assignment record with experiment ID, customer key, group, assignment time and rule version. Stable hashing can implement reproducible allocation when designed correctly, but an operations team still needs checks for identity changes and repeated enrolment. A customer should not switch groups simply because a contact file was regenerated.

Google's Conversion Lift guidance describes treatment and comparison groups for incremental advertising measurement. A retailer's direct-contact holdout applies the same comparison logic while relying on its own assignment and transaction records. Availability of an advertising study tool is a separate implementation question.

Choose the comparison experience deliberately

A no-additional-message group measures the value of sending that message above the existing experience. A standard-message group compared with a new creative measures the difference between the two creatives. A reward-plus-message group compared with a message-only group isolates the added reward more closely. These are different questions and should not share an ambiguous label such as campaign lift.

Keep promised programme benefits and essential service communication available in both groups. Document other campaigns that may reach the participants. If routine campaigns continue equally, the test measures the new intervention in that operating environment. If only one group receives an extra unrelated promotion, the interpretation changes.

Plan sample size around a worthwhile effect

There is no universal holdout percentage that makes a campaign reliable. The required sample depends on baseline behaviour, the minimum effect worth detecting, outcome variability and the chosen error tolerances. A small control may preserve reach but make the commercial estimate too uncertain to guide a decision.

Ask an analyst to size the study before launch. Include expected enrolment, delivery limitations and the full outcome delay in the calendar. For rare purchases or noisy contribution, the team may need a longer enrolment period or fewer variants. Do not stop the test on the first promising day unless a valid sequential analysis was planned in advance.

Worked hypothetical example: interpret unequal group sizes

Assume 20,000 eligible customers are assigned to a campaign experiment, with 16,000 in treatment and 4,000 in the comparison group. Over the same 30-day outcome window, 960 treatment customers and 200 comparison customers make a qualifying purchase. The conversion rates are 6% and 5%, respectively.

The observed absolute lift is one percentage point. Relative lift is one divided by five, or 20%. Applying the absolute difference to the 16,000 treatment customers gives an estimated 160 incremental purchasing customers. It would be incorrect to call all 960 treatment purchasers incremental, or to subtract raw purchase counts without adjusting for the different group sizes.

Suppose the defined campaign spend is ₹80,000. Incremental cost per purchasing customer is ₹80,000 divided by 160, or ₹500, using the point estimate. This does not establish profitability: the team still needs incremental contribution on a consistent basis. If the lift estimate is uncertain or compatible with no improvement, the cost-per-incremental-customer estimate is correspondingly unstable.

These numbers are hypothetical. The arithmetic is descriptive and does not claim statistical significance. The result applies to this eligible population, intervention and period, rather than establishing a permanent response rate for the whole retail business.

Check the experiment before reading the winner

Reconcile eligibility, assignment, attempted sends, deliveries and observed outcomes as separate stages. A delivery failure belongs in the assigned treatment population for the primary intention-to-treat comparison. Removing failed deliveries can select customers with different characteristics and change the question being answered.

Microsoft Research documents sample ratio mismatch as a warning that experimental populations may have been distorted. Compare observed assignment and analysis counts with the planned allocation using an appropriate check. Investigate faulty joins, missing identities and unequal filtering before interpreting performance differences.

Also inspect contamination: shared coupons, store staff promoting the offer to everyone, or another channel contacting the holdout. Report it rather than quietly deleting affected customers. Where contamination is substantial, revise the design so the next test has a credible separation between experiences.

Experiment-release checklist

1. Record the hypothesis, intervention and primary decision metric.
2. Approve eligibility, assignment unit and minimum useful effect.
3. Freeze the comparison experience and cross-campaign rules.
4. Verify stable allocation and complete outcome collection.
5. Allow equal follow-up and the agreed return-adjustment period.
6. Review uncertainty, contribution and customer friction before rollout.

Use campaign management and retail intelligence to discuss the required operating evidence. Request a Fundle walkthrough of your experiment register, audience controls and outcome review. Verify what the proposed configuration supports rather than assuming that any campaign dashboard is automatically a causal measurement system.

Questions from retail teams

What percentage of customers should be held out?

Choose the allocation from the measurement need, rather than a universal rule. Consider baseline purchase rate, the smallest effect worth acting on, expected sample size and outcome variability. A small holdout can leave the result too uncertain. Ask an analyst to size the design before committing the campaign calendar and contact budget.

Should customers with failed deliveries be removed from treatment?

Keep them in the original assigned group for the primary intention-to-treat comparison. The business decision concerns offering the campaign through the actual delivery process, including its limitations. Analyse delivery separately to diagnose operations. Removing failed deliveries after assignment can create a selected group and exaggerate the effect of the campaign.

Can a holdout still receive ordinary store promotions?

Yes, if that is part of the defined comparison environment and applies consistently across groups. The study then measures the extra tested campaign above those ordinary promotions. Document overlap so the result can be interpreted. If unrelated treatment differs systematically between groups, you may be measuring a bundle of changes rather than the intended intervention.

How is an A/B creative test different from a no-contact holdout?

An A/B test between two messages estimates which message performs better under the tested conditions. A no-additional-contact holdout estimates the effect of contacting customers compared with the existing experience. Either can be useful, but they answer different questions. A winning creative does not by itself prove that sending either version creates incremental value.

What should we do when the experiment result is inconclusive?

Report the uncertainty and the range of commercially plausible outcomes. Check whether the test had adequate sample, clean assignment and complete follow-up before interpreting the lack of clarity. Depending on the decision, continue a properly planned enrolment period, simplify the test or revise the hypothesis. Do not turn an inconclusive result into a positive claim.

Sources and further reading

• Google Ads: About Conversion Lift — Treatment/comparison principle for incrementality. The retailer experiment contract, numerical example and implementation checklist are original guidance.
• Microsoft Research: Diagnosing Sample Ratio Mismatch in A/B Testing — Sample ratio mismatch can reveal distorted experiment populations and warrants investigation before interpreting effects.