Incrementality Testing: Proving an Ad Channel Actually Works

A geo holdout: test cities minus control cities is the lift the ads caused, and it is the only number that should set the channel's cost per acquisition.

Every ad platform reports conversions it can take credit for. None of them reports conversions that would have happened without it.

That gap is where budgets go wrong. A retargeting campaign that "converts" at ₹300 an order looks unbeatable — until you notice it mostly shows ads to people who had already added to cart.

Incrementality testing closes the gap by running a controlled experiment instead of trusting a report. It is the third of the three credit-assignment methods covered in our marketing analytics guide, and the only one that measures cause rather than correlation.

What is incrementality?

Incrementality is the difference between what happened because of your ads and what would have happened anyway. The word for that difference is incremental lift.

Picture a Mumbai coaching institute running Google Search ads on its own name. Someone who has already decided to enrol types the institute's name, sees the ad at the top, clicks it and pays. Google counts a conversion. The ad did nothing — the organic listing was one centimetre lower.

That conversion is attributed but not incremental. Incrementality testing is how you tell the two apart.

The logic is borrowed from clinical trials. You cannot see the same customer both with and without the ad, so you build two groups alike in every way except one: one is exposed to the ads, the other is not. Whatever gap opens between them is the ads' doing.

Attribution vs MMM vs incrementality: which one answers what?

These three get lumped together, and they should not be. Each answers a different question, and this post owns only the third.

• Attribution follows individual users across touchpoints and shares credit by a rule (last click, data-driven). It describes the path a converting user took. Our attribution modelling post owns that subject.
• Marketing mix modelling uses statistics on aggregate spend and sales over a long period, with no user tracking. It describes the historical relationship between spend and outcomes. Our marketing mix modelling post owns that.
• Incrementality testing runs an experiment now, on one channel or tactic. It measures the causal effect of that thing, in that period, at that spend level.

Attribution is always on and cheap. MMM is slow and needs a year or more of varied spend. Incrementality is precise but narrow: one answer per test. Used together, attribution runs daily, MMM sets the annual split, and incrementality tests calibrate both.

Which incrementality testing methods exist?

Three practical methods. They differ in who does the randomising, what data they need and how small a budget they can work with.

1. Holdout test (user-level)

A holdout test randomly splits an audience into a test group that sees your ads and a control group that does not. The platform does the splitting, because only the platform knows who it is showing ads to.

The older PSA variant shows the control group a public-service ad instead of nothing. It answers "did they see any ad?" but costs money to show ads that sell nothing, so platforms have largely moved to true holdouts.

2. Geo experiment (geo lift test)

A geo lift test uses places instead of people. You run the channel in some cities (test), switch it off in comparable others (control), and compare outcomes.

Its strength is that it needs no user tracking. Orders, leads, store footfall, WhatsApp enquiries — anything you can count by city works. It is the method that survives cookie loss, consent rules and offline conversions.

Its weakness is matching. Pune and Hyderabad are both large and tech-heavy, but they are not the same city, and any difference in baseline behaviour looks like ad effect unless you account for it. The fix is to compare each test city against its own history and against how its control partner moved in the same window — not raw totals.

3. Platform lift study (conversion lift)

A conversion lift study is a holdout the platform runs for you and reports inside its own interface. Two matter in India.

Meta Lift studies. Meta's documentation describes creating "a randomized test group of Accounts Center accounts that see your ads and control group who don't see your ads," then comparing conversions, converting accounts and revenue between them. Two caveats from the same page: Conversion Lift measurement "is currently limited" and requires a Meta representative, and a breakdown needs at least 100 conversions across test and control combined before results display (Meta for Developers, Lift studies).

Google Ads Conversion Lift. Google offers it "based on users" (groups built from aggregated user attributes) and "based on geography" (groups built from aggregated geographic data, with offline conversions supported). Google states that Conversion Lift "isn't available for all Google Ads accounts" and points you to your account representative (Google Ads Help, About Conversion Lift).

The geo version is the more useful one here. Google's setup page lists Search, Shopping, Performance Max, Display, Video, Demand Gen and App campaigns as supported, requires campaigns to target a single country, and splits that country into "Google Marketing Areas" chosen by an algorithm to minimise spill-over between test and control (Google Ads Help, Set up Conversion Lift based on geography). It lives under Lift measurement in the Goals menu, alongside Brand Lift and Search Lift (Google Ads Help, About lift studies).

No rep access on either? The geo holdout below is the same idea done by hand.

How do you calculate incremental lift, iCPA and iROAS?

The maths is one subtraction and two divisions. The hard part is getting clean numbers to subtract.

• Incremental conversions = test-group conversions − control-group conversions (scaled to the same size).
• Lift % = incremental conversions ÷ control conversions.
• iCPA (incremental cost per acquisition) = test-group spend ÷ incremental conversions.
• iROAS (incremental return on ad spend) = incremental revenue ÷ test-group spend.

A worked example, with illustrative round numbers. A D2C skincare brand runs Meta retargeting in five test cities and pauses it in five matched control cities for two weeks, spending ₹3,00,000 in the test cities.

• Test cities: 1,150 orders. Control cities, scaled to the same baseline: 1,000.
• Incremental orders = 1,150 − 1,000 = 150. Lift = 15%.
• iCPA = ₹3,00,000 ÷ 150 = ₹2,000.
• At ₹1,800 average order value, incremental revenue = ₹2,70,000. iROAS = 0.9.

Meanwhile Ads Manager attributed 600 orders to the same campaign: a reported CPA of ₹500 and a reported ROAS of 3.6.

Both numbers are "true" in their own frame. Only one tells you the channel lost money. That gap is why the platform's own number should never be the one that sets the budget.

None of this works if the conversion events are wrong to begin with. A test comparing two noisy counters produces a confident-looking answer that means nothing — our conversion tracking post covers what "clean" means for an Indian lead-gen or e-commerce site.

How big and how long does a test need to be?

Big enough that the difference you are looking for is larger than the noise you would see anyway. That is the whole rule; every published threshold is someone's guess at where it lands for their business.

• The lift you expect. A channel you think adds 30% needs far fewer conversions to prove it than one you think adds 5%. Small effects need big samples — which is why retargeting, where honest lift is often small, is hard to test on a small account.
• How noisy your baseline is. A clinic getting 8 to 30 enquiries a day has a wider band of "normal" than one getting 20 ± 2. The wider the band, the longer you run before a real difference shows through.
• How many units you have. In a geo test the unit is the city, not the order. Two cities against two is an anecdote; ten against ten starts to be evidence.

Duration follows the same logic, plus two practical limits. Run long enough to cover a full weekly cycle and the channel's lag (a retargeting ad seen Monday may convert Thursday). Run short enough that nothing else changes — no price change, no festival, no launch mid-test.

If you want a number rather than a rule, derive it from your own data. Meta's open-source GeoLift package (R, MIT-licensed) includes market-selection and power tools that tell you, from your historical city-level sales, which cities to pair and how long to run for a given detectable lift. That is an honest threshold: yours, not a blog's.

Why do brand search and retargeting test so badly?

Because both are aimed at people who were already on their way. They collect demand more than they create it, and last-click attribution rewards collecting.

The best-known evidence is eBay's. In a large field experiment, economists Blake, Nosko and Tadelis switched off eBay's paid search in some US regions and measured sales. Their paper reports that "brand-keyword ads have no measurable short-term benefits," and that paid-search returns overall were "a fraction of conventional non-experimental estimates" because the people most likely to click were the people most likely to buy anyway (NBER Working Paper 20171).

That is one company, one country, a decade ago. Do not take it as your answer — take it as the reason to test.

The same mechanism sits under retargeting. The audience is defined by intent it already showed. Some needed the nudge; many would have come back through a bookmark, an email or a search. Attribution hands the ad credit for all of them. Only a holdout tells you the split.

The pattern usually reverses for upper-funnel channels. Prospecting on Meta, YouTube or Demand Gen tends to look weak on last-click and better on incrementality, because the conversions it causes get collected later by brand search and retargeting. A programme that cuts prospecting and doubles retargeting on last-click evidence is optimising toward its own measurement error.

When we scope measurement inside a performance marketing engagement, the first test we propose is almost always a retargeting or brand-search holdout: it is the cheapest to run and the most likely to move budget.

How to run a two-week geo holdout on retargeting

This is the small-budget version. It needs Ads Manager, a spreadsheet and the discipline not to touch anything for two weeks.

1. Pick the question. One channel, one tactic, one metric. "Does Meta retargeting produce incremental first orders?" is a test. "Is Meta working?" is not.
2. Pull 8–12 weeks of city-level history. Orders or qualified leads by city by week, from your order system or CRM — not from the ad platform. This is the baseline that tells you which cities behave alike.
3. Choose matched pairs. Pair cities whose weekly volumes move together: Pune with Hyderabad, Jaipur with Lucknow, Kochi with Coimbatore. Aim for at least five pairs. Leave your biggest city out — it is too different to match and too costly to pause.
4. Assign test and control by coin flip. Within each pair, randomly decide which city keeps retargeting. Choose by hand and you will pick the "safe" ones for control, contaminating the test before it starts.
5. Exclude control cities from the retargeting ad sets. Use location exclusions on the existing ad sets rather than new campaigns, so nothing else about delivery changes. Leave every other channel untouched in every city.
6. Freeze everything for 14 days. No budget changes, creatives, offers or launches. Log anything you cannot control — a competitor sale, a cricket final, a rain week — so you can read the result honestly.
7. Read the result pair by pair. Compare each test city's orders against its own baseline and against its control partner's movement in the same fortnight. Sum the differences for incremental orders, then compute iCPA and iROAS as above.
8. Decide, then re-test the decision. Incremental? Restore it and consider scaling. Not? Cut it and test the next channel. Either way, repeat in a quarter — incrementality is not a constant.

Two temptations. The first is to peek on day three and stop because it "looks clear" — early results are noise. The second is to read a negative result as "retargeting is bad." It is bad at this spend, for this audience, this fortnight. The same audience at a quarter of the budget may well test positive.

What do you do with the result?

Move budget, recalibrate the models, schedule the next test. A result that does not change a decision was not worth running.

• Reset the channel's target. If the platform reports ₹500 CPA and the test says ₹2,000 iCPA, the 4× ratio is your correction factor. Set the platform's CPA target so the incremental CPA hits your real limit — and set that limit from margin and repeat rate, not first-order revenue, as our customer lifetime value post explains.
• Feed it back into attribution and MMM. A test result is the calibration point both models lack. If retargeting is a quarter as effective as last-click says, weight it that way in the dashboard and give the MMM the test as a prior.
• Test in order of budget, not curiosity. The next test is on whichever channel spends the most and has never been tested. On most Indian accounts that is brand search, then prospecting, then a "more vs less" spend test rather than on/off.
• Write it down. Date, cities, spend, baseline, result, confidence, what changed. A year of these is the record that lets a founder or CFO trust the budget you ask for.

Incrementality testing turns paid media from a report you receive into an experiment you run. For where it fits alongside channels, bidding and creative, start with our performance marketing guide.