Why last-click attribution fails at iOS scale?
Last-click attribution assumes the final touchpoint caused the conversion. iOS 14.5 (2021) made that assumption impossible. Apple's SKAdNetwork reports conversions 24–48 hours late, strips user IDs, and caps campaigns to 100 distinct conversion values. On Android and web, the tracking is deeper, but the iOS gap creates asymmetry: you see full funnels on Android, blind spots on iOS. Ad networks fill the gap with modeling (Meta's Conversion Lift, Google's Incrementality), but those models trained on historical data don't account for new audiences, seasonal shifts, or your specific unit economics.
The result: your paid media team reports 3:1 ROAS on Android and guesses at iOS. Finance challenges the guess. Budget stays flat. You never know if paid media actually moves the needle at scale or just captures demand that converts anyway.
What is incrementality testing and how does it work?
Incrementality testing is a randomized experiment: run a campaign to one audience cohort, don't run it to an identical control group, measure the difference in conversion rates between the two. If the treatment group converts at 2.1% and the control at 2.0%, the incremental lift is 0.1 percentage points, or 5% lift. Scale that lift across your annual paid spend, multiply by your margin, and you have true ROI—untethered from last-click fragmentation.
There are three main approaches:
- Holdout tests: Stop spending on a user segment for one week. Measure the drop in conversions. That gap is your baseline incremental impact. Cost: lost revenue during the test window. Duration: 1–4 weeks. Best for: mature campaigns with high volume.
- Geo-lift tests: Run campaigns in some geographic markets (test), pause in others (control). Measure KPI lift in test geos after 2–8 weeks. Cost: lower (geo-level, not user-level). Duration: longer. Best for: national or multi-region businesses with independent local demand.
- Matched-market tests: Use historical data to find two cohorts (age, region, behavior) with identical conversion trends pre-campaign. Run paid to one, not the other, after matching. Measure lift post-campaign. Cost: minimal. Duration: 2–6 weeks. Best for: e-commerce, SaaS, early-stage validation.
All three bypass SKAdNetwork and device-level targeting. You're measuring cohort-level effect, not individual attribution.
How long does an incrementality test take to reach confidence?
Confidence depends on volume, baseline conversion rate, and acceptable margin of error. Most tests need 2–6 weeks of data to reach 80–90% statistical confidence (the industry standard).
A quick math example: if your baseline conversion rate is 2%, you need roughly 2,500–5,000 conversions in each arm (treatment and control) to detect a 20% relative lift at 90% confidence. At 100 conversions per day, that's 25–50 days. High-volume businesses (5,000+ daily conversions) can validate in 5–10 days. Low-volume (under 100 per day) may need 12+ weeks or smaller effect sizes to be statistically honest.
Practical timeline:
- Weeks 1–2: Set up test arms, balance cohorts, launch. Collect ~30–50% of target conversions.
- Weeks 2–4: Reach 80–90% confidence. Begin early-read analysis to validate directional effect.
- Weeks 4–6: Run full duration to account for weekly/seasonal variation. Lock final result.
Geo-lift tests run longer (4–8 weeks) because geographic demand can be noisier. Holdout tests are faster (1–2 weeks) but shrink revenue—use sparingly and for high-impact campaigns.
What does it cost to run a proper incrementality test?
Direct cost is low: a statistical consultant ($2K–$8K), a platform like Measured, Recast, or Infer (~$10K–$30K per test), or your in-house data team if you have SQL + Python bandwidth. Real cost is opportunity: the control group you don't spend on.
If you pause paid spend on a 10% holdout for 2 weeks and that cohort would have generated $50K in revenue at 3:1 ROAS ($150K total value), you've sacrificed $50K in media spend and $150K in gross value to prove causality. That's expensive validation.
Smarter approach: geo-lift instead of holdout, or matched-market cohorts (historical, no lost spend). If you run $500K/month in paid media and allocate 5–10% to incrementality testing, that's $25K–$50K per month learning budget. Within 3–4 tests, you have a clear model of ROAS by channel and audience. You stop guessing. Your annual media decision-making gains precision worth multiples of the test cost.
At Carcin, we helped a small-business operator run a matched-market test for Google Shopping ($8K spend, 6 weeks, zero incremental cost). Result: 2.4x true incremental ROAS vs. 1.2x reported. They reallocated $500/month from email to Shopping. Six-month payback on the test: $3,200.
How do you set up a test that survives iOS fragmentation?
Design your test at the cohort level, not the device level. iOS attribution gaps don't matter if you're comparing treatment and control at the audience or customer level.
Matched-market test example (e-commerce, high-velocity):
- Cohort definition: Users aged 25–40, in US metro areas, browsed category X in the past 30 days, no purchase in 60 days. ~50K users.
- Match: Split into two identical cohorts using propensity scoring (logistic regression on past behavior: search volume, add-to-cart, page depth). Validate balance: mean age, gender ratio, browsing frequency must be <2% different.
- Treatment: Run paid (Google Ads, Meta, Pinterest) to Cohort A for 6 weeks.
- Control: Zero paid media to Cohort B.
- Measurement: Count conversions (first-party CRM data, pixel, or server-side) for each cohort across all channels for 6 weeks post-test. Compare conversion rates.
- Result: If Cohort A hits 2.5% conversion and Cohort B hits 2.1%, your paid media drove 0.4 points of incremental lift = 19% relative increase. Scale that to your annual spend and you have true ROI.
Geo-lift test example (national SaaS or ecommerce):
- Choose 2–3 test geographies (e.g., Denver, Austin) with stable demand and no major competitors or seasonal anomalies. Pair with matched control geos (similar size, income, behavior, but no paid media).
- Run paid in test geos for 8 weeks. Measure signups or trials in both. Account for exogenous shocks (news, competitor move, weather).
- Calculate difference-in-differences (test lift minus control lift). That's your causal incremental effect.
Avoid these pitfalls:
- Don't use last-click attribution as your measurement post-test. Use aggregate conversion data (first-party CRM, Snowflake, revenue table).
- Don't balance cohorts on day 1, then let them drift. Re-validate balance at week 2 and week 4.
- Don't change creative, bids, or targeting during the test window. You're measuring media impact, not optimization.
- Don't confuse statistical significance (p < 0.05) with practical significance. A 0.5% lift with p < 0.05 may not justify increased spend if your margin is thin.
What happens after you have incremental numbers?
Feed the lift rates back into your media planning. If Google paid search increments at 1.8x true ROAS and paid social at 2.2x, shift budget accordingly. Run tests quarterly or whenever your audience mix or seasonal patterns shift. Teton Gravity Research validated paid search incrementality at 7x ROAS, then scaled Google Shopping. Within a year, ecommerce revenue grew 500%—confidence from incrementality, not guesswork.
Incrementality testing is the only way to prove paid media ROI in a post-SKAdNetwork world. Start with one matched-market test (low cost, 6 weeks). Measure once, shift budget twice, and you've compounded your confidence faster than iOS attribution will ever catch up.
