A report tells you what happened. An incrementality test tells you what your marketing caused. The gap between these two statements is where significant budget misallocation lives — and most Australian organisations are not structured to close it.
The Report as a Substitute for Evidence
Digital marketing programmes generate more data than almost any other business activity. Dashboard infrastructure has become sophisticated to the point where a head of performance can access real-time reporting across dozens of channels, audiences, and creative variables simultaneously. The paradox is that this abundance of data has not produced a commensurate improvement in the quality of causal understanding. Most of what passes for marketing performance analysis is correlation observed within a system that is not designed to distinguish correlation from causation. The numbers look like evidence; they are, in many cases, a sophisticated description of what happened rather than an explanation of why.
The fundamental problem is one of experimental design. Standard marketing reporting observes outcomes — conversion rates, ROAS, click-through rates — within campaigns that were not structured as experiments. The audiences who saw the ads are not compared against a matched group who did not. The channels running during the measurement period are not isolated from the other marketing activity occurring simultaneously. The business outcomes attributed to the campaign are not separated from outcomes that would have occurred in its absence. The resulting data can be described, aggregated, and trended, but it cannot answer the fundamental question that marketing investment decisions require: what would have happened if this activity had not run?
This is not a niche academic concern. The inability to distinguish incremental from non-incremental outcomes has direct financial consequences. Organisations that cannot measure incrementality are paying for outcomes that would have occurred regardless — cannibalising organic conversions with paid activity, attributing brand-driven demand to performance channels that captured but did not create it, and scaling channels based on reported returns that do not reflect actual business contribution. The cost of this measurement failure, across an industry that spends billions annually on digital advertising, is enormous.
The Logic of Incrementality Testing
An incrementality test answers a specific question: what is the difference in outcome between an audience exposed to a marketing intervention and a matched audience that was not? The experimental design requires a control group — audiences or markets that are excluded from the marketing activity being tested — and a treatment group that receives normal exposure. By comparing outcomes between the two groups over the same time period, the test isolates the causal effect of the marketing activity from baseline behaviour that would have occurred regardless.
The two primary methodologies for incrementality testing in digital marketing are ghost ads (or phantom ads) and geo-based holdout tests. Ghost ad tests operate within a digital platform, using the same ad auction infrastructure to serve the treatment group ads and to record equivalent impressions for the control group without actually delivering them. The platform infrastructure ensures the groups are matched in terms of auction eligibility and timing; the difference in conversion rates between groups reflects the ad’s incremental effect. Geo-based holdout tests divide markets geographically — typically by state or postcode cluster — allocating some markets to receive advertising and others to serve as holdout controls. The difference in business outcomes between exposed and control markets, adjusted for baseline differences, measures the advertising’s incremental contribution.
A report tells you what happened. An incrementality test tells you what your marketing caused. Most organisations are making investment decisions based on the former while believing they have the latter.
What Incrementality Tests Typically Find — and Why It Matters
The results of incrementality tests, when organisations run them with genuine rigour, are routinely surprising and frequently uncomfortable. Branded search campaigns — which consistently appear as high-ROAS channels in attribution reporting — routinely show incrementality rates of 20–40 per cent, meaning that the majority of conversions attributed to branded search would have occurred through organic search had the paid activity not run. Retargeting campaigns, which appear efficient because they reach high-intent audiences, often show incrementality rates below 50 per cent for similar reasons. The audiences being targeted were already planning to convert; the advertising claimed credit without creating the outcome.
These findings do not necessarily mean branded search or retargeting should be eliminated. They mean that the budget levels and return expectations applied to these channels should be calibrated against their actual incremental contribution rather than their attributed conversion volume. An organisation that discovers its branded search programme is 30 per cent incremental has the information it needs to make a rational decision about the appropriate investment level. An organisation that does not test remains in a state of costly uncertainty — either over-investing in a low-incrementality channel or, less commonly, under-investing in a genuinely high-incrementality one.
The Organisational Resistance to Testing
Incrementality testing is not technically complex. The methodologies are well-established, the tools to implement them are available, and the analytical requirements are manageable for organisations with modest data capability. The barrier to widespread adoption is not technical — it is organisational. Incrementality testing threatens the reported performance of channels that currently look efficient within the attribution framework, and channel owners have a vested interest in maintaining measurement frameworks that validate their programmes. An agency managing a retargeting programme on a performance-linked fee structure has limited incentive to advocate for an incrementality test that might reveal the programme is 40 per cent incremental.
This organisational dynamic means that incrementality testing is typically driven by the client organisation rather than by agencies or platform partners. CMOs who want genuine incrementality measurement need to build the internal capability to design, implement, and interpret tests independently of the parties whose compensation is tied to the outcomes being measured. This requires some internal analytical resource, a relationship with an independent measurement partner, or a structured commitment from the organisation’s primary agency to operate with full measurement transparency.
From Testing Culture to Strategic Advantage
Organisations that build a systematic incrementality testing programme do not simply gain more accurate measurement. They build a strategic asset: a continuously improving body of knowledge about what their marketing actually causes, which compounds over time into a material advantage over competitors who are making decisions based on attribution data rather than causal evidence. This advantage is not theoretical. It translates directly into more efficient budget allocation, fewer dollars spent on non-incremental activity, and a greater proportion of investment directed toward genuinely high-contribution channels and activities.
For boards and executive teams, the governance question is whether the organisation has a systematic programme of marketing experiments that goes beyond tactical A/B testing to test the fundamental causal assumptions underlying major budget allocation decisions. Organisations that do not test their core assumptions about channel contribution are, in effect, managing a significant capital allocation function on the basis of received wisdom and vendor-reported data — a governance standard that would not be acceptable in any other area of significant organisational investment.