A word before Frank starts commenting: he's spent forty years inside measurement, at just about every company that ever counted a viewer, and now he shows up in the margins here with what he actually thinks. Get used to him.

On August 18, Roku and Amazon Ads each published an “analysis of more than 400 brands,” and buried near the bottom of Amazon's version are two case studies offered as proof that the companies' direct integration outperforms the alternatives.

Here is the first one, as Amazon describes it. A pet food brand tested the direct Roku Exchange path against an indirect path through a supply-side platform with audience matching artificially disabled. The direct route delivered 21 percent lower cost per unique viewer and a 29 percent higher product page view rate.

Here is the second. An automotive brand reached 66 percent more unique viewers through the direct connection, compared to buying through a third-party SSP with data match purposefully disabled.

Read those two sentences again, because the qualifier is not mine.

It is theirs. They ran their targeting against a competing path with the competing path's targeting switched off, and reported the margin.

Nobody made them print that. It is in the copy, unprompted, offered as evidence.

I sent the full study to Trevor Testwuide, the CEO of Measured, and asked one question: can a day-level, non-randomized comparison support the causal claim being made here.

His first six words back: "In short, no, that design cannot support the claim."

The rest of what he sent is further down, and it is the most useful thing published about this study by anyone who does not sell CTV.

FRANK KOWALCZYK (Rolling Meadows, IL. Writing this from a parking lot, which tells you how the meeting went.)

They turned the other guy's targeting off and then published the margin. Forty years. I have never seen that. You know what that is? That's a footrace where you shoot the other guy. Then you put out a press release about your time.

And they wrote it down. Nobody made them. Somebody typed "artificially disabled" into a marketing page and eleven people approved it.

The headline finding is this. On days when a brand's Roku CTV campaigns were running, that brand's non-CTV channels on Amazon DSP generated 87 percent higher return on ad spend than on comparable days without Roku.

Three supporting claims come with it.

More than half of sales, 55 percent, came from new-to-brand customers, defined as shoppers with no purchase from the participating brand in the previous twelve months.

Increasing Roku spend by at least 10 percent corresponded to a 5.6 percent increase in branded search and a 6.4 percent increase in Amazon detail page views.

Brands that sustained Roku investment for six months or more saw gains above the 87 percent average, while advertisers running a month or two captured a fraction of the effect.

The design, as described: brand campaign performance on days when Roku Media was in flight versus days when it was not, with each brand serving as its own baseline, controlling for a set of variables that could otherwise explain the gap.

That is the whole disclosure. Every word of methodology either company has published is in the paragraph you just read.

I went looking for the study. There isn't one.

What exists is two blog posts. Roku's is filed on its advertising site under News and Events, not under Guides and Reports, where Roku files things it wants read as research. There is no PDF, no appendix, no technical note, no link to anything downstream.

Amazon's version carries a single source line at the bottom of the page. In its entirety: Amazon internal, full panel 400+ brands, US, Jan 2025 to Mar 2026.

That is the methodology section. One sentence, seventy-some characters, for a finding that two of the largest advertising companies on earth are putting in front of media buyers as a reason to move budget.

The chart on Roku's page is an image file named Screenshot 2026-08-14 at 12.20.35 PM. A screenshot, pasted into a HubSpot blog four days before publication. There is no underlying table.

"Full panel." They wrote full panel.

A panel is recruited. A panel is balanced. A panel has a sampling frame you can hand to somebody. What they have is a list of customers who bought a thing, which is a fine thing to have and is not a panel.

You get one word like that per press release. This one has four.

Kowalczyk Ratio: 0.19. I'll explain how the ratio works when somebody explains the other five variables.

This is the part that does not require a source, a leak, or a methodology release. You can check it yourself in about ninety seconds.

On the six-month sustained-investment finding, Roku's post says gains of more than 20 percent above the average. Amazon's post says roughly 26 percent above that average.

Same study. Same finding. Same publication date. Two numbers.

There is an ordinary explanation. Two companies pulled from a shared deck at different refresh points and nobody reconciled before publishing. Which is also a measure of how much verification stood between the analysis and the audience.

Look at the branded search finding in both versions.

Roku, written by Anna Miller, Director of Ad Measurement at Roku Advertising, says that increasing Roku spend by at least 10 percent was associated with a 5.6 percent increase in branded search.

Amazon says that a minimum 10 percent increase in Roku spend drives a 5.6 percent lift in branded search.

Amazon's page metadata goes one further and describes the ads as driving 55 percent new-to-brand sales, which is not what the 55 percent figure measures at all.

The person whose job title contains the word measurement used the weak verb. The company selling the media did not. Both were describing the same regression output on the same day.

The measurement director over there has a psychology degree. Fine. Good, actually. Psych majors take research methods sophomore year. It's the class where they explain that two things happening on the same day is not one thing causing the other.

There's a whole week on it.

There's a quiz.

So she wrote "associated with." Correct word. The only word the number supports.

Then the company selling the media took the same regression, same day, and wrote "drives."

Nobody corrected it. Or somebody did and got told the asset was already approved.

Roku says the analysis controlled for nine variables, and then names four of them: other ad spend, price, promotion, and seasonality.

Amazon never says nine. It describes controlling for promotions, seasonality and other factors in one paragraph, and for similar total investment, similar audiences, and similar creatives in another.

Five of the nine controls are unnamed in every public version of this study.

There is also a wording difference worth noting. Amazon says the presence or lack of Roku CTV delivery was the only structural difference between the compared days. Roku says it was the key structural difference.

Only, versus key. One of those sentences claims an experiment. The other one does not quite.

I am not going to tell you these numbers were invented. The evidence points the other way.

A 5.6 percent branded search lift paired with a 6.4 percent detail page view lift is exactly as unimpressive as real regression output tends to be.

The 55 percent new-to-brand figure is a number somebody had to work to make sound good. And Roku's own measurement director hedged the verb.

The numbers are almost certainly real output from a real query against real data.

They also cannot mean what the sales page says they mean, and nobody had to lie for both of those things to be true at once.

Below, the two mechanisms that account for most of the gap, the five things Testwuide says a real version of this test would require and which of them Roku and Amazon published, and the sample question neither company has answered.

And then, for members, what happened when I went looking for people qualified to evaluate a Roku and Amazon study.

Two mechanisms account for most of the gap. Neither requires anybody to behave badly.

The first is flighting.

Brands do not turn CTV on at random. They turn it on for launches, promotions, seasonal peaks, and retail events, and they raise search and social budgets on the same days.

The study names seasonality and promotion among its nine controls. But the confound in a within-brand day comparison is not the calendar. It is intent. A brand deciding this is the week to push makes a hundred coordinated decisions, most of which never reach a media log.

Testwuide put it in one sentence. "Advertisers don't flight CTV at random; they flight it into promotions, launches and high-demand periods, and they raise search and social budgets on the same days."

And then the part that matters more than the mechanism itself: "The bias runs in the same direction as the claimed effect, which is why the number is large."

The second mechanism is the outcome metric.

Neither post defines ROAS. Testwuide flagged it before I asked.

"Also ask them to define ROAS," he wrote. "If the numerator is platform-attributed sales, running CTV mechanically creates more attributable touchpoints and the ratio rises with no change in what anyone bought."

The denominator, non-CTV spend, is held roughly constant by design. So the ratio moves on the numerator, and the numerator moves on attribution.

That is the most consequential undefined term in the study, and it is unanswered in both versions.

Nobody defined ROAS. That's the whole ballgame. You can't have a ratio if you won't say what's on top of it, and every one of these studies is a fraction with a shrug where the numerator goes.

Kowalczyk Ratio: undefined. See how annoying that is.

He listed five things. Here they are in his words, each one checked against what Roku and Amazon published.

"Geo markets split into dark and business-as-usual arms." Not present. The comparison is days, not markets, and the assignment is the advertiser's.

"The effect measured against the live control rather than against zero." Not present. Roku-off days are the baseline, and Roku-off days are not a control group.

"Per-dollar metrics so budget shifts can't pass for demand loss." Not present. The headline figure is a ratio.

"Delivery logs confirming the channel actually went dark." Not present. No verification of any kind is described.

"Outcomes pre-specified." Not stated anywhere, and there is no protocol, registration, or paper in which it could have been stated.

Five requirements. Five absences. Not disputed, not weakly implemented, absent.

There is a more basic problem, and it comes before any of the above.

Four hundred brands, an unstated number of days each, and no n by cell published anywhere. Nobody outside the two companies knows how many brand-days sit behind the 87 percent, how they distribute across those 400 brands, or whether a handful of large advertisers carry the average.

Nor is it stated whether the 87 percent is a mean of per-brand ratios or a ratio of summed totals. Those are different numbers. The first behaves badly when denominators get small, which on low-spend days is exactly what happens.

No interval is published. No significance test is published. No sample structure is published.

Compare that to Testwuide, who volunteered a p-value in an email.

Measured ran a go-dark co-movement analysis on a different CTV platform. Not Roku.

The design, as he described it: "Explore whether there is co-movement of Search/Social with CTV. In other words, if co-movement exists, then turning CTV off should visibly depress Search/Social activity."

It did. "Preliminary results show CTV does amplify non-brand search: CTV dark markets lost about 3.4% more search efficiency than control, in roughly 72% of our geo experiments, p = 0.011."

His stated conclusion: "We find that non-brand search co-moves with CTV, and the co-movement is statistically significant at the α = 0.05 level."

Two caveats came from Measured directly and are stated here at their request. The testing was conducted on another CTV platform, not Roku's, and it represents a single study with additional validation experiments currently underway. He called the results preliminary himself.

So the effect is real, detectable, and in the low single digits.

Against 87 percent.

I am not going to tell you how much of that gap each mechanism accounts for, because I cannot show the arithmetic and neither can anyone else without the underlying data. What can be said is that the gap runs to more than an order of magnitude, that both mechanisms push in the same direction as the claim, and that the study contains nothing that would rule either one out.

I also put the study to the chief executive of a CTV advertising platform, on background. His position, paraphrased: the halo effect from television is real, and a number in that range is not inconsistent with what his own company observes. He competes with Roku, which is a conflict I am disclosing rather than pretending away, and it cuts against the direction you would expect a competitor to push.

Which leaves the piece where the evidence leaves it. The effect is probably real. This design cannot support this claim. Roku's marketing depends on those two sentences sounding like one.

Three point four percent. Eighty-seven percent. Same phenomenon, and one of them had to be measured by somebody who wasn't selling it.

Fifty-five percent of sales during the study came from customers who had not purchased from that brand in twelve months.

There is no figure anywhere for what share of those same brands' sales come from new-to-brand customers on days without Roku. There is no category benchmark. There is no comparison of any kind.

Without a baseline, 55 percent is a description of a customer base, not a result of a media decision. Plenty of growing consumer brands on Amazon run north of half their volume from new customers as a structural fact of being a growing consumer brand.

Amazon's page metadata converts it anyway, into ads that "drive 55% new-to-brand sales." That is a causal claim assembled out of a descriptive statistic, in a field that no human wrote as prose, which is exactly where these things get away from people.

In the section arguing for sustained investment, Amazon offers a supporting example: a toy brand ran Roku for four months during the holiday season and drove strong seasonal volume.

No number is attached. No comparison is offered.

A toy company advertised in the fourth quarter and sold toys.

That is printed as evidence, inside a study whose stated control set includes seasonality. I do not think that is deliberate. I think somebody needed a fourth example and reached for the one on the slide.

That gap is the whole business. Somewhere there's a woman with a stats degree who wrote the honest verb, and somewhere downstream there's a guy in product marketing who decided the honest verb tested soft. Guess which one has the word measurement in her title. Guess which one won.

I know how that meeting goes because I have been in that meeting maybe two hundred times. Nobody yells. Nobody's evil. Somebody says "can we tighten this," which is the most expensive sentence in advertising, and then a hedge becomes a claim and everyone goes to lunch.

And then they published two different numbers for the same finding on the same day. Twenty and twenty-six. Two companies, one deck, nobody read the other guy's post before it went live. That's not a conspiracy, that's a Tuesday.

The boat's still in the garage. Sixth year. I fiberglassed the transom in March and I have not been out there since, and I don't want to hear about it.

Amazon's source line states a panel window of January 2025 through March 2026.

Two paragraphs above it, Amazon writes that since the partnership launched, it has analyzed 400 or more brands that activated Roku through the exclusive Amazon DSP integration.

That partnership was announced June 16, 2025.

There are innocent readings of this. Pre-launch baseline data is not just defensible, it is necessary if you want to know what those brands looked like before. A data pull window and an analysis window are frequently different things, and the source line may simply describe the former. It may also be a copy and paste from a query console.

But nowhere in either post is that distinction made, and as published, the panel window opens roughly five months before the product being measured existed. It is a question, not an accusation, and it is the first question I put to Roku.

Why join ADOTAT+

Because the free version tells you what's happening. The paid version tells you what actually happened, which is a different story most weeks.

You get the sourcing, not just the headline. Every investigation runs its full reporting behind the wall, the documents, the on-record quotes, the math that made a company stop answering emails.

You get Frank uncut. Free readers get the short version. Paid readers get the version where he actually finishes the thought.

You get to read things companies asked us not to publish. We published them anyway. That's most of the point.

Ads don't buy coverage here. We run them. They don't get a vote. Your subscription is what keeps that arrangement possible.

You're not funding a hobby. You're funding the two people who chase down thirteen unanswered questions until somebody finally answers eleven of them.

$99/month, or $600/year if you'd rather stop thinking about it. Cancel whenever. We won't be offended. We'll just assume you already knew the ending.