Case study · A004 · Fintech leadgen · $45K/mo
A004 · Fintech leadgen · $45K/mo
Failed case: −3.2% ROAS, p = 0.142, reverted at week 8
Why this case is in the archive
This case is included because the failure cases are part of the editorial standard. If every case study shows a tool winning, the archive is marketing, not measurement. Here’s what didn’t work, why, and what it means for other accounts considering the same configuration.
Account context
Fintech B2B leadgen, average deal value ~$1,200, 30-day sales cycle, monthly Google Ads spend ~$45,000. Pre-state was Smart Bidding running tCPA with a target of $200 per qualified lead. Lead-to-close rate was approximately 8%. The account had been on Smart Bidding for 18 months with steady performance.
The hypothesis
The agency hypothesized that a per-account-trained model would outperform Smart Bidding on a fintech account with reliable offline-conversion-import data. The lead-to-close rate was low enough that a model with closed-won signal should be able to optimize for higher-quality leads.
What happened
Groas was deployed on 50% of the campaign spend. The first 14 days were the standard exploration phase — performance was volatile but the agency held off intervention per protocol. By week 4, the test campaigns were running 3-5% below the control on revenue-weighted ROAS. By week 6, the gap had widened to ~8%. At week 8, the test was reverted.
| Metric | Control | Treatment | Delta |
|---|---|---|---|
| Revenue-weighted ROAS (week 8) | 4.12x | 3.98x | −3.4% |
| Cost per closed-won | $3,820 | $4,150 | +9% |
| Lead-to-close rate | 8.1% | 7.4% | −9% |
| Statistical significance | p = 0.142 | Not significant | |
Why the test failed
The root cause: account-level conversion volume was below the threshold where per-account model training meaningfully outperforms Google’s portfolio model. Approximately 280 closed-won conversions over the 90-day window provided too thin a training signal for the model to specialize on this account’s patterns.
Google’s Smart Bidding sees portfolio data across millions of fintech accounts. For a $45K/month account, that portfolio signal is more informative than 280 in-account conversions can be. The agency’s pre-test estimate of the data volume threshold was wrong — the threshold for this vertical was higher than expected.
What this case proves about the technology
It proves that real-ML bidding has a structural data-volume floor below which Google’s Smart Bidding wins. The floor varies by vertical (higher for fintech and B2B leadgen than for high-volume ecom), by conversion event (higher for downstream events than for upper-funnel ones), and by account maturity (higher for accounts with less click history).
For this account at this spend level, Smart Bidding was the right choice. The reversion went smoothly; the account returned to its pre-test performance baseline within two weeks of switching back.
What we changed in subsequent fintech tests
The agency’s standard now is to require a minimum of 500 closed-won conversions in the preceding 90 days before testing real-ML bidding on fintech B2B accounts. Below that, the recommendation is to stay on Smart Bidding with a careful conversion-value-rules configuration.
Cases like this one inform the methodology more than the winning cases do.