A usable case study states the starting position, the exact interventions, the time window, the spend and the counterfactual. Without those five, a chart is decoration. Ask for the failures as well, because a portfolio with no losses means you are being shown survivors and nothing else.
The short version
- Percentages without absolutes are unreadable. A 400 percent lift from eleven units a month is eleven units and change.
- Name the intervention. "Optimized the listing" is not an intervention, it is a category.
- Fix the window. Any twelve week slice of a seasonal product can be made to look like genius.
- Demand the counterfactual. What would have happened anyway, and how do they know.
- Ask which of the five traffic channels moved. Organic, paid, promotions, influencer and off channel behave differently and most operators run only two.
You are reading the wrong document
If you are staring at a deck of before and after screenshots trying to decide whether an agency is good, you have been handed a marketing asset and asked to treat it as evidence. That is not your fault, it is how the category sells. The fix is to send your own template in advance and ask for it to be filled in. A firm that can complete the table below has done the work. A firm that cannot will send you the deck again.
The template, field by field
| Field | What it must contain | Why it matters |
|---|---|---|
| Starting position | Absolute units, revenue, conversion rate, rating, ad spend | Turns percentages into something real |
| Category and market size | Annual category revenue and growth trend | A rising tide is not a result |
| Interventions | Each specific change, dated | Separates work from coincidence |
| Traffic channels moved | Which of the five, and in what order | Shows whether the lift was bought or built |
| Time window | Start and end dates, plus the same window a year earlier | Kills seasonality games |
| Spend | Media budget and fees over the window | ROI needs a denominator |
| Outcome | Units, revenue, contribution margin, not just revenue | Revenue can be bought at a loss |
| Counterfactual | What the category or the rest of the catalog did | The single most revealing field |
| What did not work | At least one failed test | Proof somebody was actually measuring |
Failure modes, ranked by what they cost you
1. The seasonality slice
The most expensive one, because it survives scrutiny. A case study running September to December on a gifting product shows a curve that would have happened with nobody managing the account. Cost: an entire year of fees before you notice. Defense: ask for the same window in the prior year, from the same report.
2. Percentage without base
A 380 percent revenue increase reads as transformation until you learn the product was doing $900 a month. Cost: hiring for a skill set that has never been tested at your scale. Defense: absolute numbers in every field or the study does not count.
3. Revenue lift bought with ad spend
Sales up 60 percent, ad spend up 140 percent, margin down. This is the most common presentation in the industry and the easiest to produce deliberately. Cost: a year of profitable growth traded for unprofitable growth. Defense: contribution margin per unit, before and after.
4. The unnamed intervention
Vague verbs, no dates. Nobody can tell you which change caused the result, which means nobody can repeat it on your account. Cost: paying for a process that does not exist. Defense: a dated list, and a question about which of the five traffic channels was activated.
5. The survivor portfolio
Every case study wins. Real portfolios contain products that were killed, and the decision to kill one is often better work than a launch. Cost: hiring a team with no discipline about stopping. Defense: ask for the case study of something they shut down and what the criteria were.
Where our own numbers sit
I would rather hand you a benchmark you can hold anyone to, including us. Across about 70 brands managed by 50 operators, the majority reach profitability within their first year. That is a portfolio level claim, not a hero case, and it is the kind of number worth asking every candidate for, because it includes the ones that did not work.
What a case study deck will not tell you
Who did the work. A study describes an account, not a team, and the person who produced that result may have left, or may now be carrying nine other brands. Ask who ran it, whether that person is still there, and how many accounts they carry today. The answer predicts your experience far better than the chart does.
The second omission is the starting quality of the asset. Some accounts improve dramatically because they began badly, and rescuing a neglected listing is easier than growing a well run one. If your account is already competent, a portfolio of rescues tells you very little about what happens next.
Related answers
- Rank Amazon agencies by case studies and ROI
- Case study proof to request from Amazon partners
- KPIs to track for an Amazon agency engagement
- What does a good Amazon account audit include
- Hiring an Amazon agency: the complete guide
Send us that template filled with your own account, and we will return the audit at Flapen.
