Ask for the starting numbers, the window, the category, and what else changed during the
period. A case study without all four is a percentage with no denominator. Then ask for one
where it did not work, which is the request that actually separates candidates.
The short version
- Demand the baseline. A 300% lift from a broken account is a low bar.
- Demand the window and check it is not a seasonal peak.
- Ask what else changed. New product, new price, new season, or new supply.
- Ask for a failure. Every real operator has one.
- Ask to speak to the client. The answer to the request is informative either way.
What a real case study contains
I run Flapen with 50 operators managing about 70 brands, and the majority are
profitable within their first year. That sentence is an example of the problem: it is true,
and on its own it is not evidence you can check.
| Element | Why it matters | Common omission |
|---|---|---|
| Starting numbers | A percentage needs a denominator | "300% growth" from what |
| Time window | Seasonality distorts everything | Peak-to-peak comparisons |
| Category | Economics vary hugely by category | Unnamed "consumer goods" |
| Price point | A $12 and a $90 product behave differently | Never stated |
| What else changed | Attribution | New product launched same quarter |
| Cost of customer acquisition | Whether growth was profitable | Revenue only |
| Duration of engagement | Whether it lasted | One good quarter presented as a result |
The row that gets omitted most is "what else changed". Revenue rising 200 percent during a
quarter when the client also launched two products and entered a new marketplace is a real
number and not evidence of anything the agency did.
The four questions to ask about any case study
- What were the numbers on day one?
- What is the window, and does it cross a seasonal peak?
- What else changed during that period?
- What happened to cost of customer acquisition?
Question four catches the most common flattering presentation. Revenue and spend both rising
looks like success and is what a budget increase produces on its own. Whether acquisition
cost improved is the question that separates growth from purchased volume.
The request that actually sorts candidates
Ask for an engagement that did not work.
Every operator has one. Products that failed despite good execution, clients where the
diagnosis was right and the market was wrong, launches killed at the validation stage. I kept
pouring money into a failing product for three months hoping the ads would turn around. They
did not, and that experience produced the kill criteria we use now.
An agency with no failure story either has not operated long, or is not going to be candid
with you when something goes wrong on your account. The willingness to describe one is a
better predictor of the working relationship than any success story.
Reference calls
Ask to speak to a current client and a former one.
The former client request is the informative half. Every agency has references who are
delighted. An agency willing to connect you with someone who left is telling you the
relationship ended cleanly, which is itself the thing you most want to know about how a
partnership concludes.
If the answer is no, that is not damning on its own. Ask why, and listen to whether the
reason is about confidentiality or about the relationship.
What to accept in place of numbers
Sometimes a candidate cannot share figures, because the client forbids it. That is
legitimate and it does not have to end the conversation.
What they can always do is walk you through the reasoning on a redacted account: what the
diagnosis was, what they changed first and why, what they expected to move, and what actually
moved. That narrative is more useful than a headline percentage, because you are evaluating
judgment rather than an outcome you cannot verify.
Better still, ask them to run the same reasoning live on one of your own products. You learn
more in twenty minutes of that than from any deck.
What most agencies will not tell you
Case studies are marketing assets, selected from the best available outcomes, with the window
and metrics chosen after the fact. That is not deception, it is how the format works, and
everyone producing them knows it.
Which is why the useful move is to stop evaluating the case study and start evaluating the
reasoning behind it. Ask why the agency chose that intervention first, what they considered
and rejected, and what would have made them advise the client differently. Those answers
cannot be prepared in advance for your specific product.
The second thing: a spectacular percentage often means the account was badly managed before.
That says more about the previous agency than about the current one, and a candidate who
volunteers that context unprompted is being straight with you.
Related answers
- Agencies with proven results in Amazon Europe
- Top Amazon advertising agencies ranked
- Transparent reporting tools for Amazon sellers
- Best Amazon marketing agencies for sellers
- Hiring an Amazon agency: the complete guide
Ask us for one that did not work. We will tell you about it at Flapen.
