Skip to content
Vanity or Vice

Claims & Evidence

A Pile of Studies Is Not a Verdict: How to Audit a Beauty Evidence Stack

A citation count is not an evidence grade. Audit beauty studies for directness, comparator, endpoint, magnitude, population, duration, burden, replication and whether the marketing claim still fits the actual result.

Last updated

Scientist pipetting colored solutions in a laboratory

“Backed by 22 studies” sounds reassuring until you discover that nineteen of them never tested the product being sold.

A bibliography is not a verdict. It is a pile of evidence objects with different jobs. The useful work is figuring out which ones actually touch the commercial claim.

METHOD · EVIDENCEWhere the proof is standing.Systematic reviewRandomized trialSmall human trialCase seriesAnimal modelIn vitroWHAT A VERDICT WANTSPeople, randomized, followed longenough for the effect to be realrather than new.WHAT THE BOX USUALLY CITESReal research, but a long way fromyour face. Useful as a reason tolook, never as proof.Nothing here is worthless. The question is only ever how far it is from the promise on the label.
METHOD · EVIDENCEWhere the proof is standing.
  1. Systematic reviewWhat a verdict wants: people, randomized, followed long enough for the effect to be real rather than new.
  2. Randomized trialWhat a verdict wants: people, randomized, followed long enough for the effect to be real rather than new.
  3. Small human trialWhat a verdict wants: people, randomized, followed long enough for the effect to be real rather than new.
  4. Case seriesWhat the box usually cites. Real research, but a long way from your face — a reason to look, never proof.
  5. Animal modelWhat the box usually cites. Real research, but a long way from your face — a reason to look, never proof.
  6. In vitroWhat the box usually cites. Real research, but a long way from your face — a reason to look, never proof.
Nothing here is worthless. The question is only ever how far it is from the promise on the label.
Where the proof is standing. The ordinary hierarchy of study design. Marketing tends to cite the lower half and describe it in the language of the upper half; the gap between the two is where most of the work is.

This is the part of beauty research I genuinely enjoy: following the citations until the impressive pile turns back into its original handful of studies. I am the person who will keep clicking because I want to know whether twenty-two references are twenty-two answers or one answer wearing a crowd.

Start with directness, not volume

Write down what each source studied: the exact finished product, an ingredient, a prototype, a device category, a biological mechanism or something merely adjacent.

One good controlled trial on the actual formula may tell you more than twenty papers explaining why one ingredient is interesting. Relevant and direct are not the same word.

Find the comparison

Improvement from baseline is not the same as outperforming vehicle, placebo, sham, an untreated side or an active comparator. The right comparison depends on the claim, but without one you should be careful about assigning all change to the intervention.

Ask the expensive question early: better than what?

Match the measured endpoint to the marketing sentence

A study may measure hydration, wrinkle depth, pigment, redness, elasticity, investigator ratings or participant satisfaction. The sales page may translate that into “rejuvenation,” “repair,” “lift” or “younger-looking skin.”

Put those two sentences beside each other. If the study measured a narrow endpoint and the marketing implies a broad transformation, the gap is part of the review.

Magnitude matters more than a lonely p-value

A statistically significant difference can be real and still be too small to matter to the person paying for it. Look for the size of the change, the comparator, confidence intervals where available and whether the effect is likely to be visible or useful.

The consumer question remains stubbornly practical: how much changed, compared with what, and is that change worth the price, time, irritation, downtime or maintenance?

Read the methods before the adjectives

Sample size, allocation, controls, randomization, blinding where feasible, duration, prespecified outcomes and missing data all affect confidence. So does attrition. If people leave because the protocol is irritating, inconvenient or ineffective, the final group can look cleaner than the real experience.

A narrow sample is not automatically a bad study. It supports a narrower conclusion.

Then check the clock. An immediate hydration study establishes an immediate hydration effect. It does not prove a durable structural change six months later.

Likewise, a product tested under supervised application, exact dosing and unusually high adherence may perform differently when it reaches a bathroom at 10:47 p.m. If the protocol helped create the result, the protocol belongs in the verdict.

One research lineage can look like a crowd

Reviews can cite the same small trials. Several papers can come from the same sponsor, research group, dataset or protocol. A brand page can then cite the reviews and make one lineage look like a consensus.

Independent replication is not the only evidence that matters, but it is different evidence.

Funding works the same way — as context, not as an automatic conviction. Industry-funded research can be well designed. Independent research can be weak. Record funding, author affiliations, sponsor involvement and disclosure, then read the methods anyway.

The correct response to a commercial relationship is scrutiny, not a ceremonial eye roll.

Put burden beside benefit

A study that shows improvement but requires frequent irritation, complicated adherence, substantial downtime or ongoing maintenance has a different practical value from a similar result with little friction.

The evidence file is incomplete if the benefit column gets a full page and tolerability gets one sentence.

Eight things break an evidence stack, and they break it in order: the object studied, the comparator, the population, the endpoint, the magnitude, the duration, the burden, and finally the translation into a marketing sentence. Find the first one that fails and you can usually stop reading — everything above it was built on that floor.

For the claim itself, use How to Read a Beauty Claim Before You Trust It.

Count studies last

Ask what each source permits you to believe, then see whether the number still impresses you. “Promising for short-term hydration; durable wrinkle claims remain uncertain” is more useful than a victory lap around a reference count.