Not All Evidence Is Created Equal

We live in an era of unprecedented access to health information — and unprecedented confusion about what to believe. In a 2023 YouGov survey of US adults, 52% of respondents aged 18–34 used social media to learn about medical treatments 4. This measured use of an information source, not whether it replaced medical advice. Online, influencers may promote conflicting rules — carnivore, keto, seed-oil avoidance — backed by testimonials or selected studies.

A personal story is not a clinical trial. A single study is not a consensus. A confident presenter is not a substitute for decades of rigorous research.

This article is a practical guide: how to read an abstract, spot the statistical tricks used to mislead, and apply a five-step checklist before you change anything about your diet.

The evidence pyramid introduces common study designs; it is not an automatic quality rating. Certainty also depends on study limitations, consistency, directness, precision and publication bias 5.

An anecdote describes one person and cannot establish what caused the outcome. Observational studies can reveal associations and provide important evidence about long-term outcomes, but confounding must be considered. Randomization reduces confounding on average; it does not remove every source of bias. Systematic reviews assess a body of research, and meta-analyses may pool randomized or observational studies. Their reliability depends on the included evidence and methods 5.

Apply the same appraisal questions across social media, industry-funded studies, health headlines and expert recommendations. This is practical editorial guidance informed by evidence-appraisal principles 5.

Social media: Find the underlying study. Check whether its population, outcome and certainty match the post’s claim 5.

Industry funding: Inspect methods and disclosure. Historical sugar-industry documents and the beverage-article analysis show why scrutiny matters; funding alone does not prove a result false 12.

Health headlines: Compare absolute and relative effects. Distinguish health outcomes from surrogate markers, and read the limitations 5.

Expert recommendations: Ask which evidence supports the advice, what remains uncertain, and whether it applies to your circumstances 5.

An influential source is not automatically right or wrong. Examine study methods, uncertainty, relevance and funding together 25.

Funding deserves attention. A 2016 historical analysis of internal documents found that the Sugar Research Foundation sponsored a review that emphasized fat and cholesterol while downplaying evidence about sucrose and coronary heart disease; its funding and role were not disclosed 1. That analysis did not estimate deaths caused by the review. A separate analysis examined 206 articles about soft drinks, juice and milk, published from 1999 to 2003. Among articles declaring funding, exclusively industry-funded articles had higher odds of a favourable rather than unfavourable conclusion than articles with no industry funding (odds ratio 7.61; 95% confidence interval 1.27–45.73) 2. This association concerns those beverage articles, not every industry-funded study. Industry funding is a reason to examine methods and disclosure carefully; it does not by itself prove a result false. Check who paid and whether independent evidence supports the conclusion.

Use an abstract as a first pass, then read the methods and results when a decision matters.

1. Study design. Does the design fit the question? Randomization can reduce confounding; observational studies can address important questions that trials cannot readily answer 5.

2. Sample size and precision. Look at the confidence interval, number of events and study methods. There is no universal cutoff at 100 participants 5.

3. Duration. Does follow-up cover the outcome claimed? A short trial alone cannot establish a long-term mortality benefit 5.

4. Effect size. Ask for absolute as well as relative effects, and whether the measured outcome matters to patients 5.

5. Conclusion language. Compare the conclusion with the actual results and limitations. Cautious wording helps only when it accurately reflects the evidence 5.

Even a well-designed study can be presented to mislead. Here are three common issues to check.

Relative vs. absolute risk. For a hypothetical example, a treatment that cuts heart-attack risk from 2% to 1% can be reported as "halves heart-attack risk" (relative: 50%) or "reduces risk by 1 percentage point" (absolute). Both are true; they feel completely different. Always look for the absolute number.

Statistical vs. clinical significance. With enough participants, a tiny effect can be "statistically significant" (p < 0.05) and clinically meaningless. A statistically significant change in a biomarker does not by itself establish a meaningful health benefit.

P-hacking. If 20 independent null hypotheses are true and each is tested at the 0.05 threshold, one false positive is expected on average; any particular set may produce none or several. Researchers under pressure test many endpoints and report only the positive few. Prespecified analyses and transparent reporting help reduce selective reporting, but do not guarantee its absence.

Two more concepts. Number Needed to Treat (NNT) is the number treated for one additional person to benefit over a specified period, compared with the control. Interpret it with the outcome, baseline risk, harms, burden and cost; no NNT is automatically excellent or useless 5. Confidence intervals give the range of plausible effects — a 30% benefit with a 95% CI of −5% to 65% includes "no benefit."

Use these warning signs as prompts for closer appraisal, not automatic proof that a claim is false.

No traceable evidence. Ask for the original source and how it supports the specific claim.

Vague “studies show” language. Request the named studies and their limitations.

Indirect evidence. Ask whether evidence from cells, animals or a different population is being extrapolated to people without justification 5.

Selective evidence. Look for a systematic account of relevant findings, including results that disagree 5.

Commercial interest. Check financial disclosures and research methods. A conflict warrants scrutiny but does not alone disprove a result 2.

“Natural” or “ancestral” appeals. Ask what relevant outcome evidence supports the recommendation.

Testimonials. Treat a personal story as a starting question; it cannot by itself establish what caused a change 5.

Use these five questions before acting on a health claim. They are an appraisal aid, not a validated scoring system.

1. Can you trace the evidence? Open the original source and check that it supports the claim.

2. Does the study fit the question? Assess methods and relevance, not the design label alone 5.

3. Who funded it? Read funding and conflict statements alongside methods and independent findings 2.

4. How large and certain is the effect? Look for absolute effects, uncertainty and outcomes that matter 5.

5. How does it fit the wider evidence? Consider consistency, missing evidence and relevant guidance 5.

Unresolved questions should prompt further checking and, for personal medical decisions, discussion with a qualified professional.

Our editorial standards are to make evidence and uncertainty clear:

Traceable sources. Link factual claims to the research or authoritative guidance that supports them.

The relevant body of evidence. Consider systematic reviews and individual studies in context; a review is only as reliable as its methods and included evidence 5.

Clear uncertainty. Distinguish established findings, emerging evidence and unanswered questions.

Appropriate study design. Assess whether the method fits the question, rather than treating a design label as an automatic guarantee 5.

Transparent conflicts. Examine funding and disclosures and explain limitations that could affect interpretation 2.

These are standards for appraisal, not a promise that every source or every conclusion is beyond criticism.

Use the references in this article as a starting point for your own checks. The Cochrane Handbook explains how to judge certainty and interpret study results 5. The 2019 EAT–Lancet Commission is an example of a broad synthesis addressing healthy diets and sustainable food systems 3.

For further reading, search original papers through PubMed, explore nutrition topics at NutritionFacts.org, and read the cited journal articles rather than relying on a journal name alone.

Apply the same questions to all of these resources: what evidence supports the claim, how certain is it, and does it apply to the decision? 5