I recently came across this on my Facebook feed:
It is a great example of what happens when you allow a headline to make you curious, about methodology as well as topic. Dig deep, and you may uncover misleading, questionable, or just confusing decisions made by researchers and people reporting on the research. Sometimes this is deliberate, an attempt to influence the policy decisions of people who don’t have time to understand research.
However, in order to get appropriate attention, portions of almost every empirical article you see will be hyped. The introduction, sometimes, the abstract, sometimes the discussion too, will inevitably exaggerate what the researchers actually found. Those sections are designed to say “Hey, if our interpretation is correct, these results will be really interesting.”
A social scientist reads the abstract and skips right to the manipulations (what the experimenters presented differently to different people) and the measures (what they measured). That’s the concrete, meaty detail. That’s the difference between being told a movie fits in a certain genre and watching the actual movie.
Then, we look at the statistics, acknowledging that these estimates (models designed to uncover an average trend amidst all that variation) are the best they researchers could do with the tools they had. (Let’s assume they were researching in good faith). We ask questions about what they didn’t report (word limits), what they might have found but not been able to tell a clear story about (there’s a lot of messiness, even scientists like to read about clarity).
If we’re really good at statistics, we may even look for mistakes. Peer reviewers who act as gatekeepers for academic journals are generally unpaid and overwhelmed. Mistakes happened.
Then we check out the discussion (the “let’s get real” section of the paper). Then we skim the intro for any novel interpretations of the existing literature, or citations we were unfamiliar with.
This is a good approach, not just with academic papers, but with anything. What does the real evidence look like? Are people interpreting it in good faith? Are they making errors you can help correct? What other interpretations could you offer?