From The Desk of...The Chief Scientist

"Statistics Brought to You by the Letter P"

Written by Paul Sutter on Sunday, 18 February 2018. Posted in From The Desk of...The Chief Scientist

Last week I mentioned an odd term, p-value, which is commonly used in deciding whether your results are worth mentioning to your colleagues and the public. Of course it has a strict and narrow meaning, and of course that meaning is abused and misinterpreted in discussions about science.

Let's say you're performing an experiment: a pregnancy test. The box claims 95% accuracy, and if you read the fine print it's referring to a p-value of 5%. You take the test's positive! So are you really pregnant, or not?

Unfortunately your urgent question hasn't yet been answered. A p-value compares the hypothesis you're testing ("I'm pregnant") to what's called a null hypothesis (in this case, "no baby"), and a p-value of 5% says that if you were *not* pregnant, there would only be a 5% chance that the test would return a positive result.

You might be tempted to flip this around and state that there's a 95% chance you're actually pregnant, but you would be committing an egregious statistical sin - and this is the same sin committed wittingly or unwittingly by science communicators and sometimes scientists themselves.

Here's the problem: what if you're male? The test can still say you are, because it's not answering the question "am I pregnant?" but rather "if I'm not pregnant, what are the chances of the test returning a positive result?" It's a low number - 5% - but not zero. Thus males can still get a positive result despite never being pregnant.

The p-value by itself was only ever intended to be a "let's keep digging" guidepost, not a threshold for believability. To answer the question you actually want answered, you have to fold in prior knowledge. Combined with a low p-value, a healthy female of reproductive age can begin to conclude that there might be a baby on the way. A male...not so much. In either case, the p-value alone wasn't enough, and announcements based solely on that number need to be viewed suspiciously.

"P-hacking the System"

Written by Jaclyn Reynolds on Monday, 12 February 2018. Posted in From The Desk of...The Chief Scientist

Science is hard. Scientists have to stare at mountains of data and try to figure out what secrets nature is whispering to them. There are innumerable blind alleys, dead ends, and false starts in academic research. That's life, and that's why over the centuries we've developed sophisticated statistical techniques to help lead us to understanding. But if you're not careful, you can fool yourself into thinking there's a signal when really you've found nothing but noise.

The problem is in correlations, or when two variables in your experiment or observation seem to be related to each other. Uncovering a correlation is usually the first step in "hey I think I found something," and so many researchers report a connection as soon as their experiment reveals one.

But experiments are often exceedingly complex, with many variables constantly changing - sometimes under your control and sometimes not. If you have, say, twenty variables that are all totally random, then by pure chance at least two of those variables will be correlated.

So when scientists fail to spot the correlation they were looking for, sometimes they start digging through the data until something pops up. And when it inevitably does - publish! But it was just a statistical fluke all along.

This practice is called "p-hacking", for reasons I'll get into another time, and it's a prime source of juicy headlines but faulty results.


Written by Paul Sutter on Wednesday, 07 February 2018. Posted in From The Desk of...The Chief Scientist

Once again there's a fresh round of blaring headlines and nervous chatter about potential links between cell phone use and cancer, this time based on a recent study purporting to show that a group of rats exposed to radio waves had greater incidents of tumors compared to a control group.

The radiation emitted by cell phones is not at the right frequency to ionize, excite, or even vibrate the stuff you're made of, so there's little to provide a causal link to cancer. So the standards of evidence for a study like this ought to be incredibly high. And then you read the study and find that the radiation-exposed rats lived longer, as a group, than the control rats. Since cancer rates correlate with age, something tells us that the research was not conducted well.

Sometimes scientists are their own worst enemies. There are the genuinely unscrupulous ones, willing to lie and cheat to advance their careers. And there are scientists who are, let's face it, inept at experimental design and statistical analysis. While we'll always have to be vigilant against those characters to make progress, they are thankfully in the minority and we can assume that most researchers, including the ones behind this study, have good intentions and are decent at their jobs.

But there are flaws in the modern scientific system that we must acknowledge. The unrelenting pressure to publish and the exhausting lifelong chase for funding create incentives for poor research to make it into journals and into the public discourse, muddying the waters and hurting science in the long run. This is especially harmful in fields that study extremely complex systems, like epidemiology, where good statistics are naturally hard to come by.

Scientists are just fighting for their careers, but when we need another round of discussions to (re)explain how to spot mistakes in metholodogy, something needs to be fixed.

<<  3 4 5 6 7 [89 10 11 12  >>