From The Desk of...The Chief Scientist

"In Experts We Kind Of Trust"

Written by Paul Sutter on Monday, 17 September 2018. Posted in From The Desk of...The Chief Scientist

It's a delicate balance. We want communities to trust, respect, and understand science. But science is a method, a process. The people who practice it are fallible. The results they produce are provisional and incomplete (at best) or flat-out wrong (at worst).

How can people honestly trust a method that, by design, changes its collective mind? How can people honestly respect a process that is, by design, more often wrong than right? How can people honestly understand a philosophical approach that, by design, steeps itself in arcane mathematics and jargon?

Let's start with trust. When we ask people to "trust" a particular scientist or result, we need to make sure that it's not the person itself that is necessarily deserving of trust, but the method and structures that they represent.

Through painstakingly meticulous work, agonizingly slow timescales, and incessant revisions we come to ever more-refined descriptions of the world around us. And that is worthy of trust. Only when a particular result is placed in the proper context - motivations, current knowledge, scope of limitations, etc. - can we show communities what it means to trust scientists.

"Check Your Bias"

Written by Paul Sutter on Monday, 10 September 2018. Posted in From The Desk of...The Chief Scientist

I talk about bias a lot because bias is kind of important. And as if it weren't already difficult enough to constantly be on the lookout for the ways that biases can sneak and slither their way into a dataset or analysis or presentation, there's another source of bias that is much more pernicious and insidious. Thankfully it's surprisingly easy to find its source: just look in the mirror.

It's so easy. You go into an experiment or observation or study or analysis expecting a certain result. You can't help it, even when you're trying to be impartial; it's human nature. And as soon as the data start to lean in a particular direction, or the analysis starts to confirm your suspicions, it's oh so tempting to call it a success, write it up, and move on.

But this is what gets you into trouble. What if you're looking at a false positive? What if there aren't enough data to justify your statistics? What if you made a mistake somewhere and overlooked it because you got the answer you thought you wanted?

Replication is one of the keys to the scientific paradigm, but we can't be lazy about it and just assume that someone else will do the boring repetitions for us. Nobody will be as close to the original data and setup as we are - it's up to us to perform the first rounds of repeating and cross-checking results ourselves.

It's easy to lie with data and give everything a veneer of respectability. And the person we most often lie to is ourselves. So one key to cracking internal bias is simple and straightforward: get more data, and do the whole thing again.

"Getting Your Hands Dirty with Data"

Written by Paul Sutter on Monday, 03 September 2018. Posted in From The Desk of...The Chief Scientist

We always question everything, right? Right. But what if we're faced with a chart, or a graph, or a table of raw numbers? It's easy to fetishize the data, and assume that as long as we have data our arguments must be sound. But as I've talked about before, the data can lie - and more twistedly, the presentation of the data can lie even more. So here are some handy-dandy questions to ask yourself when confronted with data:

- Is the presentation of the data hiding something? Was anything excluded or minimized? Was anything glossed over? Was one part of the graph highlighted or emphasized to draw attention away from something else?

- How were the uncertainties calculated? You can't just pick error bars out of a hat - there's a methodology behind it, and that technique must be explained.

- What are the limitations of the data? What were they not able to cover? Does this introduce a bias?

- What choices were made to lead to the result? Are they explained? Are those choices reasonable or unreasonable?

- What choices were made in the presentation of the data? Percentages or absolutes? Linear or logarithmic? Broken axes? Is there a curve "fit" to the points? Are the data grouped?

- What are some other interpretations of the data?

This is, of course, far from a complete list, and each of these questions could be a note of its own. But it's a good start...

[12 3 4 5  >>