Lies, Damned Lies, and Statistics

Last updated on: December 23, 2014 at 6:42 pm by willduquette

Lies, Damned Lies, and Statistics 2014-12-23T18:42:45-05:00 willduquette

LOGO Patheos Explore the world's faith through different perspectives on religion and spirituality! Patheos has the views of the prevalent religions and spiritualities of the world.

Every day we read statistics about everything under the sun: “Eating a bar of chocolate every day reduces your chance of dying of acute hydrophobia by 23%”, or “People who are more than 20 pounds over weight are 90% more likely to die during cataract surgery.” We see them, and if we like them we share them with our FaceBook friends and if we don’t we harrumph and click on something else. We can’t avoid them, because statistics is the killer tool for virtually all research involving human beings. Sometimes we wonder if they are true; and we are right to do so. Today I want to tell a cautionary tale.

I was a math/economics major in college, and I took quite a bit of statistics (my advisor was a statistician by training). One of the classes involved something called “regression analysis,” where you gather data on a bunch of variables and then run a line through it. Suppose, for example, that you think you’ve found a relationship between the amount of honey produced in San Bernardino County in a given year and the number of cellular telephone towers. You can gather year by year numbers for each, and then fit a line to the data; and maybe the line says that for each new cell tower, honey production drops by five quarts. So far, so good. Regression analysis is a useful tool, and will absolutely give you the line that best fits the data.

But is it true? Does it mean anything? There are a number of statistical tests that you can do, tests that depend on the degree of error in your measurements. Everyone who uses regression analysis is taught to look at the “t-scores”; and if the t-scores are too low, you’ve got nothing.

The final project in this class was to find a published journal article that used a regression model, and to try to replicate it by running the data ourselves. I found such a paper in an economics journal; and I contacted the author, whom I will call Dr. X. The good doctor was gracious enough to send me his data set. (This was before the web; he had to send me a big old tape through the mail.) I set up the regression problem using a software package called SHAZAM, ran it, and lo and behold! I got exactly the same results, right down to the t-scores, which were good and strong.

But here’s the problem. The validity of those t-scores depends on a number of statistical assumptions. If the assumptions are met, you can trust the t-scores. If they aren’t, the t-scores are meaningless. Our professor, Sue Feigenbaum, hammered the assumptions into our heads, and insisted that we check them. The first check was for a condition delightfully named “heteroskedasticity,” which I can explain if anyone wants me to. It’s an obvious test to do, and I did it; and lo and behold! The model exhibited rampant heteroskedasticity, and all of the pretty results were suspect. So I corrected for it, which is often possible; and lo and behold! All of the pretty results were gone. Nothing. Nada. Zilch. In short, the paper was a travesty, proving nothing, and adding nothing to the sum total of human knowledge.

I don’t know whether Dr. X checked for heteroskedasticity or not; his paper didn’t say. But either he didn’t check, or he checked and then ignored the problem. I’d prefer to think the former; but I’ll note that this article was based on Dr. X’s doctoral dissertation, which he surely had to defend before a panel of his predecessors; and his professors didn’t catch the error either.

I’d promised to send him a copy of my write-up; and I spent an hour or so trying and failing to write a cover letter that didn’t accuse of him being either a fool or a liar. Eventually I gave up, shoved a copy of my paper in an envelope, and sent it to him with no cover letter. (I never heard back from him, unsurprisingly.)

A couple of years later, while getting my masters at Stanford University, I had to take a class on how to use a particular statistics package that was available for student use. The class taught statistical methods as a cookbook: if you’ve got this kind of problem, you put the data in here, turn the crank, and get your data out. There was no discussion of correctness, so far as I recall; the presumption was that everything was just going to work. The text, and the instructor, recommended to us several techniques for “getting a good fit” that Sue Feigenbaum had specifically warned us about because they biased the results.

I’d like to think that Dr. X had learned his statistical techniques in a class like this one rather than a class like Sue Feigenbaum’s. And I think of all of the sociologists, psychologists, and research physicians who publish studies every year, and I ask myself, how many of them learned statistics as a cookbook? They aren’t statisticians; most of them probably learned just enough to get by. And of those who do know better, how many are above overlooking problematic assumptions if it means getting published?

Using statistics is hard. Even when you know precisely what you are doing, it’s easy to fool yourself, as Michael Flynn has noted on his blog. If you’re using a cookbook, you don’t even need to fool yourself. And if you’re dishonest, all bets are off.

And just looking at the study in a journal, or (much worse) in a popular publication, you have no way of knowing whether the statistics were done right or not. My guess: they probably weren’t.

Deep Thoughts from the Doghouse

regression analysis

statistics

Previous Post

January 12, 2014

Divine Office, Part III: Fruit for Meditation

Next Post

January 13, 2014 Crusader Cooties!

Browse Our Archives

Select a Category Admin App o' the Morning Ask the Dogmat Believing Thomas Bits of Fiction Blasts from the Past Bootstrapping the Interior Life Catholic Mom Christian Living Deep Thoughts from the Doghouse Dominicana Evangelii Gaudium Interviews It Was A Dark and Stormy Night Links to This and That Lumen Fidei Meme Watch On Writing Operation Gratitude Pedant Errant Photography Quill Random Barkings S'Mary's World Shakers of Salt Software Development The Dog Whistler The Dogmat Sez Third Orders Thomas Aquinas Uncategorized Vikings at Dinos Woof Guide to Cinema Woof Guide to History Woof Guide to Literature Woof Guide to Obscure Catholic Music Woof Guide to Philosophy Woof Guide to Popular Music Woof Guide to Technology Woof Guide to Theology Words I Wish I'd Written Works For Me

Select a Month September 2016 August 2016 June 2016 May 2016 April 2016 March 2016 February 2016 January 2016 December 2015 November 2015 October 2015 September 2015 August 2015 July 2015 June 2015 May 2015 April 2015 March 2015 February 2015 January 2015 December 2014 November 2014 October 2014 September 2014 August 2014 July 2014 June 2014 May 2014 April 2014 March 2014 February 2014 January 2014 December 2013 November 2013 October 2013

TAKE THE

Religious Wisdom Quiz

How did Jeremiah's scribe escape death after Jehoiakim sought to arrest them?

Select your answer to see how you score.

They paid a bribe They fled to Babylon They hid A prophet intervened