Chasing that **
Posted by Evan Herrnstadt on July 3, 2008
I do a lot of econometric work, and I rarely post on it. This is mostly for the benefit of the audience, who may not want to read about the Bayes Information Criterion or Maddala and Wu’s panel unit root test.
Inspired by a post at EnvEcon about Dierdre (then Donald) McCloskey’s take on the academic life, I wanted to post a paper from the Journal of Economic Literature that is generally applicable to applied econometrics in all subfields. I’ve found McCloskey an interesting figure, not just because she was, until 1995, “Donald”. Her pursuits are wide-ranging and interdisciplinary, covering history, economics, feminism, law, rhetoric, and philosophy. From McCloskey’s bio:
I describe myself as a postmodern free-market quantitative rhetorical Episcopalian feminist Aristotelian woman who was once a man.
I first discovered McCloskey while reading David Colander’s The Changing Face of Economics, which is a fascinating series of interviews with unorthodox (I wouldn’t necessarily say that all were heterodox) economists, such as ecological economists and behavioralists. I took notice of McCloskey partly due to her interesting work and partly because she used to teach at my alma mater, the University of Iowa (for which she had some harsh words). Anyway, one of her major points on the rhetoric of economics is covered in her JEL article from 1996 with Stephen Ziliak entitled, “The Standard Error of Regressions.”:
That is, most beginning econometrics books even now, unlike DeGroot and Goldberger and before them the modern masters of statistics, do not contrast economic and statistical significance…
…The student from the outset of her statistical education, therefore, is led to believe that economic (or suvstantive) significance and statistical significance are the same thing. Hoel explains: “This word ['not significant'] arises from the face that such a sample value is not compatible with the hypothesis and therefore signifies that some other hypothesis is necessary” The elementary point that “there is no sharp border between ‘significant’ and insignificant’, only increasingly strong evidence as the p-value decreases” is not found in most of the earlier books from which most economists learned statistics and econometrics.
McCloskey and Ziliak go on to survey papers from AER to see whether these issues of interpretation show up in practice. An appalling proportion of papers gravely misused the concepts of significance in some way; thankfully, the problem seemed to be diminishing: misuse was decreasing in PhD vintage and the sample is from the 1980s.
I find myself in a lucky situation where economic significance is generally emphasized, as policy-related applied work must be set in context. However, there are times when I feel like I’m chasing that elusive asterisk for p < 0.05 (or even better, two of ‘em for p < 0.01). I was recently at a presentation where a scholar gave a result with p = 0.06, and he played it down as considerably less significant than a coefficient with p < 0.05. I’ve also had the experience of presenting economically meaningful coefficients significant at 0.05 < p < 0.10 and being told that they might be worthwhile but in effect were ignored. I realize that confidence is important, but I feel a type I error happening with a 1 in 10 probability is not a reason to ignore economically signficant coefficients. When you think about the somewhat arbitrary nature of what passes for statistically significant and what does not (and, thus, seriously influences what gets published and what does not), it kind of puts a damper on the whole “applied economics is a science” thing. Not to underplay the extremely useful nature of these methods, but I’m once again reminded that metrics has an artistic flourish to it.