Significance

Evan’s post last week about statistical versus economic (I might call it actual) significance reminded me a recent set of articles in Wired magazine that announced “the end of theory“.  The article offered an interesting (if florid) take on “the Petabyte Age”, where huge volumes of raw data make the scientific method obsolete:

At the petabyte scale, information is not a matter of simple three- and four-dimensional taxonomy and order but of dimensionally agnostic statistics. It calls for an entirely different approach, one that requires us to lose the tether of data as something that can be visualized in its totality. It forces us to view data mathematically first and establish a context for it later. For instance, Google conquered the advertising world with nothing more than applied mathematics. It didn’t pretend to know anything about the culture and conventions of advertising — it just assumed that better data, with better analytical tools, would win the day. And Google was right. …

Scientists are trained to recognize that correlation is not causation, that no conclusions should be drawn simply on the basis of correlation between X and Y (it could just be a coincidence). Instead, you must understand the underlying mechanisms that connect the two. Once you have a model, you can connect the data sets with confidence. Data without a model is just noise.

But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. …

There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

As you can probably imagine, this prompted some discussion both among natural and social scientists.  (Both those links are worth reading, as are the posts they link to.)  My only specific comment on the Wired articles is that while they do a nice job exploring some fields where data is proving very powerful, the implicit premise seems to be that causation is dead — long live correlation! — and this is oversold.

The link to Evan’s post that I wanted to highlight was that it is exactly this abundance of data (combined with the powerful statistical tools we have to analyze it) that have made for such an intense focus on statistical significance in recent years.  This is completely appropriate when you are just mucking around in data looking for correlations.  After all, if I look at a set of 20 random variables (that have no relationship in reality) then on average I should should find one relationship between them that is significant at the p=0.05 level.  Remember, finding statistically significant correlation at p=0.05 means that there is a 1 in 20 chance that the correlation is just a random artifact.  Well, when you start piling through hundreds (or thousands) of variables and trying out scores (or hundreds) of specifications, random artifacts start showing up! (Everyone who actually understands statistics is now allowed to bang their head against their desk and correct my horribly sloppy and imprecise language in the comments.)

And this gets back to that difference that Evan was talking about.  When you are doing reduced form econometrics and not pretending you have any idea what the relationship between any of your variables should be, you had better show some pretty impressive t stats — and start coming up with a reasonably good post hoc argument — before I pay much attention to your paper.

On the other hand, if theory tells us that there is good reason to suspect that a set of variables share a causal relationship — if we have a model that we think represents reality — then it is quite a different thing to find that there is a 1 in 20 (or even 1 in 10!) chance that our correlation is just random.  When data analysis backs up a theoretical prediction — even with only 90 (or even 80!) percent confidence — then our belief in the theory should get stronger.

The upshot is that I’m not ready to declare that correlation is enough.  Data is great, but it’s a lot better when you combine it with some theory.

Chasing that **

I do a lot of econometric work, and I rarely post on it.  This is mostly for the benefit of the audience, who may not want to read about the Bayes Information Criterion or Maddala and Wu’s panel unit root test.

Inspired by a post at EnvEcon about Dierdre (then Donald) McCloskey’s take on the academic life, I wanted to post a paper from the Journal of Economic Literature that is generally applicable to applied econometrics in all subfields.  I’ve found McCloskey an interesting figure, not just because she was, until 1995, “Donald”.  Her pursuits are wide-ranging and interdisciplinary, covering history, economics, feminism, law, rhetoric, and philosophy.  From McCloskey’s bio:

I describe myself as a postmodern free-market quantitative rhetorical Episcopalian feminist Aristotelian woman who was once a man.

I first discovered McCloskey while reading David Colander’s The Changing Face of Economics, which is a fascinating series of interviews with unorthodox (I wouldn’t necessarily say that all were heterodox) economists, such as ecological economists and behavioralists.  I took notice of McCloskey partly due to her interesting work and partly because she used to teach at my alma mater, the University of Iowa (for which she had some harsh words).  Anyway, one of her major points on the rhetoric of economics is covered in her JEL article from 1996 with Stephen Ziliak entitled, “The Standard Error of Regressions.”:

That is, most beginning econometrics books even now, unlike DeGroot and Goldberger and before them the modern masters of statistics, do not contrast economic and statistical significance…

…The student from the outset of her statistical education, therefore, is led to believe that economic (or suvstantive) significance and statistical significance are the same thing.  Hoel explains: “This word ['not significant'] arises from the face that such a sample value is not compatible with the hypothesis and therefore signifies that some other hypothesis is necessary”  The elementary point that “there is no sharp border between ’significant’ and insignificant’, only increasingly strong evidence as the p-value decreases” is not found in most of the earlier books from which most economists learned statistics and econometrics.

McCloskey and Ziliak go on to survey papers from AER to see whether these issues of interpretation show up in practice.  An appalling proportion of papers gravely misused the concepts of significance in some way; thankfully, the problem seemed to be diminishing: misuse was decreasing in PhD vintage and the sample is from the 1980s.

I find myself in a lucky situation where economic significance is generally emphasized, as policy-related applied work must be set in context.  However, there are times when I feel like I’m chasing that elusive asterisk for p < 0.05 (or even better, two of ‘em for p < 0.01).  I was recently at a presentation where a scholar gave a result with p = 0.06, and he played it down as considerably less significant than a coefficient with p < 0.05.  I’ve also had the experience of presenting economically meaningful coefficients significant at 0.05 < p < 0.10 and being told that they might be worthwhile but in effect were ignored.  I realize that confidence is important, but I feel a type I error happening with a 1 in 10 probability is not a reason to ignore economically signficant coefficients.  When you think about the somewhat arbitrary nature of what passes for statistically significant and what does not (and, thus, seriously influences what gets published and what does not), it kind of puts a damper on the whole “applied economics is a science” thing.  Not to underplay the extremely useful nature of these methods, but I’m once again reminded that metrics has an artistic flourish to it.

The curse is broken?

Why hasn’t this been reported more widely?

The correlation between resource dependence and slow growth and conflict, therefore, does not imply causation from the former to the latter. Instead, causality appears to be running from weak institutions and conflict to resource extraction as the default sector, which produces resource dependence as the final outcome. Resource dependence appears as a symptom, rather than a cause of underdevelopment.

The authors argue that previous research on the ‘natural resource curse’ has been unable to correctly identify which way causality runs:

The standard resource variable used by Sachs and Warner, as well as by Collier and Hoeffler, is primary exports divided by a measure of national income. It thus captures the resource dependence of economies, rather than abundance. A negative correlation between this variable and growth could mean that resources lead to slower economic growth, as suggested by the curse proponents. Alternatively, it could mean that poor economic development policies–leading an economy to become dependent on its primary exports–dampen growth. Similarly, although a negative correlation between the resource variable and institutional quality may imply that resources undermine institutions, it might also capture that the resource sector is the “default sector” in the absence of decent institutions when nobody is willing to invest in alternative forms of capital. Finally, a positive correlation between the resource variable and conflict may indeed mean that resources trigger conflict. But it may also be the case that conflict makes countries dependent on resource extraction–the default activity that still takes place after other economic sectors (more mobile or, perhaps, better linked to the rest of the economy) have come to a stop. If so, resources are not a curse to development, but rather a safety net to support people and economies even under adverse circumstances.

They argue that economic dependence on natural resources is endogenous in exactly this way and use data on resource endowment — rather than resource exports — as an explanatory variable for economic growth.

When using the new World Bank variable to proxy for resource abundance, we find that the direct effect of resource wealth (particularly the subset of mineral resource wealth) on income growth is positive and significant. All things considered, an increase in subsoil wealth by one standard deviation–roughly the difference between Senegal and Sweden–would have brought Senegal’s growth performance on a par with that of Mozambique or Kenya; not a huge improvement, but certainly not a growth curse.

Similarly, resource wealth also attenuated the risk of conflict. This is due to a positive indirect effect: Resource wealth raises income, and higher incomes, in turn, reduce the risk of conflict. Again, although the aggregate impact of resource abundance is slight–amounting to less than a 5% reduction in the risk of war in case of a standard-deviation increase in resources–it is still statistically significant.

If this result holds up it will be a significant finding in development economics and could overturn almost 2 decades of conventional wisdom on the curse of natural resources.  The full text is here (gated); a non-gated version is here.

Assorted links

1. Agricultural Subsidies: Still a Bad Idea. Felix Salmon explains why removing ag subsides and taxing carbon are similar, and why they both make sense. Free Exchange squares the circle with a discussion of biofuels.

2. Who Pays a Tax? Tim Haab’s two-part series is here and here.

3. 6 Cities That Were Caught Shortening Yellow Light Times For Profit. What happens when your city stands to make money off of lawbreaking? Yep, that’s right, they make it harder to avoid breaking the law.

4. Malaria and the politics of disease. Efforts to fight malaria seem to be ramping up quickly. But even if near-term success can be achieved, will many be left worse off in the long run?

5. Congestion pricing works. Evidence from California.

6. Location, location, location. The premium for urban living.

7. The cost of siting transmission lines.  This came up yesterday in the seminar on curbing electricity demand at RFF as one of the key uncertainties in the future of electricity, given the political or economic forces that will bring new types of resources onto the grid in the coming years.  (Video from the event should be up in the next few days.)

More thoughts from the Weitzman event

That Marty Weitzman talk I attended last month not only featured Weitzman but also had some pretty amazing discussants including Richard Posner and Tom Schelling. There were plenty of juicy tidbits (besides the one I’ve already mentioned) and so I’ve gone through my notes and have tried to pull out a few of the highlights. There’s enough here that I’m putting it all below the fold…

Words of truth and challenge

In my view anyone doing policy economics has an obligation to learn more about ethics — much more — than the guy in the street would know. Would someone doing experimental economics feel free of the obligation to learn some empirical psychology? Would someone doing trade feel free of the obligation to learn some trade law, some history, and some political science? No. What’s the difference? Economists like to separate the “positive” and “normative” aspects of what they do, but this distinction has not much impressed the moral philosophers who have looked at it nor has it impressed Amartya Sen. The very decision to use economic tools emphasizes some considerations and excludes others. The final policy analysis is not just pure prediction but rather it is also an implicit presentation and weighting of both different kinds of information and different values. So if you are doing policy economics, it is imperative that you think about ethics at a very deep level, and read widely in ethics. You are doing ethics whether you like it or not!

That is Tyler Cowen. He is writing in the specific context of trade — rather than environmental — economics, but the point is a broad one. (Indeed, he links trade to environmental economics in his closing paragraph.) I was struck forcefully by his words. It’s easy to fall into the trap of explaining what the most “efficient” policy is, and then try to duck the distributional questions. Or assume that they are left to be negotiated by “society.” More fundamentally, as Cowen points out, using economics as a tool of policy analysis means weighting values and considerations, thus implying a set of ethical choices.

I feel challenged to try to more explicitly identify the weighted “priors” I bring to any question in environmental economics. This doesn’t mean I’m throwing out benefit-cost analysis as a tool for making policy decisions, but it does mean I’ll try to identify more clearly what influence those priors have on my analysis and be more open to discuss how alternative values could result in different conclusions.

I also started thinking about what texts would be considered required reading in ethics for economists (and perhaps particularly environmental economists). Here is a list from Cowen. Do readers have any suggestions of their own?

Can’t we all just get along?

Over on Gristmill Michael Tobis wonders, “should economists rule?”. While that’s clearly a loaded question, his real argument, that we should stop pursuing economic growth, is much more interesting. I’m going to post my response to this on Gristmill as well as below. Looking at the comments on there so far, I realize that this post could provoke a fair amount of rage and anger from those, like Tobis, who are disturbed by a perceived monopolization over public debate by economists of late. Though it probably won’t matter, I’d like to point out that I certainly don’t believe that we should be categorically opposed to non-economic arguments. In fact interdisciplinary dialogue is imperative for solving most policy questions, including those related to the environment. At the end of the day, what matters is how good our answers are, not how we got there. Hopefully we can use this space to narrow some of the economist/ non-economist divide. With that being said, here we go.