Wednesday, March 15, 2017

Where Should We Put Economic Empiricism on the Hubris-Humility Spectrum?

A bit of a kerfuffle has broken out over the claim that, as economics gets more empirical, it also gets more reliable.  Russ Roberts says that, in the name of empiricism, economists are trotting out contested results to adjudicate questions that are vastly more complicated than their methods can allow for, and that they should acquire a bit more humility instead.  Balderdash, says Noah Smith: whatever lack of humility is evinced by researchers who jump the gun in empirical work is swept away by the tsunami of hubris issuing from those who have only vague, unsubstantiated assumptions about how the world “really” works.  Cowen rebuts, arguing that motivated, hubristic reasoning can seize on empiricism just as readily as any other academic raw material.  Smith retorts, going halfway to meet Cowen, but expressing optimism that empirical methods carry their own antibody against motivated bs and will pull knowledge in a more realistic direction over time.

So what do I say?  (I thought you’d never ask.)  First, Roberts is simply recycling, for a forgetful age, the now-ancient claim of Hayek regarding the Fatal Conceit.  I agree that it is desirable to not be Fatally Conceited, although it is widely recognized, I think, that, as he drew it,  Hayek’s circle of unknowability is both too wide (not respecting degrees of evidence) and too narrow (sweeping assumptions about market process).  Roberts can challenge me on this if he wants, but I see Fatal Conceit-ism in just about every paragraph of his post.

Meanwhile, I think really empirical economics would place intrinsic barriers to hubris and ideological cherry-picking, since its methods would be inherently self-critical.  For instance, cataloging all plausible explanations for a possible relationship between X and Y and identifying potential markers for them in the evidence, as I suggest here, would be a prophylactic against overconfident “empirical” claims about a researcher’s favored theory.  Similarly, serious attention to issues of proxy measurement would give pause to those eager to read in a sweeping interpretation of what is often just indirect evidence.  I have also expressed concern over methods that are based on the assumptions that impose a priori uniformities on the economic relationships one tries to estimate.

So my position is that Roberts and Cowen are more right, and Smith less, than I would like, mainly because what passes for empiricism in economics at present is often deficient in an empiricist, self-critical spirit and methodology.  At the same time, the debates over topics like the minimum wage, the effects of charter schools on educational outcomes and the like are on a vastly higher plane when they are about data sets and analytical assumptions than the certitude of my unquestioned beliefs against the certitude of yours.  It’s also a cheap and not altogether forthcoming dodge to respond to econometric disputes with a flip “There is never a clean empirical test that ultimately settles these issues.”  (Roberts)  That's a Merchants of Doubt epistemology.

12 comments: said...

"Cowen," not "Cowan."

Peter Dorman said...

So right. I'll fix that. (Humbly.)

Thornton Hall said...

Someday people will notice that there are other academic disciplines like philosophy and social psychology.

Thornton Hall said...

Until then, economics will be a pale shadow of other fields, an ongoing experiment in what philosophy and social psychology would be like if participation were limited to math savants.

Noah Smith said...

I...think this post agrees with me? Maybe?? It's hard to tell...

Peter Dorman said...

Sorta, Noah, but I think there are dark spots in econometrics as now widely practiced that are as faith-based as much of the theory we are now so willing to be skeptical of. To the extent that's true, it means that "actually existing empiricism" is less corrective of hubris and motivated reasoning than it ought to be. We need serious, self-doubting Feynmanistic empiricism. (Not that there isn't any in econ, of course, but not nearly enough.)

Or maybe I'm just trying to be obscure! That way people will think I'm smarter.

Sandwichman said...

Economic empiricism? Isn't that like Western Civilization?

Sandwichman said...

It is a plain fact if people over 55 were employed at the same rate in 2016 as they had been in 2007, there would be 1,143,000 fewer over 55-year olds employed in 2016. Meanwhile there were 918,000 fewer 16-19 year olds and 720,000 fewer 20-24 year olds employed in 2016 than if members of those age groups had been employed at 2007 rates.

Might it not be the case that if 1,143,000 over 55 year olds suddenly disappeared, job vacancies might open up for some of the unemployed 16-24 year olds?

"No!" the economist would answer impatiently, because the loss of those million odd productive oldsters would result in complementary job losses among youngsters that would more than offset any substitution effect. The economist would then present empirical evidence consistent with this explanation.

Of course what counts as the economist's "empirical evidence" depends on a whole set of assumptions, some of which are stated and others tacit. The raw comparative stats of plus-employed seniors and minus-employed youths doesn't count because theory tells us it is too simple-minded. But the economist's more theoretically-informed assumptions are not just simple-minded -- they are more complexly and complacently simple-minded.

What is the sound of one invisible hand clapping?

AXEC / E.K-H said...

Humble pie economics: the next fake
Comment on Peter Dorman on ‘Where Should We Put Economic Empiricism on the Hubris-Humility Spectrum?’

Hubris/humility is the proper issue for psychologists. Economists, on the other hand, are supposed to figure out how the economy works. More, what is expected from economists is not storytelling about economic events or comments on economic policy or psychological self-reflection but the true theory: “In order to tell the politicians and practitioners something about causes and best means, the economist needs the true theory or else he has not much more to offer than educated common sense or his personal opinion.” (Stigum)

The sole criterion to judge economists is whether economic theory is scientifically true/false and not whether they are arrogant/humble or likable/unlikable or conservative/liberal or whatever the sitcom characteristic may be. Scientific truth is well-defined since 2000+ years as material and formal consistency.

From scientists knowledge is expected and NOT the humble confession that they know nothing because reality is complex and uncertain and unknowable.#1

Knowledge, though, is not what economists have to offer. Walrasianism, Keynesianism, Marxianism, Austrianism are axiomatically false. But make no mistake, economists are rather comfortable in their proto-scientific swamp where “nothing is clear and everything is possible” (Keynes) and where scientific incompetence reaffirms itself with phrases like there is no truth, empirical tests are always inconclusive, all models are false, and the self-deception that the pluralism of false theories is a proof of tolerance and not of failure.

The current wave of empiricism and humility is just another exercise in reputation management and another example for the cargo cultic crap which economists produce and consume since 200+ years.#2

Egmont Kakarot-Handtke

#1 See ‘Failed economics: The losers’ long list of lame excuses’

#2 See ‘Where economics went wrong’

Daniel Kuehn said...

Peter - in your response to Noah in your comments could you be more specific? Do you have econometric methods in mind or people in mind? I'm a fairly up to date applied micro guy, I try to second-guess my work as much as I can and actively seek out other people to do the same. I feel like this is what most fairly up to date applied micro guys do. When there are people I'm not 100% on board with they seem to understand all the arguments on either side and have put together good reasons for thinking what they think. Obviously one of us is wrong and obviously we've all got cognitive bias blindspots but where are you thinking there is hubris exactly?

You seem to think it is actually practiced with hubris but I have no idea what you have in mind.

Peter Dorman said...

Hi Daniel, and thanks for asking. The two links in the OP are a starting point. I'm not thinking ad hominem in these criticisms, although some hominem are more implicated than others; mostly I'm thinking of methods that, IMO, are insufficiently self-critical.

1. Most structural econometrics drives me crazy. It is more about calibration than testing as such, and in the areas I've worked in the models are implausible -- lots of U max stuff in areas of behavior where U max doesn't hold.

2. I've railed in the past against the "is consistent with" dodge, which is an extreme version of the sins of NHST. Rejecting the null is not "support" for your theory if there are three other theories out there that all imply the null is wrong. Do you think econometricians generally do all they can to put *their* pet theory to the test against its competitors?

3. Andrew Gelman's "garden of forking paths" is a whole agricultural district in most conventional econometrics. Do you think this generalization is unfair?

4. Heterogeneous effects, in my experience, is more a fallback position when the researcher isn't able to demonstrate a population-wide effect as originally intended; it's not a lever to test generalizability. If a general effect sneaks past .05, how often do you see researchers disaggregate and partition, testing to see if the pattern of appearance and non-appearance of the effect makes any sense?

Well, this is starting to be a post of its own, so I should break it off. I hope that's a clear answer to your very proper question.

Denis Drew said...

The most all-pervasive factor in US economics is non-measurable because it is totally missing: not just organized labor -- but more trenchantly (never used that word before so I hope I got it right) the missing criminal (and for the most part civil) enforcement mechanism protecting organized labor from being beaten to death by employer market mechanisms (read firing organizers and joiners).

The only way to measure the impact of the absent enforcement mechanism (union busting has been illegal in this country for 80 years) might be to compare outcomes (political as well as economic) with economies in which labor is free to collectively bargain without a firestorm of resistance to same.

I'm thinking continental Europe foremostly where collectively bargained contracts are universal -- including the all important centralized bargaining so Walmart cannot come in and gut good supermarket contracts for the race to the bottom.

Walmart in point of fact closed 88 big boxes in Germany where it could not compete successfully on quality alone.

French Canada right next door is a good example -- having sector wide labor agreements; as do Argentina and Indonesia I believe.

Anecdote: For a couple of weeks at San Francisco's Marriott Hotel on Fourth Street there was a union demonstration on the sidewalk. The leader would chant: "Hotel Marriott, you're no go; sign that contract like you should -- San Francisco should beware; Hotel Marriott is unfair."

You can imagine how much good this did.

A concierge I was taking to work further up Market Street one Sunday morning in my cab told me that part of the deal with the city for allowing the Marriott to build there was that the union would be allowed.

WHY, OH WHY, DID A UNION NEED TO BE ALLOWED? Why cannot people simply collectively bargain if they simply wish -- AS FEDERAL LAW HAS SPECIFIED SINCE 1935 Oh I get it; no penalty for destroying the only process by which the majority of Americans can exercise economic and political strength.

Measure US wages and democracy v. Germany -- to get back on topic.