I figured you would reply that you have made money in the market and would offer that as evidence that econometrics is phoney. This part of the discussion isn't worth a lot of time, but in fact you have demonstrated nothing at all about econometrics, only that you are a good market analyst. You have to show me reasonable evidence that you made your money specifically at the expense of other market participants who based their actions on econometrics and that your actions were taken specifically in response to misinformed econometrics. Otherwise, you are merely saying that some people make money in the market and some don't. I return to the market test. Financial institutions hire econometricians because doing so reduces the inefficiencies and cognitive errors they make. It doesn't eliminate the errors, which leaves room for savvy guys like you to make money. My point is that, by reducing the noise in the market, the econometricians reduced the scope for you savvy guys to make money off other people's blunders.

Enought of that. Let's get to the important stuff.

Your question about the nature of econometrics ("Isn't the real point whether econometrics succeeds in living up to your definition of science?") misses the point. Econometrics is not and does not claim to be a science. It is a statistical tool used by economics and finance, which are the sciences in question. You then go on to say that econometrtics "posits cause and effect relationships in data that cannot be proven -- and uses these very relationships themselves as "proof" of its claims -- it is neither 'logical' nor 'subject to verification.'"

Econometrics posits no such thing. Economics and finance do that. Econometrics provides the tools for testing the posited relationships or estimating relevant magnitudes. Furthermore, your argument that the relationships cannot be proven either misstates the scientific method or is vacuous. Science never proves anything; it only fails to disprove.

If you want to claim nothing is ever proven, you are formally correct, as any decent scientist knows. If you want to argue that such a state of affairs means econometric estimates are worthless, then so are all quantitative methods in all fields of science. In that case, I challenge you to provide a superior method of establishing knowledge of any aspect of reality.

You add that econometrics cannot make "semantically meaningful statements about the real world." What do you think of beta coefficients? They don't capture all aspects of an asset's price (e.g., there is a confidence interval on the estimate and there is an unexplained residual), but do you think they have no meaning at all? If they have meaning, then they are an example of useful econometrics. My understanding is that market participants use them to help price and trade trillions of dollars' worth of assets every year. Hard to see how that is merely mental masturbation.

I did indeed spend a lot of time discussing economics and not merely econometrics, but that's because you seem to be confusing the two. I am quite prepared to defend econometrics from the criticisms you have offered so far. Moreover, I do agree with you that some practitioners oversell their product, and you are right to be intolerant of them. However, their misbehavior says nothing about the validity of econometric methods, only about the character of the practitioners in question.

I must add that I find your willingness to publish this kind of give and take quite admirable. Even if we end up not agreeing, you have put before your readers full arguments by your supporters and your critics alike, allowing the readers to make informed judgements for themselves.

We both know at least one prominent newspaper and one prominent columnist working for it that would never do such a thing. (Oops, that's unscientific of me. All I can say for sure is that they haven't done so yet). I join your anonymous critic in urging you to keep up the good work.

Update... the anonymous reader who commented before comments again:

I guess we'll simply disagree on this. I'll just leave you with one final thought on an issue for which I believe we agree.

The scoring of tax cuts by the Congressional Joint Committee on Taxation presently does not incorporate macroeconomic feedback effects. To properly produce dynamic estimates, an economist needs various taxable income elasticity parameters to enter into a macroeconomic model.

Under your econometric critique - and as, I assume, a supporter of dynamic analysis - how would you obtain these elasticities if not through an econometric study of previous responses to tax cuts? Would you make them up? Draw a line on a graph?

Or would you argue that all historical data - and the accompanying statistical analysis - is irrelevant because they arose in a situation that does not exactly match the present policy proposal? And if so, how can you ever do any analysis of any sort? A time machine?

As economist Donald Luskin at Trend Macrolytics pointed out recently, inflation from excessively loose monetary policy during the past couple of years is now baked in the cake, and "there will be a great deal more inflation than anyone expects now - and a lot more market turbulence." Chairman Bernanke must come to terms with the fact that there is little beyond instilling confidence in his future leadership of the Fed that he can do to dampen that turbulence through monetary policy but there is a great deal he can do to amplify it. We must ride it out with an eye to smooth sailing on the other side.

...the American Prospect...commissioned a "study" of donations by Indian tribes represented by Jack Abramoff, and found that contributions to Democrats had dried up or remained stable once Abramoff was directing things, and that donations to Republicans had skyrocketed.

This has resulted in a large part in the shrillness with which liberal bloggers have greeted any attempt by the media to be even-handed about the Abramoff scandal. They feel entitled, because a flagship journal of the American Left has told them that there's essentially nothing to the claims that Abramoff directed money to the Democrats.

Paul Krugman has probably spread the myth (linked at a liberal blog to avoid Times Select) the furthest through his column in the New York Times on January 30:

But the tribes were already giving money to Democrats before Mr. Abramoff entered the picture; he persuaded them to reduce those Democratic donations, while giving much more money to Republicans. A study commissioned by The American Prospect shows that the tribes’ donations to Democrats fell by 9 percent after they hired Mr. Abramoff, while their contributions to Republicans more than doubled. So in any normal sense of the word “directed,” Mr. Abramoff directed funds away from Democrats, not toward them.

Numerous major lefty blogs have pushed this "study" as proving that Abramoff was not an equal opportunity employer, that he drastically curtailed spending to Democrats and radically increased it to Republicans...

I quickly discovered an obvious flaw. The Prospect article claimed:

At the same time, two of those four tribes -- Saginaw and Chitimacha -- saw their giving to Democrats drop or remain static.

But when you looked at the information on the Saginaw tribe, it said:

1) Tribe: Saginaw Chippewa (Michigan)
Pre-Abramoff contributions to Dems (1991 - 9/2000): $371,250
Post-Abramoff contributions to Dems (9/2000 - 2003): $191,960

Okay, so pre-Abramoff the Saginaw Chippewa gave $371,250 to the Democrats over about 9 years, that's a little over $41,000 per year, while post Abramoff, they gave the Democrats $191,960 over three years, that's $64,000 per year.

So to the American Prospect, going from $41,000 per year to $64,000 per year--a 50% increase in donations from that tribe per year to the Democrats--means that tribe "saw their giving to Democrats drop or remain static."

But it gets better. The Chitimacha? Again, the Prospect kindly gives us the figures so that we can work it out:

2) Tribe: Chitimacha Tribe of Louisiana
Pre-Abramoff contributions to Dems (1991 - 9/2000): $61,320
Post-Abramoff contributions to Dems (9/2000 - 2003): $64,000

Okay, $61,000 divided by 9 years is $6,800 per year, while $64,000 per year divided by 3 years is $21,000 per year. So going from $6,800 per year to $21,000 per year means that the tribe "saw their giving to Democrats drop or remain static."

You can see what's going on here, right? The American Prospect is actually looking at the total number of dollars donated without looking at the number of years. Gee, before Abramoff the Chitimacha donated $61,000 to Democrats, after Abramoff they donated $64,000; it's hard to see that represents a 200% increase per annum.

And no kidding, that's where they get the absolutely ridiculous 9% decrease in spending on Democrats that Krugman cites in his column.

A study commissioned by The American Prospect shows that the tribes’ donations to Democrats fell by 9 percent after they hired Mr. Abramoff, while their contributions to Republicans more than doubled.

What about the number of years that Abramoff represented the tribes and the number that he didn't? We have seen in the cases of the Saginaw and Chitimacha tribes, what the Prospect described as stable or declining funding for the Democrats was actually a pretty substantial increase on a per annum basis. And if you look at the number of years before and after Abramoff, there was only one tribe (Mississippi Band of Choctaw Indians) that spent the majority of the years in the study represented by Abramoff; most of them spent fewer than 1/3 of the years as his client.

But the clincher was when I downloaded the spreadsheet (Excel file) that showed the annual contributions by Abramoff clients to each party in each election cycle. Now, they don't make it easy for you, you have to wade through all the other tribes, but once you have the Abramoff clients isolated, these are the contributions by election cycle to the Democrats by the seven tribes:

1992: $3,700
1994: $5,500
1996: $43,000
1998: $231,000
2000: $175,970

Ah, here's the part that fits "the narrative". The tribes, which had been loyal to the Democrats suddenly abandoned ship in 2000. Except that's really not what happened. In 1998, the Saginaw Chippewa tribe donated $222,000 to the Democrats and all the other tribes donated a grand total of $9,000; in 2000 the Saginaw Chippewa reduced that to $57,000 but the other tribes increased their giving to $119,000.

In 2002, as you can imagine, the Democrats saw their funding cut dramatically. Most of the tribes were now under the merciless orders of Jack the Knife, and so their contributions to Democrats suffered draconian cuts:

2002: $550,980

No kidding. All of Abramoff's clients (except the Cherokee Nation, which didn't donate a significant amount of money) were on board before the end of the 2002 campaign. And yet in that campaign Abramoff's clients didn't reduce their funding for the Democrats, they more than tripled it!

But in 2004, Abramoff must have really cracked the whip on his tribal clients, right? I mean, Paul Krugman assured us there was a 9% decrease!

Sorry, Paul. Total donations to the Democrats from those same 7 tribes:

2004: $648,000. That's about an 18% increase over the 2002 level. Meanwhile the Republicans must have gotten the big bucks, right? Nope, they were cut by 33% by Abramoff clients in the last election cycle.

I don't mislead my readers like the American Prospect has its readers. The Republicans did get dramatically increased funding as well from the tribes represented by Abramoff, especially in the 2002 election, when funding went from $387,000 to $1.22 million (a much higher rate of increase than the Democrats received). So if the argument is over whether Abramoff's tribes gave more to the Republicans than Democrats, hey no question. But if the argument's over whether the "Democrats Don't Know Jack", all I can say is the American Prospect doesn't know how to count the Jack.

So to reiterate, the Democrats received from Abramoff's clients, both before and after they were clients, the following amounts by election cycle:

2000: $175,970
2002: $550,980
2004: $648,000

That's what the American Prospect wants you to believe is stable or decreased funding.

Dubner and I wrote a column in the NY Times that told people to bet on Seattle in the Super Bowl. The bet lost. Not more than three people mentioned this to me afterwards. Not a single angry e-mail from a stranger who lost their college tuition fund because of our column.

IÂ?m glad people didn't write me, but still, there should be punishment for wrong predictions.

Thanks to our correspondent "Irrational Exuberance" for the link.

Update... Reader Josh Hendrickson writes,

It is with great curiosity that I ask why you consider econometrics to be a "pseudo-science". I am aware of studies that have shown that sun spots effect GDP and that an increase in the minimum wage increases employment and other hogwash. However, I would point out that these ridiculous findings have led to further adjustments and advancements in producing empirical results. A great deal of science deals with trial and error. As with all science many theories or accepted practices are often flawed and can only be fixed when such flaws are exposed.

Many have come down hard on Steven Levitt for using econometrics to show that abortion reduced crime rates. This is hardly an argument for the purging of the children of poor, single mothers. Even so, his argument does not take into account spatial dependence, which is extremely important in studies involving crime because it factors in neighboring effects. Therefore his argument may be deeply flawed. Unfortunately there has yet to be anyone to attempt to prove him wrong using a spatial model. Nevertheless, this is not enough to consider econometrics a pseudo-science.

Many individuals manipulate numbers and studies to produce the results that they desire. Econometrics is not immune to these manipulations. However, those who manipulate studies can easily be disproved.

I am interested to hear why you think econometrics is a pseudo-science.

Update 2... An anonymous reader writes,

I read your site regularly and usually agree with your politics --- but you are doing yourself, your ideology, and your readers an immense disservice by referring to econometrics as a pseudo-science.

First of all, econometrics is not a field of inquiry per se, it is a research tool. And like any tool, it can be used or abused.

Your statement that econometric studies cannot be duplicated is false. Reputable economists (and other social scientists) make their data available if requested so that other researchers can replicate their results. Indeed, a common approach in social science to test the validity of an empirical result is sensitivity analysis, whereby researchers can examine the stability of statistical results upon a priori assumptions or data.

Your statement that econometric studies cannot be falsified is ignorant. The whole point of quantitative analysis is falsification. If a particular variable is thought key to a given phenomenon, it can be evaluated as a statistically significant using regression analysis. If other variables reduce its contribution, it will not be found significant, and the hypothesis is rejected. That is science. It isn't junior high physics, whereby a controlled lab experiment can be used to generate data, but few frontier level science fields are. Most scientists use some for of regression analysis or statistics, including economics, where it is known as econometrics.

A pseudo-science label is better aimed at some aspects of climatology, whereby results are claimed to be proven regardless of what the data indicates. For example, warming temperatures are supposedly evidence of human-induced climate change. However, so are cooling temperatures (aka increased temperature variability). In this case, the hypothesis cannot be rejected. That is pseudo-science.

Sorry for the rant --- but your rants against econometrics are very odd.

Otherwise, keep up the good work.

Update 3... Ecnomist John Seater writes,

Your criticisms of econometrics as a psuedo-science have an element of truth but are inaccurate, overstated, and unreasonable.

*Econometrics* is not and does not claim to be a science. Econometrics is a statistical method that may be useful in testing economic hypotheses or in estimating economic quantities. It is *economics* that claims to be a science. Many years ago, I memorized the following definition of science, found in my old Funk and Wagnalls dictionary:

"Science: Any department of knowledge in which the results of investigation are logically arranged and systematized according to general laws and hypotheses subject to verification."

Economics seems to fit this definition, and it is unclear from your remarks which aspect of this definition you think economics fails to satisfy. Economics starts with observation. Upon finding patterns in the data, economists formulate logically-connected explanations of the observed patterns. Those explanations lead to further hypotheses that are testable ("subject to verification"). A few examples: (1) Demand curves slope down. (2) Persistent inflation arises from money growth in excess of the growth of real output. (3) Inflation and unemployment will be negatively related in the short run and unrelated in the long run. (4) The rate of return on a risky asset will reflect that asset's correlation with the return on the total market through the asset's "beta," which is nothing other than the regression coefficient of the asset return on the market return (i.e., the result of an econometric estimation based on an economic theory). All of the foregoing are inexact in the sense that all contain error terms. Economists and econometricians know that, which is why they keep working on refining the theories and the estimation. Nevertheless, all of the foregoing examples get really strong support from the data. They derive from theories that in turn were formulated after observation of patterns in the data. What aspect of these substantiated testable hypotheses do you think fails to meet the definition of science?

You say the data are bad. Of course the data do contain measurement error, as all economic data always will. Econometrics, however, does not ignore measurement error; rather, it has methods that explictly take it into account. The fact that the data are imperfect does not invalidate statistical inference. Indeed, statisical inference is designed precisely to make inference in the presence of error. The only reason one needs *statistical* inference rather than just measurement is that there is random error, arising for several reasons. Statistics and econometrics are organized methods for making such inference. What alternative would you propose?

You say the hypotheses are dubious. Isn't that the reason they are tested?

You say the estimation results cannot be replicated. That really has me baffled. What prevents me from repeating someone else's regressions? Perhaps you mean that the definition of the data changes through time, introducing problems in replicating earlier results with subsequent data. Maybe so, but isn't that a problem with the way the data are collected rather than with the methods used to organize (economics) and analyze (econometrics) them? Exactly what is it you are criticizing here - economics, econometrics, or data collection techniques?

You set up straw men. You seem to imply that the whole exercise is worthless because it is inaccurate. Is weather forecasting worthless? No statistical exercise can be perfectly accurate, by its nature, but then no econometrician or statistician would claim otherwise. Given that your implications do not characterize the views of those to whom you attribute them, why do you think they are valid criticisms?

You are a practical man of the financial world, so presumably you believe in market tests. If economics, econometrics, and statistics are as worthless as you say, why do financial instititutions pay economists, econometricians, and statisticians for their services? Are they just fools that have been bamboozled? If so, you should be able to enrich yourself at their expense. If you can't do that, then why shouldn't we conclude that your criticisms are incorrect?

Besides, John, your whole defense is one of economics, not econometrics. You know that I have my doubts about economics, too, but that's not what we're supposed to be talking about at this moment. By defending economics when it is econometrics that is being attacked, aren't you damning econometrics with faint praise?

Posted by Donald L. Luskin at 5:45 PM | link    

Posted by Donald L. Luskin at 1:12 PM | link    

There's more...visit the archives!