Lies, damned lies and ‘peer-reviewed’ research

Ross McKitrick and Bruce D. McCullough summarize the results of their paper on the need for proper due diligence in verifying research used as a basis for "public policy" (a.k.a. "pissing away giant wads of other people’s money"):

Empirical research in what are commonly called "peer-reviewed" academic journals is often used as the basis for public policy decisions, in part because people think that "peer-review" involves checking the accuracy of the research.

...Academic journals rarely, if ever, check data and calculations for accuracy during the review process, nor do they claim to.

... But the other dirty secret of academic research is that the data and computational methods are so seldom disclosed that independent examination and replication has become nearly impossible for most published research.

... Our report also explores numerous examples from other academic disciplines, such as medicine, history, environmental science and forestry, in which prominent or policy-relevant research was shielded from independent scrutiny by withholding data and/or computer code.

....One striking example in the context of the current U.S. housing meltdown concerns a 1992 study by economists at the Boston Federal Reserve, published in the prestigious American Economic Review, that purported to show statistically significant evidence of racial discrimination in U.S. mortgage lending practices.

Based on this study, federal regulations were rushed into place that forced banks to loosen lending standards and threatened them with severe financial penalties for failure to correct the alleged discrimination.

... It took nearly six years, and a Freedom of Information Act request, for independent economists to discover coding errors in the data that invalidated the original conclusions. But by this time the new lending rules were in place that ultimately contributed to the buildup of bad mortgage debt now ravaging the U.S. financial system.

In conclusion:

... Academics rightly insist on the freedom to do their research without public or political interference. But when that research influences policy, the public has a right to demand independent verification. Researchers might want to influence policy but if they plan to keep their data and computer code to themselves, they should keep their results to themselves too.

See also.