Economics: The ivory fortress: ONE of the more interesting responses to this week's Reinhart-Rogoff debate was this, from the economics blog Cheap Talk. "I Move That The AEA Stop Publishing Papers and Proceedings", the post title reads. Then:
Non-peer reviewed, inaccessible data, and punditry that can’t tell the difference between P&P and a regular AER article can’t be good for the reputation of the journal, the AEA, or the profession.
"Papers and Proceedings" means that the original Reinhart-Rogoff work was presented at the annual meeting of the American Economic Association and deemed interesting enough to be included, alongisde many other results, in the May edition of the American Economic Review (of 2010). It was in the journal, in other words, but it didn't go through the peer-review process that a typical journal article might have (a process that often takes a lot longer than the five months between the conference and the May publication). The implication is that only the rigorous, professional, and lengthy journal process can protect the reputation of economics and its licensed practicioners.
Last week, I attended a very thought-provoking conference at the Kauffman Foundation in Kansas City, where Clay Shirky gave a talk on how new technologies were affecting old models of education. He mentioned a blog dedicated to solving the famous P=NP problem and other vexing computer science issues. In August of 2010 the blog published a post discussing a possible solution launching, as Mr Shirky noted, one of the most high-brow comment threads in internet history. As it happened, the proof fell short, but the commenters effectively returned to its author (Vinay Deolalikar) a detailed set of criticisms, questions, and recommendations that might help him improve the proof and (potentially) solve the problem.
As Mr Shirky pointed out, that is a very different model of research and publication from what academic departments are used to or comfortable with. The work is all out in the open, anyone can pipe up and pitch in, and the whole process takes days or weeks rather than months or years. If there are embarrassing errors made by researchers or commenters they are immediately and publicly identified and addressed. There is no masking the inevitable—and often useful—parade of mistakes that accompanies the development of a new idea. The process is revealed in all its messiness.
Now, one might worry that without the journal process the quality of scholarship will suffer. Yet as Paul Krugman noted in a wonderful post from 2011, journals have been mostly irrelevant to the production of scholarship for some time:
The concern, or maybe just issue, is whether the rise of econoblogs is undermining the gatekeepers, whether any old Joe can now weigh in on economic debate, whereas in the good old days you had to publish in the journals, which meant getting through the refereeing process.My take is that the system never worked like that — or at least not in my professional lifetime. And when you consider how economic discussion actually used to work, you see the blogs in a different and more favorable light.First of all, policy-oriented research was never as centered on refereed journals as we liked to imagine. A lot of the discussion always took place via Federal Reserve and IMF working papers, and even reports from the research departments of investment banks. The rise and fall of Fed policy via targeting of aggregates, for example, was not a debate played out in the pages of the JPE and the QJE.Second, even for more academic research, the journals ceased being a means of communication a long time ago – more than 20 years ago for sure. New research would be unveiled in seminars, circulated as NBER Working Papers, long before anything showed up in a journal. Whole literatures could flourish, mature, and grow decadent before the first article got properly published – this happened to me with target zones back in the late 1980s, where my original 1988 working paper had spawned a large derivative literature by the time it actually got published. The journals have long served as tombstones, certifications for tenure committees, rather than a forum in which ideas get argued.What the blogs have done, in a way, is open up that process.
The problem, one might argue, is that the openness has not come far enough, fast enough. The Reinhart-Rogoff data were made available in 2010, but the authors didn't also provide their spreadsheet, which would have quickly allowed the masses to check both the Excel coding and the methodology. Given the straightforwardness of the work—this was not the sort of mind-bending mathematics one sometimes sees in economics papers—I suspect many of the points made this week would have been made back then. That might have been embarrassing to the authors (or it might not have been; we all make mistakes), but they certainly would have had less reputational capital riding on that one particular result and less to lose from corrections.
I am quite confident that Carmen Reinhart and Kenneth Rogoff would be a bit sceptical of this, given the way much of the press has misrepresented both their work and academic criticism of it over the past few days. But that, again, is a product of building substantial public interest in an idea over a period of years without giving the masses the necessary source material to understand it. You get a mad rush instead of a civil conversation in which more responsible elements can help guide participation and criticism.
The notion that journals are there to certify an idea as "ok" has long been anachronistic and a little odd. And while journals were once a means of making knowledge more widely available, they have become, as Mr Shirky noted, a means for restricting access to information. This episode should convince economists that more openness is better. Retreating back behind ivory walls will only protect their ideas by helping to make them irrelevant.