History Does Not Have a Replication Problem

Anton Howes of the Age of Invention published a really interesting essay on August 29th, 2023. Titled "Does History Have a Replication Crisis", it compared the ways history is done to science and especially to the 2011 replication controversy in psychology. I think Anton either exaggerated or misunderstood some central elements of how history is done, but I completely agree with his conclusion.

Describing the psychology replication fiasco, Howes said, "Many of them had made mistakes in the experiments, through negligence, unintended bias, or simple error. A few, quite simply, had been faked. Whole swathes of research and media coverage, including some globally best-selling books, turned out to be based on foundations of sand." This is indeed alarming, and the really crazy part of this story is that no one in psychology seems to have cared. He goes on to say:

I’ve become increasingly worried that science’s replication crises might pale in comparison to what happens all the time in history, which is not just a replication crisis but a reproducibility crisis. Replication is when you can repeat an experiment with new data or new materials and get the same result. Reproducibility is when you use exactly the same evidence as another person and still get the same result — so it has a much, much lower bar for success, which is what makes the lack of it in history all the more worrying. Historical myths, often based on mere misunderstanding, but occasionally on bias or fraud, spread like wildfire. People just love to share unusual and interesting facts."

The assumption I think is implied here seems to be that two historians looking at the same data ought to see the same things and come to the same conclusions about causality, importance, and interpretation. I think this sort of mis-states what history is about and how it differs from science. Science is designed to turn experience (observations of the world or the results of controlled experiments) into generalizations about the properties of things and processes. These should be repeatable because they are general. Water should always boil at 212F, with some adjustments for elevation and barometric pressure. I can't think of a lot of historical events that could be understood in this way. Howes says:

Take the oft-repeated idea that more troops were sent to quash the Luddites in 1812 than to fight Napoleon in the Peninsular War in 1808. Utter nonsense, as I set out in 2017, though it has been cited again and again and again as fact ever since Eric Hobsbawm first misled everyone back in 1964. Before me, only a handful of niche military history experts seem to have noticed and were largely ignored. Despite being busted, it continues to spread. Terry Deary (of Horrible Histories fame), to give just one of many recent examples, repeated the myth in a 2020 book. Historical myths are especially zombie-like. Even when disproven, they just. won’t. die.

The assumption here seems to be that this piece of erroneous information has been taken to mean or to show or to prove the same thing every time it has been deployed, which I doubt. I think the significance of an error might be judged by what it is used for and how central it is to supporting an argument or interpretation. Often these odd facts of history, whether true or erroneous or completely fabricated, are decorative rather than central. He then relates a story about a misunderstanding that seems to have emerged from mistranslation in 1869 about a supposed prize given by Napoleon for food preservation. In this case, I agree that this was unfortunate. I'm glad he has set the record straight. Still not sure, however, whether my understanding of the past will be significantly altered by this new datum. Maybe part of my point is that if a single data-point is that dispositive, there's probably something wrong with the history you're reading.

Howes says "History, like any other field, very often relies on trust". I agree, and that's exactly what my students said when I asked them. They said they try to get a sense of whether or not they can trust the person making the historical claim. They also sometimes double-check extremely unusual claims, especially if these seem central to an argument that seeks to overturn conventional understanding. Seems reasonable to me. They should also do things like understand the context of quoted passages from primary sources, to see if they're truly representative of an author's perspective or cherry-picked to support the secondary author's argument.

Howes continues, "the sheer pervasiveness of errors also allows unintentionally biased narratives to get repeated and become embedded as certainty, and even perhaps gives cover to people who purposefully make stuff up." This is probably true, but again I don't think the main issue is erroneous facts creeping in accidentally or being made up by propagandists. There are so many legitimate primary accounts of the past that it's easy for even well-meaning people to differ on causality and on what was important in the past. To answer the question I asked earlier, I think it would be entirely legitimate and in fact I would expect a different historian who looked at the mountain of data that went into Peppermint Kings to find different stories and interpretations supported by it.

He says, "I don’t think I’ve ever heard of anyone systematically looking at the same sources as another historian and seeing if they’d reach the same conclusions. Nor can I think of a history paper ever being retracted or corrected...In the 1960s you could find an agricultural historian saying of another that he was 'of course entitled to express his views, however bizarre'." Personally, I wouldn't expect these outcomes and I think this is either a red herring or the author misunderstands what historians do. And regarding the 1960s, the decades-long debate over the "Market Transition" is a perfect example of people arguing over interpretations, often of the same data. But the combatants were also finding their own data. There's so much of it out there to be found that getting it wrong or making it up is a relatively minor problem, in my opinion.

Howes concludes with a story about an article by Jenny Bulstrode, which I haven't read, so I'm relating his narrative of it. She apparently told a story that suggested an 18th-century innovation in iron-rolling had actually been stolen from a Jamaican mill using slave labor. The implication seems to have been that a white British inventor profited and has received historical recognition unjustly, for something actually created by enslaved Africans. This is entirely possible, but Howes noticed Bulstrode had not really cited sources proving her contention. He then says a graduate student named Oliver Jelf has checked the sources she cites and has published them to the web along with an argument that she drew the wrong conclusion from them. I think this is a really valuable illustration of how although history is certainly subject to people making claims that can't be backed up by evidence or that could be argued using other evidence, it also does have a mechanism for addressing such issues. Maybe even a more robust mechanism than psychology, since historians expect to disagree with each other over causality, importance, and interpretation. If (and I haven't read it so I don't know for sure) Bulstrode overreached in the interest of telling a "woke" story, it seems kind of promising that her evidence is being examined.

The solution Howes proposes is brilliant. I am 1000% in favor of historians actually making the archival sources they use available to the public. While there are some practical issues (I'm not sure I'd be able to obtain permission to publish all the thousands of photos of documents I took at all the places I went to do research), I would love to do it! I'd love to look at the documents people used to write dissertations and monographs. Especially because, since I have a different set of interest and an unusual perspective, I'd probably find some really fascinating material I could use in ways the author hadn't even imagined. Not that I'd "disprove" what the original author had done with the information; that's not the point. There's almost always more in a source than its utility for one narrative or interpretation. Having them available would open up rich new veins that would lead to new and probably better historical understanding. Just please let's not call it "metahistory"! That term already has too much baggage.