Why Fact Checkers Need to Accept That They Can Err
One of the more interesting statements that stood out in my interview with PolitiFact’s editor this past April was her assertion that much of the disagreement with her site’s fact-checking verdicts is aired on social media and blogs rather than sent as formal letters of complaint to PolitiFact itself. I noted at the time that this suggested that either fact-checking organizations needed to better educate the public about how to file complaints with them or that the public simply didn’t think fact checkers would change their verdicts even in the face of contradictory evidence. This raises the question: What recourse do the subjects of negative fact checks have to dispute rulings they believe are inaccurate?
Major fact-checking organizations are typically signatories of the Poynter International Fact Checking Network’s Code of Principles that lays out a set of five basic requirements for transparent fact checking. The fifth and last of those principles relates to corrections and states only that fact checkers must agree to this statement: “We publish our corrections policy and follow it scrupulously. We correct clearly and transparently in line with our corrections policy, seeking so far as possible to ensure that readers see the corrected version.”
IFCN regularly reassesses each signatory to ensure they are continuing to adhere to its principles by using a simple checklist. Yet, the portion of the checklist relating to corrections states only that the fact-checking site must agree to make corrections it believes are warranted and that the site provides at least two examples of corrections it has made in the past.
A brief summary of each fact checker’s correction policy and the IFCN’s evaluation of that policy’s appropriateness is included in the assessment report beside their entries on the signatories page.
Browsing a sampling of these assessments confirms that the majority of fact-checking sites have corrections policies that say little more than that they will make what corrections they deem fit but offer no recourse in cases where there is a disagreement between the fact checker and the person/organization whose claim they rated false.
The IFCN’s perspective on corrections appears to view fact checkers as largely infallible. In its assessment of the Washington Post’s fact checker, the IFCN reviewer wrote, “[Y]ou would hope that a blog devoted to fact-checking wouldn’t need a correction, but mistakes happen.” In short, fact checkers are seen as only making “mistakes,” with no acknowledgement of the complex landscape of many claims that can lead to genuine disagreements over the interpretation of basic facts.
To learn more about how IFCN views the corrections process and especially what requirements it places on its signatories regarding how the subjects of fact checks can contest verdicts they disagree with, I asked IFCN Director Alexios Mantzarlis for his perspective.
When asked what guidance IFCN provides its signatories regarding their corrections process and whether it enforces a minimum set of requirements, Mantzarlis responded that the only requirement is that each signatory provide “a clear, transparent step-by-step explanation of its corrections policy” and list two fact checks it has corrected in the past. No other recommendations or requirements are enforced or planned for the future.
This raises the question of what rights the subject of a fact check has to contest a verdict they believe is unsupported by the facts. After all, fact checkers are only human and many claims involve complex and nuanced evidence that can support multiple conclusions -- and fact checkers may rest their verdict on evidence that is itself highly contested.
In such cases the first step is for the subject to file a formal letter of complaint with the fact-checking outlet, outlining point by point the elements of the fact check they believe are incorrect, providing the evidence documenting those errors and requesting a correction. All IFCN signatories provide for this basic process.
The problem is that what happens next is entirely in the hands of the fact checker, who can simply respond that they stand by their ruling and will not make any changes. Fact checkers are not under any obligation under current practices to consider new evidence. Nor are they required to respond in any detail to the complainant beyond acknowledging the letter and responding that no change will be made.
Few fact-checking organizations appear to have independent ombudsmen to mediate disputes, and for those that are part of a larger organization (such as a newspaper) that does have an ombudsman, it is unclear what authority those individuals have to force changes to a fact check. Indeed, a review of a random selection of IFCN signatories did not immediately turn up any that explicitly outlined an appeals process involving an ombudsman’s office. None in the random selection examined even outlined a formal appeals process in cases where the fact checker refused to change a verdict.
In short, in the absence of any kind of neutral third-party arbitration, there is no accountability in the fact-checking world. Fact checkers are simply left to police themselves.
A similar process plays out each day in the academic world where papers are published that criticize past work. In such cases, if the author of the original work files a formal complaint that the journal feels does not rise to the level of a retraction, they will frequently permit that author to publish a rebuttal letter to the editor that is published in the journal as their response so that readers are at least aware of the differing perspective. A review of a random selection of IFCN signatories did not turn up a single one that codified such a right in its corrections process.
Mantzarlis confirmed that this was the case and that IFCN has no plans to strengthen its corrections process or require that fact checkers offer an appeals process of any kind. “If the complaint requires making a correction, then the corrections policy addresses this issue,” he said. “If the complaint does not require making a correction, then I don't see any reason to provide the subject with space on a fact-checking website to spin some more facts. Subjects are free to publish their complaint on another platform but fact-checking shouldn't be reduced to a he said/she said format. Either the fact check is accurate and correctly presents the veracity of a claim or it is not and must be revised.”
This statement sidesteps the very question at the heart of the corrections process: Who determines whether the verdict is correct when there is a dispute? Mantzarlis leaves it up to the fact checker to decide if a correction is needed or not -- if they believe their verdict is correct, then it is correct and any evidence to the contrary is merely “spin.”
This is problematic in that it reflects a worldview in which fact checkers are above reproach and that their selection of what constitutes valid evidence and their interpretation of that evidence constitute “truth” by the mere virtue of their role as a fact checker. That the IFCN would not acknowledge legitimate differences of interpretation or commit to independent appeals processes raises concern over how seriously the fact-checking community takes the corrections process.
This is precisely the reason that the academic world provides appeals processes, rebuttal letters and other mechanisms to address the natural contradictions that arise as part of the scientific method. Perhaps the fact-checking world should learn from the scientific world it seeks to emulate and acknowledge that just because a person has “fact checker” in their title does not mean they are infallible.
Mantzarlis’ lone concession was that individuals may forward their complaints to the IFCN, which will ensure the fact checker is aware of the complaint and then file it away in a “dedicated folder” to be considered when the fact-checking site is reevaluated. He cautioned that “we do not litigate individual fact checks by all of the signatories” and provided no further detail about what might constitute grounds for decertifying a signatory other than “gross violations of the code — e.g. the signatory removed its methodology, tampered with evidence or overtly backed a political candidate.”
An example of how IFCN reviewers consider complaints can be found in a reevaluation of Snopes, in which the reviewer lists the complainant’s name publicly and dismisses two of her complaints without providing substantial detail regarding what evidence was used in that dismissal. A third complaint, related to the site’s funding, is dismissed based on the reviewer’s personal opinion, rather than by citing independent third-party research that supports the conclusion. This suggests that most third-party complaints are unlikely to significantly influence whether a fact-checking site is reaccepted as a signatory and that reviews are based more on the personal beliefs of the reviewer than on a rigorous evidence-based evaluation.
It is particularly notable that Mantzarlis advised those who disagree with fact checks and find themselves ignored to simply air their concerns on “other platforms” such as blogs and social media – the very places PolitiFact’s editor suggested most of their criticism is published.
Putting this all together, fact checkers appear to occupy a rarefied position in the modern information ecosystem in which they are the ultimate arbitrators of truth, able to effectively remove any claim they disagree with from the online conversation. Yet, unlike the academic world they seek to emulate, which has developed an elaborate set of checks and balances over the centuries, fact checkers are accountable to no one. No peer review process plays devil’s advocate with evidence or interpretations and no ombudsman or appeals process is available to question the final results. In short, if a fact checker says it is so, it is so. This view appears pervasive, from the comments by IFCN reviewers to the organization’s founder, meaning it is unlikely to change anytime soon. In the end, without a strong acknowledgement of the legitimacy of dissent and an effective appeal process to evaluate that dissent and provide accountability, fact checking will continue to lack the legitimacy of the academic review process.