Peer Review Is a Guild

Restoration of Scientific Integrity Entails a Truth- "Marketplace"

TL;DR

Peer review is a guild, protecting insiders. COVID exposed it. Human nature needs channeling, ideally via rewards-based self-correction, self-policing. Oddly enough, football/fantasy sports show how to fix the adjudication of “science”/knowledge: transparent stakes, scoreboards, convergence. My MODEST PROPOSAL: an open-ledger prediction-market system to replace the current “guild-” architecture of peer review, which is structurally incapable of producing reliable truth at scale. A new “Republic of Evidence“-scoreboard, emanating from open claims, transparent data, real stakes, replication bounties, and continuously updated “batting averages”of both researchers and reviewers will replace, brick-by-brick, accepted “fact”. Once that mechanism exists and gains legitimacy, today’s science (mis)information wars self-resolve: gradually and credibly.

[now republished within the The Daily Sceptic as “How to Fix Science”]

Part 1: The Problem

The Moment The System Revealed Itself

My inspiration emanates from conversing with NIH chief Dr. Jay Bhattacharya, November 1 – subsequent to his Brownstone event keynote, in which he spoke of the failure of scientific adjudication during COVID.

“What you have then is a fundamentally broken scientific sociology... that emphasizes authority rather than ground truth-- (enabled by) an utterly incurious press devoted to propping up essentially a political party.”

With no open forum for scrutiny, institutions act with the impulse unity of impunity. That allowed ...

“a pretty vicious attack on me (within Stanford). It became clear my friends and colleagues wanted me to be excommunicated.”

His colleague (and erstwhile Trump-appointee) Scott Atlas was formally censured by Stanford’s faculty senate members for his (rational) anti-lockdown, anti-school closure views which they literally called “anathema“. This is not a scientific-- but, per Bhattacharya-- “essentially a religious” disagreement,

Against that backdrop, Dr. Bhattacharya outlined to what form science must return, if it hopes to recover integrity: a system in which every claim is paired with its data, replications and independent checks:

“Truth... is determined by whether other people, humble scientists, looking at the same thing, find the same thing, or something very close to it.... Is there consilience? Is there a different way of approaching it that still yields the same answer?”

Truth, he said, cannot come from hierarchy. It must arise from independent convergence. His framework points toward what he calls a “Second Scientific Revolution”. I see it instead as a Scientific Restoration. The problem is not a lack of intelligent or capable scientists; rather the structure. When the Soviet Union collapsed, we learned its engineers had been world-class, but trapped inside a system rewarding conformity. Modern science is in a similar bind. The talent is there. The architecture is wrong. In Bhattacharya’s model, truth would rise through convergence rather than hierarchy.

I agree with this direction, but transparency alone will not overcome the entrenched incentives that allow insiders to ignore or decline to replicate what threatens their priors. A Restoration requires a structure of feedback and consequences. My proposal builds on Dr. Bhattacharya’s, but goes further. It borrows from markets, bitcoin, gaming, and fantasy sports, where entire ecosystems work (only) because participants accept rules, carry risk, and live with outcomes.

Science lacks that external scoring today. The model outlined below, my Modest Proposal, supplies what the current system cannot: incentives that reward accuracy and discourage gatekeeping.

The Guilded Age (Of Science)

Peer review was intended as an impartial check on truth; it has hardened into a guild. We see the same drift in the judiciary, where circuit courts often track power over neutrality. While the design is impartial; the humans are not – presenting the oldest problem in governance: who judges the judges? Peer review is no exception-- a mechanism built to protect inquiry (even with anonymity and conflict-of-interest forms) now sorts research and researchers into in-group and out-group faster than a high-school lunchroom.

This pattern predates COVID-politics.

• H. pylori and ulcers (1980s–1990s)
  Barry Marshall and Robin Warren were told repeatedly that ulcers were caused by stress and personality. Reviewers downgraded his bacterial theory (1983). Ultimately in 1985, Marshall found an editor willing to “stick his neck out very far to publish”.

• The ‘amyloid mafia’ within Alzheimer’s research (1990s–2024)
  Researchers who questioned the amyloid theory were screened out. Thirty years later, fraud revelations reshuffled the deck.

• Prions (1982–1997)
  Stanley Prusiner’s idea-- that an infectious agent could be made of pure protein was rejected for more than a decade. Then came mad cow disease.

• Mitochondrial replacement (1990s–2015)
  Papers on mitochondrial replacement were stalled for 20 years with regulators’ insisting ethical concerns outweighed the science.

• Zika and microcephaly (2016–2025)
  When I submitted a commentary explaining the complete disappearance of any microcephaly surge(s) after the initial 2015 panic, JAMA’s chief medical editor tentatively accepted it, then demurred (early 2020) because it might “sow doubt in the public health establishment at this crucial juncture” (of impending COVID policy, societal disruption). That is textbook guild behavior: protect the institutions.

• Climate (1990-present)
  A 2019 Nature Communications analysis (in an admission against interest) found contrarian scientists featured in just 1% of mainstream scientific publications.

  • “my career prospects would be much better if I never wrote papers that question the IPCC. The skewing of the literature (and careers) can only be bad for society.” Ross McKittrick, 2011

  • “It is now virtually impossible to publish anything against the alarmist grain. The piles of unpublished manuscripts sitting on active scientists’ desks are growing into gargantuan proportions.” Pat Michaels, 2011

In metabolism, heart disease, infectious disease, psychiatry, nutrition-- lines of inquiry get steered away euphemistically with “outside scope,” “unlikely to change practice,” or “does not advance the field” -denials.

And then there was covid, the winter year(s) of our discontent, with its excommunications:

• The “Proximal Origin” paper’s authors suspected a lab leak; but buried it after calls from Fauci and Collins. Outsiders who kept pushing lab-leak evidence were labeled conspiracy theorists and desk-rejected for years.

• Positive ivermectin trials during COVID were held to microscopic standards while negative ones sailed through; the ideological thumb on the scale was obvious and costly.

Current double-blind reviews and disclosure boxes didn’t prevent the system’s degradation. Referees’ sharing grant funnels, rationales, and career-paths inverted the mechanism built to protect inquiry into one of protection from inquiry.

Peer Pressure

The irony around the word “peer” completes the circle. In a courtroom, a “jury of one’s peers” refers to equals bound to the truth. In British history however, the “Peers of the Realm” were “equals” only in allegiance to the monarch. Peer review was intended to follow the jury model, judging work on its merits. It has drifted toward the monarchical: more Royal Court than Royal Society (which itself has drifted).

Dr. Bhattacharya– without mentioning “guilds”, per se – is all over this:

“A few scientists control how everybody thinks about the physical world;... publish something in a top journal, that is “truth” (sic).”

Bhattacharya’s spring-2020 seroprevalence study showed far lower fatality rates than panic narratives asserted, whereupon our modern Galileo, for “heresy… was excommunicated.”

This is not how a scientific civilization behaves. This is how a guild protects its prerogatives, much like the old Billboard charts-- opaque, manipulated, and dominated by a handful of promoters’ deciding what is played; and for whose profit-- an arrangement which yielded the term, “Payola”, in 1938. Bhattacharya’s critique of scientific authority converges Payola with Savonarola.

Buried in the ArXiv

ArXiv is the closest thing modern science has had to a spontaneous “open republic.” Its rise in the 1990s showed what can happen when researchers bypass bottlenecks: physics, cosmology, and machine learning all accelerated because work was posted immediately, debated immediately, and iterated immediately. But arXiv also reveals the limits of an honor system. Once AI tools made it possible to spray the server with autogenerated “papers,” arXiv had to clamp down-- proof that openness without cost or consequence becomes noise. A true Restoration must keep what worked about arXiv (speed, “daylight”, iteration) while correcting what broke it: no stakes, no skin in the game, and no mechanism to distinguish workmanship from spam. A minimal bond or stake would have stopped the deluge instantly. The problem was not open posting-- it was costless posting.

Part 2: The Reasons

Human Nature Abhors A Power Vacuum

Humans form fiefdoms wherever power concentrates. In government, whole strata of executive employees become effectively unfireable, and predictable petty tyrannies emerge. Give men and women a system with no consequences and they drift toward comfort, conformity, turf protection, and suspicion of outsiders. These are not (purely or only) moral defects; they are universal tendencies. Any system that denies this will drift into stagnation and hierarchy.

The Tenure Trap
One professor told me (with unusual candor) that tenure (supposedly created to protect academic freedom) has accomplished the opposite. Instead of liberating scholars, it has built fortresses. “Once a Chair gains tenure, the drawbridge goes up. Those allowed inside are precisely those who will not challenge him.” If a tenured Marxist runs a department, he is not going to hire Thomas Sowell or Milton Friedman. “He is going to hire disciples who echo the worldview and preserve the lineage.”

There is pervasive nostalgia of a mid-20th century Golden Age of Universities, when brilliant, difficult, contrarian thinkers found space and support. But by the late-1970s, romantic “gold” was replaced by functional “steel”-- bigger campuses, bigger budgets, bigger bureaucracies. That’s exactly when formal, mandatory peer review is locked in as the gatekeeper.

Lord Acton’s “absolute power corrupts absolutely” foresaw that an insulated, funded, guaranteed authority to decide knowledge will drift toward self-parody– by which time it is visible, it is deeply nested. Consider the spectacle of Nature’s publishing “a... feminist queer crip approach to the gut,” complete with claims of “intergenerational trauma ...of famine” and “queer stoma pride”. Serious fields like oncology or particle physics (may) still survive-- but in the softer zones, ideology strolls in while dissent is shown the door. The annals of science are nearing a bad pun.

Peer Review as Finishing School

Think of peer review the way anxious parents think of an expensive private school. You do not pay that tuition so your kids can pursue wild ideas. You pay to file off rough edges, polish manners, groom résumés, and earn the stamp of “one of us”. Peer review does the same for science. It signals to NIH, Wellcome, or Gates that this lab cites the right priests, asks “constructive” questions, and will not embarrass the family or threaten the narrative that keeps money flowing.

When COVID hit, and real dissent might have saved lives, the guild did not open the doors. It circled the wagons and protected the family reputation. We keep hoping the faculty will police itself with honor. The framers of the Constitution knew better. They designed checks and balances because men are ambitious, jealous, and tribal.

Why Football Does This Better Than Science

Football may channel human nature better than peer review. Antsy kids (as well as 380-pound angry men) with energetic exuberance line up, hit each other (within the lines and rules) and let the scoreboard settle the question. Everyone accepts the outcome because the process is open and the numbers don’t lie. Science, conversely, has no scoreboard.

Fantasy football gamifies, democratizes, and commodifies this logic. Millions argue all week, but the market closes the case. Nobody is still insisting Justin Fields is an improved Lamar Jackson. The scoreboard humbles pretension. One Thursday in November provided a Socratic gridiron debate between the Bills’ high-power offense and reigning MVP vs. the Texans’ #1 defense. Only one side’s hypothesis survived harsh contact with reality. The scribes and reviewers who predicted this surprising outcome benefit immediately and obtain “cred” thereafter.

Part 3: The Restoration

A Modest Proposal: Building a Scoreboard for Science

If the problem is a guild, the solution is a league.

Imagine a Republic of Evidence modeled on the competitions we already trust:

• Claims, not whole papers, become the meaningful unit-- discrete, testable statements with preregistered protocols.

• Code, data, and lab notebooks go on an open, tamper-proof ledger the moment the claim is registered.

• Everyone who signs a claim stakes something small-- reputation plus a modest bond.
  Labs insure juniors so no postdoc chooses between rent and publishing.

• Prediction markets and replication bounties allow anyone to bet for or against a claim.

• Liquidity comes from a 1 percent surcharge on existing federal grants plus matching philanthropy.

• Whales are boxed out: stake caps tied to income, verified academic identities, transparent position sizes, and a rotating referee council akin to the NFL’s replay center in New York.

• Long-horizon problems stay open, paying out incrementally as Bayesian milestones are hit.

Put together, it works like this:

A scientist registers a claim and puts $800 of lab-insured skin in the game. Within weeks, the raw data is public; prediction markets are pricing the odds of replication; and rival labs are bidding on bounty contracts to test it. A year later, the independent referee council settles the final score using pre-agreed rules. The winners cash out, the losers pay, and every career résumé now carries a public batting average that no dean or editor can fake.

The guild’s gate collapses overnight-- because truth is no longer decided by three anonymous friends of the Principal investigator; rather, by strangers who bet real money and real reputation on what actually replicates, in ever closer (Zeno’s paradox) steps to the inevitably limned truth.

This will not create immediate utopia. But it will create something far better than the priesthood we have now. It will look less like a cathedral and more like the evolution of professional football: a messy, competitive ecosystem that gradually built real standards by exposing everything to daylight.

Different fields will play variants of “games” but under the same rubric, understanding that physics might be closer to the way football gets adjudicated and social sciences might be more like figure skating. Nonetheless, every “league” will run on the same core principles: open data, real stakes, and a scoreboard that never lies.

A guild cannot do that. A market can. To wit, RealClear Markets’ John Tamny makes a business observation that applies to science:

“Innovation requires an enormous amount of trial, error, and discarded prototypes.The failures are not waste; they are the information-generating cost of discovery. Clamp down on experimentation, and you do not merely slow progress-- you erase the very process by which breakthroughs emerge.”

Our next Gilded Age requires an un-Guilded Age.

________________________________________________________________________

Postscript

Several correspondents raised thoughtful objections:

• “Science is too complex: there are no clear rules for calling a claim true.”
Correct. Physics is crisp; epidemiology is captured; social science is oxymoronic. But every field already behaves as if stakes exist: grants won, careers lost, reputations accumulated through opaque channels. The market I propose simply makes that sub rosa economy explicit.

• “We already have staking: social capital.”
Yes, but it is invisible, unpriced, and manipulable. What the public sees is résumé, pedigree, and journal mastheads; none of which correlate with accuracy. A transparent record of claims, replications, and results is not a replacement for social capital. It is the scoreboard that reveals whether the social capital was earned.

• “Your system does not solve the [fill in the blank*]-controversy. *(vaccine-, autism-, climate-, et al.)”
Exactly. It isn’t supposed to. You cannot repair a scientific culture by beginning with the most radioactive questions. You start by repairing the adjudication machinery so that future disputes-- whether autism, statins, nutrition, reproducibility in AI, or climate are handled by an open, legible, rule-bound process instead of a priesthood. You do not rebuild the roof before stabilizing the foundation.

• “What about a deluge of AI-generated garbage?”
ArXiv already learned the hard way that costless submission invites collapse.
A staking model of small bonds, transparent identity, and real reputational accounting would prevent this instantly. Spam dies when it becomes expensive.

The goal is modest and radical at the same time: to replace “trust us” with “show us,” to do for scientific claims what fantasy football did for statistics and what markets do for price discovery. It is not a universal solvent. It is the framework within which solvents can exist.