A group of researchers from Germany, including Mario Fritz, published the preprint “Style over Substance? Linguistic Bias in LLM-based Peer Review” (arXiv:2610.35254) on October 26, 2026, regarding linguistic bias in AI peer review. The authors demonstrated that large language models acting as scientific judges systematically assign higher scores to papers written in a confident, categorical style, even when the content is identical. The most biased judge was Claude Sonnet 4.6, which accepted the “confident” version of the same work and rejected the “cautious” one in 21% of cases, while the opposite outcome occurred in only 2.8% of cases. According to the authors, with an acceptance threshold of 4.5 out of 10, a single change in tone was sometimes enough for a paper to cross the accept/reject boundary.

image
image

What happened

In the course of the study, the authors took 504 real submissions to the ICLR 2026 conference and used Claude Opus 5 to rewrite each one in two stylistic versions, “confident” and “cautious,” while keeping the scientific content unchanged. This paired design isolates the effect of style from the effect of content, so the resulting differences in scores can be confidently attributed to the tone of the text. Six models served as judges: GPT-5.5, Claude Sonnet 4.6, Gemini 3.5 Flash, Gemma 3, Qwen3 VL, and GLM 4.7, and all six assigned systematically higher scores to the “confident” versions. The direction of the bias was consistent across all tested models, meaning that a confident tone increased scores regardless of the vendor.

Context

The development of AI peer review is linked to the fact that conferences, primarily ML events like ICLR, have faced an explosive growth in the number of submissions, partly driven by AI tools, and organizers have begun experimenting with automated triage and reviewer assistants. The LLM-as-a-Judge pattern has long extended beyond the academic sphere: language models are used as judges in evaluation pipelines, moderation, and comparative benchmarks, so documented bias affects not only scientific peer review. The authors are inclined to explain the same direction of bias across all six models by a common training pipeline for assistant models, where confident and energetic delivery is usually rewarded, although this is an interpretation that requires verification. It is worth noting the strictness of the research design: paired rewrites on real submissions provide a cleaner estimate of bias than typical survey methods, as each paper is compared with its own revised version.

Why this matters for the industry

For the industry, the work describes a systemic bug in an entire class of judge models: the default versions of six vendors were not ready for evaluation tasks without stress testing and fine-tuning, so switching from one model to another does not solve the problem. Pipelines where the fate of a text or object is decided by a single AI judge likely systematically overrate categorical materials, and their numerical verdicts should be considered stylistically biased until calibration is performed. The authors point to a path for correction—specialized fine-tuning of judges using fine-tuning or LoRA methods on calibrated rating scales. It appears that paired content-preserving rewrites will become a standard section of eval sets and bias packages, and calibration of reviewer models will emerge as a separate direction, with those who first integrate such checks into their workflow gaining user trust and differentiation from thin wrappers over “raw” models. Conference organizers have grounds to avoid launching AI triage of submissions without preliminary stress tests, as the cost of error is the integrity of peer review.

Why this matters for users

For readers, the conclusion applies to any “judging” task: if you use a language model to evaluate texts, code reviews, resume screening, or moderation, its verdict may depend on the confidence of the phrasing rather than the substance of the material. A minimal check is available immediately and almost for free: take a representative sample of key texts, generate a pair of rewrites in a “confident” and “cautious” tone for each without changing the meaning, run both versions through your judge model, and compare the final scores and verdicts. The authors recommend testing models on meaning-preserving rewrites before trusting their numbers, and treating the model’s “enthusiasm” as a systemic bug rather than a sign of quality. If the judge’s scores diverge on semantically identical rewrites, the evaluation is currently measuring style, and it should not be relied upon for decisions affecting people.

What is still unknown / limitations

The results were published as a preprint submitted on October 26, 2026, and have not yet undergone full peer review, so the figures should be considered preliminary. The work covers one conference, ICLR 2026, and one type of text, scientific papers, and the rewrites in both tones were generated by a single model, Claude Opus 5, so only part of the possible stylistic spectrum was investigated. Direct verification of generalizability to other genres, such as READMEs, pitches, or business documentation, was not conducted in the experiment. The explanation of the bias by a common training pipeline for assistant models remains a hypothesis requiring separate verification.

Sources

Author

Look at AI, editorial team