Peer Reviews: The Politeness Trap in AI's Textual Analysis

Peer reviews are supposed to guide authors, but there's a hidden trap. The politeness of reviews masks true rejection signals, especially in AI's textual analysis. A study of over 30,000 ICLR submissions from 2021 to 2025 exposes this issue.

Score vs. Text: A Failing Comparison

Here's the raw data: score-based models boast a 91% accuracy in predicting paper acceptance. Text-based models? A mere 81%, even with the help of large language models. The numbers don't lie: scores tell the truth, text doesn't.

Why should we care? Because authors misinterpret reviews, leading to false hope or confusion. This ends badly. The data already knows it.

The Misleading Politeness Principle

Why does text fall short? It’s not just about AI's shortcomings. Politeness is the real culprit. Rejected papers often receive reviews littered with positive sentiment words. They mask the rejection signals. Authors are left in the dark, misjudging outcomes.

Take a look at the 9% of papers where score-based models failed. High kurtosis and negative skewness in score distribution were common. One low score can be the death knell, regardless of average scores.

Zoom Out: The Bigger Picture

Peer review isn’t just academic bureaucracy. It's a system that determines who gets to share their findings and who doesn’t. If politeness muddies the waters, are we really advancing knowledge, or just playing nice?

Everyone has a plan until liquidation hits. For researchers, that liquidation is rejection. The politeness principle is lying to them. Can we afford to let that continue?

ICLR's case is a stark reminder. Text needs a rethink. Scores are blunt but honest. The funding rate is lying to you again if you think otherwise.