CrowdMath: Can AI Solve the Puzzle of Collaborative Math?

In the race to enhance AI's prowess in mathematical reasoning, the focus often lands on straightforward problem-solving abilities. However, real-world math isn't just about arriving at a final answer. It's an intricate dance of collaboration, error detection, and iterative improvement. Enter CrowdMath, a dataset that sheds light on AI's ability, or lack thereof, to understand these complex interactions.

The Dataset and Its Origins

Between 2016 and 2025, the MIT PRIMES, Art of Problem Solving (AoPS) CrowdMath program hosted a collaborative research initiative. These discussions, rich with incremental contributions, partial arguments, and proofs, have now been transformed into a dataset of 164 expert-annotated progress chains. Each chain maps out a forum discussion, tracing the journey from an open-problem statement to a completed proof. Notably, posts are tagged for their roles, such as partial progress or error identification, showcasing the entire spectrum of collaborative problem-solving.

Model Performance: A Mixed Bag

Six advanced AI models have been put to the test against this dataset. They shine predicting the next post in a discussion, boasting an accuracy range of 83-88%. Yet, when tasked with identifying the significance of each contribution, the results are far less impressive. The leading model barely scrapes a macro-F1 score of 0.42 in post-role classification. This gap highlights a critical limitation: while models can follow the dialogue, understanding the deeper significance of each input remains a challenge.

Why Does This Matter?

Let's apply the standard the industry set for itself. If AI can't grasp the nuances of collaborative problem-solving in mathematics, can we truly trust it to handle more complex, real-world tasks requiring teamwork and incremental progress? The burden of proof sits with the team, not the community. The marketing says distributed, yet the practical applications suggest otherwise.

This isn't just a technical hurdle. it's a fundamental question about AI's role in fields where collaboration is key. The potential for AI to revolutionize industries is undeniable, but only if it can understand and contribute meaningfully to team dynamics. Isn't it time we held AI to the standards it espouses? Skepticism isn't pessimism. It's due diligence.