ATANT v1.1 Challenges Memory Benchmarks in AI Continuity Evaluation

Continuity, a nuanced system property in AI, has been reevaluated in the latest release of ATANT v1.1. This iteration doesn't redefine the original standard but instead highlights deficiencies in how current benchmarks measure continuity. It's a bold move, challenging the status quo of memory evaluations like LOCOMO, BEAM, and others.

Examining the Benchmarks

ATANT v1.1 presents a stark finding: none of the existing benchmarks fully captures continuity as originally defined. With seven essential properties outlined in ATANT v1.0, the average benchmark barely covers even one. Specifically, the median evaluation addresses only one property, while the average score stands at a paltry 0.43. If you're thinking partial credit will save the day, think again.

The report isn't just about numbers. It exposes methodological quirks, like the LOCOMO benchmark's 'empty-gold scoring bug,' which leaves almost a quarter of its corpus unscorable. This makes you wonder: what are these benchmarks really measuring, if not continuity?

A Calibration Conundrum

ATANT v1.1 doesn't just criticize. it calibrates. It presents its own LOCOMO score of 8.8% against a 96% cumulative-scale score for ATANT. The 87-point gap isn't about which system is superior. Instead, it's a revelation that these benchmarks assess different capabilities altogether. It's not that one is inferior, but they speak different languages of evaluation.

Each benchmark measures a real capability, but conflating them with continuity evaluation is like comparing apples to oranges. The AI-AI Venn diagram is getting thicker, but without the right measures, we risk misinterpreting what lies in the overlap.

Investing in the Right Metrics

The real takeaway? The field has under-invested in the properties ATANT v1.0 emphasizes. While existing benchmarks are valuable, their inability to fully assess continuity could lead to skewed research priorities. If we're building the financial plumbing for machines, shouldn't we ensure our tools are up to the task?

ATANT v1.1 isn't adversarial. It's a call to refine our understanding of what continuity means in AI. The proliferation of benchmarks has merits, but clarity in what they measure is key. Otherwise, we may miss out on developing truly continuous AI systems.