The Hidden Challenges of Sustaining Quality in Long-Term Annotation Projects

In the relentless march toward better natural language processing, the integrity of data annotation often gets overshadowed by the allure of model performance. But pull the lens back far enough, and a pattern emerges that ties the quality of annotation to the very nature of time and attention.

Unveiling the Setswana Sentiment Dataset

Consider the Setswana sentiment dataset, a collection of 3,565 tweets, meticulously annotated by three native speakers. Across eight batches, these annotators embarked on a task that ostensibly promised consistency. Yet, the aggregate Randolph's free-marginal Kappa, a statistical measure of inter-annotator agreement, stood at a respectable 0.76, labeled as excellent. However, beneath this veneer of excellence, a more troubling story unfolded.

In reality, the per-batch Kappa values plummeted by over 32 points throughout the process. The culprit? It seems to be a combination of label confusion and drift in annotation patterns, particularly as the annotators slipped into the autopilot zone, especially on the delicate line between negative and neutral sentiments.

The Tyranny of Time

The better analogy here's not one of skill, but of simultaneity. The dataset's analysis shows that annotations made within one minute of each other boasted a Kappa of 0.98, while those separated by more than a day languished at 0.65. This disparity starkly highlights how attention and focus can fray over time, undermining the dataset's reliability.

Curiously, annotation speed and the linguistic features of tweets showed no significant tie to Kappa values. Instead, the temporal chasm became the dominant predictor. So, is the future of annotation destined to be a race against the clock?

Benchmarking the Future of NLP

In the race to optimize, three open multilingual encoders and proprietary models like GPT-5 and the Gemini model were put to the test. The results are telling, with fine-tuning delivering gains of 29 to 43 macro-F1 points over pretrained baselines. GPT-5's few-shot capability emerged as the frontrunner, achieving a macro-F1 score of 62.2. But one must ask, does superior model performance justify the erosion of annotation integrity?

As we release the dataset, complete with per-annotation timestamps and the analysis code, the hope is to foster reproducible quality audits for future African language NLP resources. But the larger question looms, how do we ensure the survival of quality in a landscape where speed often trumps precision?

To enjoy AI, you'll have to enjoy failure too. It's within these setbacks that we forge the path forward, ensuring that the data underpinning our AI models is as strong and reliable as the predictions we seek to make.