Rethinking Zero-Shot Learning: A New Approach to Human Activity Recognition

Zero-shot learning (ZSL) human activity recognition has always been a challenging puzzle, but recent advancements are shedding light on a path forward. At the heart of the issue: how to effectively connect inertial measurement unit (IMU) sensor data to semantic class representations. It's a problem that, until now, had stumped many.

Breaking Down the Barrier

On the PAMAP2 dataset, researchers tested seven configurations combining various training pipelines and inference methods. They worked with 14 seen and 4 unseen activity classes, holding out subjects 108 and 109 for testing. The findings? The gap between modalities is primarily a training-time issue, influenced by the encoder's objectives.

One standout method involved a temporal convolutional network (TCN) trained using cross-entropy over label-name Sentence-BERT prototypes. This method achieved a sensor embedding mean cosine similarity of 0.30 against text prototypes. However, when the label-name targets were swapped for detailed activity descriptions, this similarity soared to 0.69. The improvement was consistent across all tested inference methods. But why should we care?

The Numbers Tell the Story

For ZSL aficionados, the numbers are significant. A combination of contrastive training with inverted softmax correction pushed accuracy to 73.2%, with a macro F1 score of 0.583 on unseen classes. This is a considerable leap from the 58.3% accuracy and 0.34 macro F1 of the baseline method.

What's the takeaway here? A richer, more descriptive text leads to better alignment, even though it might reduce inter-prototype separability in the Sentence-BERT space due to shared vocabulary. This doesn't detract from the alignment benefits as long as the prototype descriptions maintain some distinct vocabulary.

Rethinking Metrics

But there's another lesson hidden in the data. Relying solely on overall accuracy as a metric can be misleading, especially with imbalanced test-set class distributions. The researchers argue for using macro-averaged F1 as the benchmark for ZSL-HAR evaluations. It's a call to rethink how we measure success in AI fields.

So, why does this matter to you? In a world increasingly reliant on AI to interpret data, these findings pave the way for more accurate, reliable activity recognition. It's not just an academic exercise. it's a step towards smarter, more responsive technology in our everyday lives.