CoGate-LSTM: A Lean, Mean Toxic Text Detector

online moderation, toxicity detection remains a tough nut to crack. Especially rare but high-stakes categories like threats and severe toxicity. Enter CoGate-LSTM, a sleek new model promising to outshine its bulkier competitors.

Why CoGate-LSTM Stands Out

CoGate-LSTM isn't your average AI model. Instead of relying on sheer size and complexity, it utilizes a unique cosine-similarity feature gating mechanism. This approach hones in on the most informative feature directions, essential for detecting minority toxic classes. Traditional models often miss these subtleties. What you need to know: CoGate-LSTM leverages frozen multi-source embeddings like GloVe, FastText, and BERT-CLS, along with a character-level BiLSTM.

The number that matters today: CoGate-LSTM's 0.881 macro-F1 score on the Jigsaw Toxic Comment benchmark. It's not just about accuracy, though. With only 7.3 million parameters, this model delivers efficiency, achieving 96% accuracy with a mere 48 ms CPU inference latency.

David vs. Goliath: Outperforming Bigger Models

This model doesn't just compete, it excels. CoGate-LSTM surpasses fine-tuned BERT by 6.9 macro-F1 points and XGBoost by 4.7. That's significant. Especially considering CoGate-LSTM uses about 15 times fewer parameters than BERT. Gains are particularly noticeable on minority labels: +71% for severe toxicity, +33% for threats, and +28% for identity hate compared to BERT.

One thing to watch: the power of cosine gating. Ablations reveal this mechanism as the primary driver of CoGate-LSTM's performance, with performance dropping by 4.8 macro-F1 when removed. Character-level fusion and multi-head attention further enhance its capabilities, though to a lesser extent.

An Efficient Solution for Imbalanced Data

Why should we care? Because CoGate-LSTM offers a practical alternative to heavyweight transformers. It's efficient, effective, and adaptable. The model even performs well across datasets, achieving a 0.71 macro-F1 score zero-shot on the Contextual Abuse Dataset.

In a world dominated by oversized AI models, CoGate-LSTM proves you don't need to be big to be powerful. Isn't it time we start prioritizing efficiency and precision over sheer size?