Navigating Competency Questions in Knowledge Engineering

Competency questions (CQs) play a critical role in knowledge engineering. They're essential for designing, validating, and testing ontologies. But how do different approaches to formulating these questions stack up?

Three Approaches Under the Microscope

In an insightful new study, researchers conducted an empirical evaluation of three distinct methods for generating CQs: manual formulation by seasoned ontology engineers, pattern instantiation, and state-of-the-art large language models (LLMs). The study focused on cultural heritage requirements, assessing the outputs based on acceptability, ambiguity, relevance, readability, and complexity.

This paper's key contribution is twofold. First, it offers the inaugural multi-annotator dataset of CQs, all generated from the same source using different techniques. Second, it provides a systematic comparison of these approaches' characteristics. It's an ambitious endeavor, providing clarity in a field where systematic comparison is scarce.

The Role of LLMs

The key finding? LLMs show promise as an initial tool for eliciting CQs. However, they're not a catch-all solution. These models are sensitive to the choice of LLM, often requiring further refinement before they can effectively guide requirements modeling.

Is the reliance on LLMs overshadowing traditional methods? It's worth considering whether the efficiency gain from LLMs justifies the possible trade-offs in precision and depth that manual or pattern-based methods might offer. After all, an LLM's output needs human oversight and adjustment. Automation should enhance, not replace, expert input.

Why This Matters

Understanding these different CQ formulation methods is key for any organization involved in knowledge engineering. It informs decisions about which approach to adopt based on specific needs and constraints. The ablation study reveals that while LLMs can kickstart CQ generation, they can't yet replace the nuanced expertise of human engineers.

Crucially, the study opens the door for further research into optimizing blend approaches, where the strengths of LLMs and manual methods are combined. It's a balance of speed and accuracy that could redefine ontology development.

In the end, this paper provides a valuable roadmap for navigating the often complex terrain of competency question generation. As LLMs continue to evolve, their role in this space will likely grow. But for now, the collaboration between machine and human remains indispensable.