GenAI in Law: Training Unlocks True Potential

Can generative AI truly boost productivity in professional settings? Recent research with 164 law students suggests that the key lies in training. The study tested three groups: one with no AI access, another with optional access to a large language model, and a third with AI access plus a brief training session.

The Training Effect

The findings are striking. Students with untrained AI access performed worse than those without any AI help. They wrote shorter answers, made more errors in case law, and scored lower overall. While some differences weren’t statistically significant, the trend was clear: AI without guidance can be a detriment.

What happens with training? A reversal. Trained students adopted AI at a much higher rate (41% compared to 26%, p = 0.044) and scored 0.27 grade points higher than their untrained peers (p = 0.027). They also stated rules more accurately (p = 0.014). It’s a clear indication that a little instruction goes a long way in maximizing AI’s utility.

Adoption Over Effectiveness

Here’s what the benchmarks actually show: the benefit of training stems more from increased adoption than the inherent effectiveness of the AI itself. The adoption effect's lower bound (1.06) surpasses the upper bound of effectiveness (0.42), although the confidence intervals remain broad.

This finding challenges the assumption that GenAI is primarily for lower-skilled workers. Without training, it's the higher-skilled students who tended to ignore AI, while lower-skilled ones used it less effectively. The reality is, tapping into GenAI's potential requires investing in both access and thorough instruction.

Implications for Professionals

So, why should professionals care? The numbers tell a different story when training is involved. For industries considering AI integration, the focus should be on educating users. It’s not enough to offer AI tools. users must know how to harness them effectively.

Is it time for companies to rethink their AI strategies? Absolutely. Without the right training, AI can be more of a hindrance than a help. This study serves as a wake-up call for organizations to prioritize AI education, ensuring tools are used to their full potential.