Self-Assessing AI Efficiency

When an attorney says "AI has made me more efficient", is that a reliable statement? There's good reason to think it may not be. As professionals whose stock and trade is evaluating the reliability of evidence, we definitely need to consider that possibility.

Two data points (both with their own reliability issues):

🖥️ - "[D]evelopers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%." This was one of the conclusions of a study, not about lawyers, but about LLM usage among open-source software developers. (Link to the article.) The study concluded that AI use 𝐬𝐥𝐨𝐰𝐞𝐝 developers down by 19%, but the developers thought it had sped them up by 20%. The study was published in July 2025 and so the data is from even earlier. You could, therefore, make a good argument that developer tools/knowledge have gotten better. But that doesn't change the point I'm focusing on that the participants were bad at assessing their own efficiency when evaluating their own AI usage.

✒️ - "Lawyers are terrible at estimating how long it will take them to do things." This is a quote from me from five minutes ago. 😇 It is, however, based on 20+ years working as a lawyer and with other lawyers. Still, that's admittedly not a scientific study. I will only add that, in addition to my personal experience with actual attorney estimates (including my own), it has also been my experience that this is one of the things that has made lawyers resistant to fixed-fee arrangements.

I'm not arguing that AI isn't making lawyers more efficient (or that it won't in the future). Nor do I think that the debate over that question at its full generality ("Does AI make lawyers more efficient?") is particularly relevant. AI is here in the legal world, and it ain't going anywhere.

To me though, it is an important question when it comes to the particulars. That is, it's pretty important to know 𝐞𝐱𝐚𝐜𝐭𝐥𝐲 𝐰𝐡𝐞𝐫𝐞 AI is and isn't making lawyers more efficient. That informs where law firms should be investing LegalTech dollars, where they should be encouraging (mandating?) AI usage, and conversely where they should be discouraging (prohibiting?) it.

And if I were the one making those decisions, I wouldn't be relying solely on lawyers reporting that they 𝐭𝐡𝐢𝐧𝐤 it makes them more efficient. I'd want data.