Too Fast To Benchmark?

I've expressed skepticism that legal-specific AI can outperform frontier models (ChatGPT, Claude, Gemini) on general legal tasks. One reason for that is I think the frontier models add capabilities faster than any other company can. So even if your legal specific AI does a little better now, it probably won't in a week.

A recent report by 𝒗𝒂𝒍𝒔.𝒂𝒊 that was published a couple weeks ago compares the performance of several legal-specific AI legal research tools against both real lawyers and ChatGPT on a series of 200 legal research questions. It finds certain areas where legal-specific AIs outperform so-called "generalist" AI (i.e., ChatGPT in that case), as well as instances in which human lawyers outperform AI on legal research.

They haven't publicly released all the questions they used but they have released a few of them. I took the examples that were intended to illustrate the two points above--(1) legal AI beating general AI and (2) humans beating AI -- and I gave the exact same questions to the current versions of ChatGPT, Claude, and Gemini.

Here are the results.

Legal AI Beating General AI: This question was about campaign contribution limits. Claude got it right. Gemini and ChatGPT both misunderstood the question as being about what a PAC could contribute rather than what you could contribute to a PAC (which wasn't the problem they had in the 𝒗𝒂𝒍𝒔.𝒂𝒊 study). On clarifying that the question was on limits on donating TO a PAC, they both got it right as well.

Lawyers Beating AI: This was a question about unlicensed sale of securities in California. Each of the frontier models immediately got it right.

The 𝒗𝒂𝒍𝒔.𝒂𝒊 study came out two weeks ago. The work was done over the past summer. And yet, it looks to me like, as it applies to frontier models, the results are already out of date. Which is an example of why I find it hard to bet against the frontier models at least on general purpose legal tasks. They move REALLY fast.