Legal AI Can Fail Before It Answers

A Courtroom Finds the Fiction

On August 7, at a conference in Los Angeles Superior Court, lawyers for homeowner Fa'alagilagi Meni-Siliga told the court that State Farm's pretrial filings had a problem no argument could fix. The cases the insurer cited could not be found. Seven of them did not exist. Others were real decisions given the wrong names, or quoted as saying things no judge who wrote them ever said.

According to a later filing by Meni-Siliga's lawyer Eric Khodadian, State Farm attorney Kenneth Katel was visibly angry after the problem was raised and followed the plaintiff's lawyers into the hallway, asking to see their list of errors. Katel disputed that characterization, although he acknowledged later that he had been upset because the allegations were serious. They were.

Katel, lead trial counsel from the Los Angeles firm Musick, Peeler & Garrett, subsequently apologized to the court and the plaintiff. His co-counsel, Jacquelene Robinson, conducted a review and reported that seven case citations appearing in eight filings simply did not exist. Her review also found incorrect case names and purported quotations that the cited decisions did not contain.

There is a familiar version of this story by now. A lawyer asks an AI chatbot to conduct research, the machine invents cases, nobody checks, and the fiction makes it into court.

This case is more interesting because Robinson believed she had taken a safer route. She was using legal AI.

This Was Not Some Random Chatbot

Robinson said she used Irys, a platform built specifically for legal work. More importantly, she said she mistakenly believed that Irys was connected to her firm's Westlaw subscription and that it performed an internal citation check.

That belief changes the story.

Robinson was not describing a moment in which she knowingly substituted a general-purpose chatbot for legal research and hoped for the best. Her explanation suggests that she thought she was operating within a professional research workflow, with a recognized legal database somewhere behind it and verification occurring within the system.

It was not an insignificant technical misunderstanding. It affected what she thought had already been checked.

For readers outside the United States, State Farm is one of the country's dominant insurance brands. State Farm Mutual Automobile Insurance Company and its affiliates are the largest combined providers of auto and home insurance in the U.S., with roughly 96 million policies and accounts in force as of 2025.

In this suit, State Farm is the defendant, named as Meni-Siliga's homeowners insurer. The underlying dispute is painfully real. Meni-Siliga's home in Carson, California, was damaged by fire in 2020 and later suffered water damage during reconstruction. She alleged that delays involving State Farm contributed to repairs never being completed. In a court filing, she said her family exhausted savings and retirement funds, entered bankruptcy, and ultimately lost the home to foreclosure. State Farm has disputed liability and raised defenses to her claims. The case is scheduled for trial in October.

Into that litigation came authorities that existed only inside a machine-generated version of the law.

What Did the Software Actually Do?

The easiest way to misunderstand modern legal AI is to think of "legal AI" as a category with a predictable set of properties. It is not.

One system can generate prose without searching case law. Another can retrieve decisions from a legal database. A product may perform both functions but require the user to activate a separate verification feature. Citation checking itself can mean several different things, from confirming that a case exists to determining whether a quotation is accurate or whether the decision still represents good law.

Those differences are not cosmetic once a document is headed for court.

Current Irys documentation illustrates the point. Its Research & Cite Check materials describe a case-law search function powered by CourtListener and a separate Cite Check function designed to confirm whether citations exist, whether the citation string is accurate, and whether the authority remains good law. Irys also currently markets itself as an alternative to Westlaw and states that no Westlaw subscription is required.

That does not establish exactly what Robinson saw, which Irys configuration she used, or which features were available at the time. It does establish something narrower and more useful: a professional cannot safely infer the architecture of a legal AI product from its appearance, purpose, or location within a familiar workflow.

Robinson believed Westlaw was involved. That assumption apparently influenced her understanding of verification. A system does not become Westlaw because it feels like legal research.

Familiarity Can Be Borrowed

Westlaw has been part of American legal practice for decades. Lawyers understand what it means when they retrieve an opinion from Westlaw, run KeyCite, and open the underlying decision. That familiarity carries institutional weight.

Put an AI interface beside the same documents, give it legal terminology, make it produce properly formatted citations, and some of that trust can migrate to the new tool even when nobody consciously decides that it should.

This is one of the less discussed problems with professional AI products. General-purpose chatbots arrive with a visible warning label in the user's head. Most lawyers now know that ChatGPT can invent a case.

A specialized interface can feel safer precisely because it has been designed to look and behave like professional software.

The software may genuinely be better suited to the task. It may use retrieval, legal databases, and verification technology that dramatically reduce the risk. None of that makes every function interchangeable.

And "reduce" is the honest word for it. When Stanford researchers tested the leading purpose-built legal research tools, the products sold as the answer to hallucination were still producing it — Lexis+ AI on more than one in six queries, Westlaw's AI-assisted research on more than a third. The same researchers found that some vendors had overstated how close to hallucination-free their tools actually were. Being better than a consumer chatbot is not the same as being safe.

If a lawyer believes a research database has been queried when it has not, the problem starts before the output appears. If the lawyer thinks citation verification is automatic when it requires a separate action, the control exists only on paper.

A button that can check citations is useful. A workflow in which everyone assumes the button has already done so is something else entirely.

California Had Already Written the Warning

The timing makes the incident especially uncomfortable. In May, three months before these filings became public, the State Bar of California approved updated guidance on lawyers' use of generative AI. The document goes considerably further than telling lawyers to watch out for hallucinations.

California's guidance says technological competence includes developing a reasonable understanding of an AI system's capabilities, data sources, limitations, and risks before using it in legal services. Lawyers are then expected to apply independent judgment by reviewing and correcting AI-generated work.

The guidance is unusually relevant here because it addresses the machinery, not merely the answer.

Knowing that AI can hallucinate is no longer sufficient technological competence. A lawyer needs to understand what the particular system in front of them is doing. The same guidance makes the question of responsibility remarkably simple. A lawyer's duty of candor to the court cannot be delegated to AI, and AI-produced analysis and citations must be independently reviewed before submission.

In other words, even a perfect belief that a product contains excellent verification technology would not transfer the professional obligation to the product vendor. A mistaken belief about the product certainly cannot do it.

Lead Counsel Still Owns the Filing

Katel said he had not known that AI had been used in preparing the filings. He nevertheless accepted responsibility as lead trial counsel.

That is more than a ceremonial apology. It identifies the second control failure in the episode.

An organization can regulate AI use only if supervisors know where AI enters the work. A policy that says citations must be verified will not solve much if one lawyer thinks the software performs verification automatically while another lawyer does not know the software was used at all.

Musick, Peeler & Garrett told the court that it had changed its AI policy after the incident, although details of the new policy were not disclosed.

State Farm, meanwhile, said it was reviewing what happened. A spokesperson, Tom Hartmann, said the insurer expects its outside counsel to hold to the highest ethical and professional standards and to confirm that every filing is accurate. He did not answer questions about how many outside firms represent State Farm, or whether the company asks about their AI policies before hiring them.

That response raises a question that reaches well beyond this particular insurer. Large companies routinely entrust litigation to outside firms, and those firms are rapidly adding AI to research, drafting, and document workflows. Corporate legal departments therefore inherit part of the operational risk even when no employee of the company touched the tool.

There is a further irony here. This is not State Farm's first brush with courtroom AI hallucinations. In 2025, in a separate federal case, Lacey v. State Farm General Insurance Co., it was the lawyers suing State Farm who filed a brief built on AI-fabricated citations, drawn from an AI-generated outline nobody had verified. A special master struck the brief and ordered the two plaintiffs' firms, one of them among the largest in the country, to pay roughly $31,000, finding that their use of AI had misled him. There, State Farm was on the receiving end of the fiction. This time, its own counsel filed it.

"Legal AI" Is Not a Control

The legal profession has spent several years discussing hallucinations as an output problem. The model invents a case, and the solution is to check the case. That remains necessary, but the State Farm episode exposes an earlier question: what exactly is the person checking?

A lawyer needs to know whether the system generated a proposition from model output or retrieved it from an actual opinion. If research was performed, the lawyer needs to know which database supplied the authority. If the product offers citation verification, the user needs to understand whether verification occurs automatically or only after a separate process is run.

None of those questions requires the lawyer to become a software engineer. They require the kind of basic product knowledge already expected when professionals rely on any technical system.

There is a strange tendency around AI to regard misunderstanding the technology as an unfortunate but understandable side effect of innovation. In legal practice, that tolerance has limits. People lose money, property, liberty, and sometimes years of their lives through litigation. A court filing is not a sandbox for discovering what a button meant after the document has been submitted.

Robinson apologized and accepted responsibility. Katel accepted responsibility as lead counsel. Those responses are appropriate.

The more useful lesson comes earlier in the chain. Before the next AI-assisted filing leaves a law firm, the lawyer signing it should know what system generated the work and what sources that system actually searched. The lawyer should know whether verification really occurred, rather than assuming that a professional-looking product performed it somewhere in the background. And the authorities that carry the argument should still be opened and read.

If those things are unclear, the document is not ready for a court.