Courts Crack Down as AI 'Hallucinations' Produce Fake Citations in Legal Briefs

When a Louisiana district judge discovered fabricated quotations in a lawyer's brief earlier this year, the attorney admitted he had relied on Claude, an AI chatbot, to draft the filing. The disclosure led to a $1,000 fine and a mandatory three-hour course on AI-assisted legal practice — an episode courts say is emblematic of a growing problem.

What Happened

Judge Jerry Edwards Jr. reviewed a motion to exclude evidence in a personal-injury case and found seven quotations attributed to prior rulings that either do not appear in the cited opinions or were materially mischaracterised. According to the court's May 18 memorandum, the lawyer had asked Claude to revise a draft after a law clerk flagged fabricated quotes, then filed the revised brief without sufficient review. The judge wrote that 'ignorance of the risks of AI usage is no longer an excuse.' He imposed a $1,000 fine and ordered training, while still granting part of the underlying motion.

Scale And Patterns

French lawyer Damien Charlotin, who has been compiling a crowd-sourced database of court filings that contain AI-generated errors since April 2023, says the problem extends far beyond a single case. His collection now includes roughly 1,600 examples from 35 countries: fake quotations and case citations, plagiarised passages, incorrect references and out-of-date legal advice. By far the largest share — 1,116 examples — come from the United States, followed by Canada (173), Australia (74) and the UK (59).

Charlotin's analysis indicates that hallucinations in filings have increased about eightfold over the past year compared with the previous 12 months. Many of the offending documents were produced with mainstream chatbots such as OpenAI's ChatGPT; specialised legal AI tools appear more reliable but are not immune to errors.

Consequences And Court Responses

Judges across jurisdictions have warned that fabricated citations waste court time, burden opposing parties and can harm the reputations of judges whose names are falsely invoked. New York State Supreme Court Judge Linda Kevins highlighted those risks in a January ruling. In Mississippi, federal Judge Sharion Aycock recently fined four lawyers $8,000 in total and barred two from appearing in the Northern District of Mississippi for two years after briefs cited non-existent cases, calling the episode a warning against acting as a 'rubberstamp' for unverified AI output.

Why AI Hallucinates And What Lawyers Should Do

Charlotin explains that AI models excel at pattern recognition and may fabricate plausible-looking legal citations by following familiar formats. Because no practitioner can know every precedent, any citation produced by AI must be verified by checking the primary source. Judge Scott Schlegel, a member of the American Bar Association's Task Force on Law and Artificial Intelligence, urged attorneys to read and verify cases themselves rather than relying blindly on AI-generated drafts.

Practical takeaway: AI can speed research and drafting, but lawyers must adapt verification and vetting processes, double-check citations against primary sources, and treat AI as an assistant rather than an authoritative source.

Looking Ahead

Some developers and researchers, including Charlotin, are building tools to detect hallucinations before filings are submitted. Those tools may help, but courts emphasise that responsibility ultimately rests with the attorney who files the document. As judges begin to impose fines, training requirements and even temporary bans, legal professionals are facing growing pressure to update their workflows and preserve professional responsibility in the age of AI.