New York Federal Judge Bans Unsupervised AI Use in Petitions
In an unprecedented decision, a New York federal judge has prohibited the use of generative artificial intelligence to draft judicial petitions without human supervision. The measure came after a lawyer submitted a document containing fake citations fabricated by an AI tool, highlighting the risks of 'hallucinations' in legal proceedings.
The Case That Prompted the Decision
The incident occurred when a lawyer filed a petition with references to non-existent case law, all generated by a language model. The court discovered the citations were fictitious, leading to legal embarrassment and the need for retraction. Although the lawyer claimed ignorance of the AI's flaws, the judge ruled that human oversight is indispensable to ensure the integrity of legal documents.
Hallucinations and Legal Risks
'Courts cannot blindly trust systems that invent information,' the judge emphasized in his order. The so-called 'hallucinations'—when AI generates false data that appears credible—pose a particular danger in the legal environment, where accuracy of citations and precedents is paramount. The decision aims to prevent similar errors from compromising justice.
Reactions and Impact
The American Bar Association supported the measure, emphasizing that AI tools can be useful as aids but must never replace human judgment. Lawyers specializing in legal technology warn that the decision may inspire courts in other states to adopt similar rules. Conversely, critics argue the ban could limit innovation and efficiency in the sector.
The Future of AI in Courts
Experts suggest the solution is not to ban AI but to establish rigorous verification protocols. Legal technology companies are already developing systems that automatically flag potential hallucinations. Meanwhile, New York's decision serves as a warning: ultimate responsibility for the content of a petition always lies with the lawyer.
Conclusion
Whether the ban on unsupervised AI use in U.S. courts becomes temporary or permanent will depend on future cases and regulations. For now, it is clear that technology, while promising, still requires caution to avoid undermining the credibility of the judicial system.