YOUR LAWYER MIGHT BE FILING 'HALLUCINATING' AI INFORMATION ON YOUR CASE

• The Florida Supreme Court has issued an opinion on the use of AI in court filings, amending its rules to require attorneys to verify the accuracy of legal authorities cited in court filings.
• This change comes after several instances of lawyers and self-represented litigants using AI that generated fictional or "hallucinated" legal cases.
• Florida joins several other states in an effort to regulate the use of artificial intelligence in the legal profession.
• Legal experts and The Florida Bar advise attorneys to meticulously verify any information generated by AI tools before submitting it to a court.

Judges are at wit's end with attorneys using artificial intelligence, inserting fictions in their pleadings and even manufacturing evidence. With lower courts howling on the issue, the Florida Supreme Court became the latest to put its foot down.

Generative artificial intelligence tools are being used as drafting or research aids for court filings, the state’s high court said in its recent opinion: “Though these tools can be helpful, they also can generate content that appears plausible but is in fact inaccurate, including fabricated or 'hallucinated' authorities."

The justices amended the rules governing state attorneys to include new language requiring attorneys signing pleadings to represent that the legal authority that is identified actually exists and is accurately cited. The amendment authorizes courts to impose appropriate sanctions.

It's been a tough week for AI. Besides the high court's opinion, Florida's Attorney General James Uthmeier sued OpenAI, alleging that the company marketed ChatGPT over alleged safety lapses, such as collecting data from minors. President Trump also signed an executive order seeking more oversight over AI models.

Florida Supreme Court moves to rein in attorneys using AI in legal filings

John Morgan, the founder and face of powerhouse Morgan & Morgan, told USA Today Network-Florida that his firm learned one of the early lessons on the use of AI when one of its lawyers generated fictional supporting cases in a federal case filing.

“We were early adopters of AI, we were moving fast, and an attorney used it improperly. We took it as an immediate opportunity to learn and grow,” Morgan said.

Florida joins the supreme courts in Illinois, Ohio, Pennsylvania, Montana and South Carolina in an effort to rein in the use of AI. Other states — New York, New Jersey, Connecticut — have created task forces within their state judicial branches.

Generative AI for legal research can produce up to 33% falsehoods, study found

Brent Hartman, an Orlando attorney with Rumberger Kirk, said jurists must rely on attorneys to truthfully cite precedent and fact patterns.

“Judges in this state are incredibly intelligent, but they cannot be expected to have memorized every piece of law and every prior decision,” he said. “So they rely on the people who are in front of them, making arguments and submitting filings for the court to review and make rulings upon.”

The Supreme Court heard from appellate courts, Hartman said, on the growing concern of AI and the need to create a uniform standard.

“This is an issue they are facing with growing frequency,” he said.

According to one study, generative AI programs for legal research invent legal authorities between 17 and 33% of the time, one state appellate court cited in December.

Judge Jonathan Lott of Florida’s Fourth District Court of Appeal in West Palm Beach wrote in another opinion that AI programs “do not ‘think” but instead make predictions about what words ought to come next in response to a prompt.

“This technology is very good at sounding right, but less adept at being right,” Lott wrote in March to address a pro se defendant in a contract dispute who used AI. 

While the use of AI is prevalent among attorneys, it has caught fire among defendants who defend themselves, known as pro se litigants. The well-worn saying that "a man who represents himself has a fool for a client” may need to be updated for when a defendant farms out legal work to ChatGPT, Claude or Gemini.

These litigants stand to be sanctioned monetarily by the court if taken down the wrong path in their filings by AI. In one case, also out of the 4th DCA,  a pro se litigant facing sanctions argued he had a right to use AI under the Americans with Disabilities Act “because retaining counsel would require him to pay five or six figures in attorney fees and instead he can pay $20 a month for an AI service," Hartman said.

Florida Bar's advice in using AI: Verify, verify, verify

Co-chairs of the Florida Bar’s Special Committee on AI Tools and Resources say they saw the collision of AI and legal filings as soon as tools like ChatGPT became widely available. The committee spent its first year focused on ethics and discipline — pushing for rule changes to spell out lawyers’ obligations when using AI. 

“These tools are very powerful, very efficient, but they are not perfect, and that’s where the challenge comes in,” said committee member Duffy Myrtetus, a Richmond, Va., attorney who also practices in Florida. “You can only really be effective if you’re going through and checking and verifying and validating.” 

When embarrassing stories started to emerge of lawyers hoisted by their own AI, their colleagues said they could never allow that to happen. “But in state and federal courts around the country, it continued to happen,” Myrtetus said.

While it is easy to point fingers and blame laziness, it is also important to note the immense pressure on young associates of law firms trying to make court-imposed deadlines.

Gordon Glover, committee member and an attorney in Ocala, said the mistakes With AI have gone beyond just citing make-believe cases within pleadings. He said evidence, such as deep-fake videos, have been introduced in some cases.

“It’s important to emphasize: verify, verify, verify everything — whether you’re a pro se litigant, an attorney, or just the public generally utilizing these tools,” Glover said.

Why Morgan & Morgan ran into AI problems

In February 2025, three attorneys with Morgan & Morgan were fined $5,000 for producing a pleading with “hallucinated” case citations in high-stakes litigation targeting Walmart involving e-bike batteries catching fire. U.S. District Judge Kelly Rankin’s rebuke has now become the standard bearer: “While technology continues to evolve, one thing remains the same–checking and verifying the source.”

Morgan said the incident led to a robust overhaul of the firm’s internal policies, “not just to fix what went wrong, but to build proactive fail-safes for the future.” Like it or not, AI is not going anywhere, he said.

“You either learn to use it responsibly or get left behind. The Florida Supreme Court’s new rule is a great first step toward acknowledging that reality and ensuring the integrity of our profession,” Morgan said.

John Pacenti is the Government Impact Reporter for The USA TODAY NETWORK-FLORIDA. You can get all of Florida’s best content directly in your inbox each weekday day by signing up for the free newsletter, Florida TODAY athttps://palmbeachpost.com/newsletters.