(Image via ChatGPT)
If you’re a Pennsylvania lawyer wondering whether you need to disclose AI use in your court filings, Google’s AI summary has an authoritative answer for you. It’s a wrong answer, mind you. But authoritative!
As Christine Lemmer-Webber put it, it’s “Mansplaining as a Service” — unwavering confidence, no matter how incorrect.
If one opens an incognito window and searches Google for “Pennsylvania AI disclosure lawyers,” the AI-generated summary will explain that “Key developments include mandatory disclosure of Generative AI (GAI) in court filings.” Throw in “August 2024” because you vaguely remember seeing something about AI on that date and the result reads “As of August 2024, Pennsylvania mandates explicit disclosure of AI use in all court submissions, making transparency a mandatory filing requirement.”
None of that is true.
The Legal AI Governance tracker, an invaluable tool maintained by Brian Alenduff of Desired Path Consulting, provides a comprehensive rundown of Pennsylvania’s AI rules. There are standing orders in some courtrooms, and the state supreme court issued a rule governing court personnel only, but as for the state of Pennsylvania writ large, there is no statewide rule as of now. The tracker notes that what Pennsylvania does have is Joint Formal Opinion 2024-200, a 2024 advisory ethics opinion from the Pennsylvania Bar Association and the Philadelphia Bar Association flagging AI as a competence issue under existing rules. But that opinion explicitly states that it is “advisory only and is not binding.” The ABA’s own 50-state survey classifies Pennsylvania as “court dependent.”
Scrolling down to the search results would reveal to anyone interested that there’s not really a mandatory disclosure policy. So how did it end up in the Google summary? Alenduff investigated and believes it first emerged from a vendor blog post.
Google flags a Paxton.ai post titled “2025 State Bar Guidance on Legal AI” among its sources. That post includes a bullet point that reads:
Pennsylvania: Mandates explicit disclosure of AI use in all court submissions. Transparency isn’t optional; it’s a filing requirement. Implementation: August 2024.
From there it seems that Google’s AI Overview inhaled that sentence and started serving it back to lawyers as the law of the Commonwealth. Hallucinations are all the rage right now, but over the long haul the greater AI risk will be an unfailingly credulous bot elevating and validating mistakes until the error gets picked up as reality. It’s a chicken and the egg problem, but another post by Twinladder.ai repeats Paxton’s language. Whether they got it from Google AI or their mistaken citation led Google to consider the mistake independently confirmed doesn’t really matter — it’s out there now.
This is the hot dog champion problem for lawyers. Journalist Thomas Germain recently seeded a few corners of the web with the claim that he was a competitive hot-dog-eating world champion, and watched ChatGPT and Google Overviews dutifully repeat it. Funny when it’s hot dogs, but decidedly less funny when it’s “the ethical rules binding the legal profession.”
This is the compression problem that AI brings. As those developing this technology push to condense the legal workflow into a series of “agentic” steps flowing from input to deliverable, it’s both obfuscating the process and robbing lawyers of the slow-thinking opportunities that produce optimal work. Lawyers become — even against their conscious instincts — more trusting of output that looks like a polished finished product. They also become incentivized to move on from a project faster, sacrificing the down time between turning drafts that used to birth epiphanies.
And miss me with the “human in the loop” platitudes. How often has Google’s AI produced a summary that you took as gospel without bothering to scroll down? I’ll go you one better… Google includes links to its sources, but how many times have you actually clicked on one of them? I’m willing to wager you — at best — registered that it cited something and counted it as a win without bothering to consider what it cited or whether the underlying source was right. Human in the loop is good ad copy, but in practice it’s a glorified Google source panel: an invitation to check that lawyers will never click on. As long as the bot purports to be showing its work, then it must be correct, right?
We risk a tech-enabled speedrun of qualified immunity. The doctrine of qualified immunity rests on a literal scrivener’s error, as an 1874 federal compilation simply deleted language that Congress wrote into Section 1983 specifically rejecting any state law limitations on the new cause of action. From then on, courts acted upon the error. AI can absorb and uncritically repeat false claims at scale and every time someone falls for the error, the bot interprets that as more proof that it’s right. The redux of the qualified immunity problem now accelerated by technology instead of just racism.
Speaking of racism… imagine what AI will do for originalism. There’s a growing industry of amateurs cosplaying as historians compiling disingenuous claims into secondary sources. Judges already misapply precedent to smooth the way for entrenching this revisionism into the law. Create enough of an echo chamber of half-truths and all of a sudden it’s “fact” by the bot’s reckoning.
The answer, as always, is that lawyers need to remain highly skeptical of AI conclusions. It can be a powerful tool for pointing users in the right direction, but no matter how authoritative it sounds, it’s only an invitation to deeper human research and not a replacement.
In other news, did you know that I’m also a former world hot dog eating champion? If you don’t believe me, check Google in a few weeks.
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter or Bluesky if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.