Audit faults State Bar for rushed February 2025 bar exam rollout

The state auditor has released a long-awaited report finding that rushed preparations preceded the disastrous February 2025 bar exam, which has cost the bar at least $5.7 million so far.

The report, released Thursday morning, found the State Bar of California rushed development of the multiple-choice section of the exam, then failed to disclose that it turned to artificial intelligence to help fill gaps. These errors were compounded by communications and oversight problems within the agency.

The audit also found one motivation for these moves was to reduce administrative costs, but these decisions had the opposite effect.

"The State Bar experienced no cost savings on the February 2025 bar exam and at least $4 million in revenue losses," the summary found. "The February 2025 bar exam alone will cost the State Bar at least $5.7 million, not including the costs of pending legal matters."

"We found that the State Bar's timeline and poor planning created challenges with the development of the exam's questions. ... The State Bar did little to prevent the technical and administrative failures that occurred in the February 2025 bar exam, despite similar problems having occurred during a pretest in November 2024," stated the "key findings" section of a summary released by State Auditor Grant Parks.

"Since the February 2025 bar exam, the State Bar has undertaken significant work to strengthen the administration of the California bar examination and improve our practices and processes including contracting, exam development and validation, and internal and external communication," said State Bar Board Chair José Cisneros in an emailed response. "We fully acknowledge the unacceptable experiences exam takers faced leading up to and during that exam."

The statement also said the bar accepts all six recommendations from the auditor's report and has already begun making improvements to exam planning, question review, contracting, vendor oversight and communication. The last two bar exam administrations were held in person and used multiple-choice questions from the National Conference of Bar Examiners, without the widespread disruptions that marred the February 2025 exam.

The exam made national headlines after hundreds of test takers reported problems ranging from confusing and incorrect questions to dropped connections on remote exams. These stemmed from decisions the bar made to develop its own multiple-choice section and "to provide both remote and in-person options" to test takers.

The bar hired Kaplan Exam Services LLC to develop the new questions but then failed to specify all the topics that needed to be covered. The audit found that delays in conducting the search for a vendor and finalizing a contract left just six months to develop and validate questions for the exam. The National Conference of Bar Examiners can take up to three years to develop multiple-choice questions, according to the audit.

Those issues led directly to one of the most talked about problems with the exam: confusing questions developed by AI. The bar turned to ACS Ventures LLC, a decision that led to 29 questions developed with AI. Those questions were three times more likely to show "performance issues," such as correlating badly with a test taker's performance on other sections of the exam, and almost twice as likely to be among the more than two dozen questions removed from scoring.

Mary Basick, assistant dean for academic skills at UC Irvine School of Law, was among a small group of law school staff and bar exam experts who began warning months ahead of the February 2025 exam that the new test was not ready for use. She told media outlets last year that she and other experts were denied the chance to look at questions before the exam.

"I thought their question development process was bad, which is why I was trying to sound the alarm, but even a quick glimpse at this report reveals it was much worse than I had imagined," Basick said by email on Thursday. "Not only did they use only one subject matter expert for the question review, but they did not even have that expert review all the questions that appeared on the exam."

The problems extended even to "benchmark questions," several of which were "not reviewed in advance by the subject matter expert and were later found to be legally inaccurate and had to be removed," she said. These are questions included on the exam that are supposed to be well-vetted and where test takers' performance correlates closely with their performance on the rest of the exam. Basick also said the bar "included six questions on the exam that they knew in advance were legally inaccurate."

Similar problems plagued the bar's dealings with Meazure Learning, which the bar contracted with to offer the remote exam. The audit found the "State Bar did not clearly define its exam administration requirements" and failed to verify the company's ability to administer the exam.

The report included a June 18 letter from Cisneros and new State Bar Executive Director Laura Enderton-Speed. They wrote that the bar has already started making changes, including better exam planning, stronger question review, tighter contracting rules and clearer procedures for elevating major decisions.

"While we agree with the recommendations, we disagree with certain findings in the report," they wrote. "The State Bar is in active litigation with Meazure Learning as of the date of this response, the State Bar will not address its concerns with those findings here."

The bar sued Meazure for fraud, negligent misrepresentation, and breach of contract weeks after the bar exam debacle. Its attorneys filed an amended complaint in March after obtaining internal memos from the company in discovery. State Bar of California v. ProctorU Inc., d/b/a Meazure Learning, 25STCV13089 (L.A. Super. Ct., filed May 5, 2025).

The bar responded to the problems by lowering the passing score and regrading hundreds of individual exams, resulting in the highest pass rate in 60 years. The fallout led to the departure of several bar staff, including Executive Director Leah Wilson. Last fall, Gov. Gavin Newsom signed SB 253, requiring the bar to return to use the Multistate Bar Exam for multiple-choice questions and implement several transparency provisions.

Senate Judiciary Chair Tom Umberg, D-Santa Ana wrote both SB 253 and the bill requiring the bar audit.

"The State Bar failed to put the necessary safeguards in place on its own, so the Legislature did," Umberg said in an email. "That is why I authored SB 47, requiring this independent audit to provide a full accounting of what went wrong. The State Auditor has now issued several recommendations, and I expect the State Bar to fully implement them so future applicants never face another preventable failure like this."