AI, technological cashouts and exam reform: Inside the California bar exam crisis

The controversy surrounding changes to the California Bar Exam, along with problematic administration of the February 2025 exam and troubling headlines involving the use of artificial intelligence by the State Bar to generate test questions, have triggered a chain of events that the State Bar is struggling to address for future exam sessions. These recent shortcomings have left both prospective attorneys and established members of the legal profession in a state of confusion, frustration and disappointment with the State Bar of California.

In August 2024, the State Bar announced that it would be awarding a no-bid, five-year, $8.25 million exam development agreement to Kaplan North America, LLC. This agreement was sought with the intention of closing a gap in the Admissions Fund, bringing change to the longstanding Bar Exam administered in California. This decision proposed changes to the content and distribution of the Bar Exam. Primarily, the new exam format would replace the longstanding National Conference of Bar Examiners' multiple-choice question set, the MBE, with Kaplan-created California-specific multiple-choice, essays and performance questions. Changes to the multiple-choice questions were set to replace the Multistate Bar Examination in time for the February 2025 exam. "State Bar, Kaplan, Sign Five-Year California Bar Exam Development Contract," State Bar of California (State Bar), (Aug. 13, 2024).

The February 2025 Bar Exam, which introduced Kaplan-created multiple-choice questions, encountered numerous complications. Technical difficulties disrupted test administration, and it was later revealed that AI was used in developing some of the multiple-choice questions of the examination. "Board Offers Free July Bar Exam to Applicants Who Withdrew or Fail February Bar Exam," State Bar, Feb. 21, 2025.

As stated in the State Bar's filed petition with the California Supreme Court, 29 out of 200 multiple choice questions, 23 of which were used for scoring, were developed with the use of AI, generated by ACS Ventures, LLC (ACS). After entering into the agreement with Kaplan, it was expected that multiple-choice questions for future bar examinations, including the February 2025 bar examination, would consist of Kaplan-drafted questions, along with State Bar provided First Year Law Students Examination (FYLAX) questions as source materials. It was later determined that there were not enough multiple-choice questions for each of the subtopics of the seven subject areas tested on the Bar Examination, therefore, ACS, a psychometric and test development consulting company was contracted to draft additional questions for the February 2025 Bar Examination. Prior to this, ACS was primarily contracted by the State Bar to assist with examination analysis, grading, and related services, not including the drafting of examination questions. ACS drafted question prompts were run through OpenAI and ChatGPT. The 29 ACS drafted questions are reflected in the chart below:

Bar examiners failed to disclose the enlistment of ACS Ventures to the legal community, test-takers or the California Supreme Court, which oversees the State Bar of California. Petition for Approval of Raw Passing Score and Scoring Adjustments for the February 2025 California Bar Examination, State Bar, (April 29, 2025).

In response to the extensive issues surrounding the administration of the February 2025 Bar Exam, the State Bar implemented remediation measures to address test takers' valid concerns, including offering fee waivers to individuals who either withdrew from the exam or did not pass. "Board Offers Free July Bar Exam to Applicants Who Withdraw or Fail February Bar Exam - The State Bar of California," (Feb. 21, 2025). The Board of Trustees also authorized an independent, privileged investigation into the exam's administration. "Board of Trustees Orders Independent Investigation into February 2025 Bar Exam Issues - The State Bar of California - News Releases," (March 6, 2025).

Amidst the challenges associated with the February exam, scores were understandably impacted. The various technical issues presented a magnitude of challenges, which only further added to the already stressful experience of taking the California Bar Exam. In response, the Committee of Bar Examiners (CBE) recommended scoring adjustments to account for the level of disruption. Specifically, the CBE proposed lowering the raw passing score from 560 to 534, applicable to examiners who answered at least 114 or 171 multiple-choice questions or completed 4 of the 6 written sections. "CBE Recommends Scoring Adjustments for February Bar Exam," State Bar, (April 21, 2025). On April 29, 2025, the State Bar filed a petition with the California Supreme Court regarding the CBE recommendations. The Court issued an order agreeing to those recommendations on May 2, 2025, allowing the CBE's recommendations to be applied to the February 2025 Bar Exam results. "State Bar Files Suit Against Vendor Responsible for February Bar Exam Administration," State Bar (May 5, 2025).

As a short-term solution to prevent the repeat of difficulties faced by test-takers, the State Bar Board of Trustees approved exam software and testing locations to administer the July 2025 California Bar Exam in person. "Board Approves Testing Locations, Vendor for In-Person July Bar Exam," State Bar (April 3, 2025). To ensure the success of the Bar Exam moving forward, the Board approved a resolution affirming guiding principles. "State Bar Board Discusses Bar Exam Lessons Learned, Remedies, and Moving Forward Options," State Bar (May 23, 2025).

The CBE proposed additional remedies for affected applicants such as "provisional licensure, a supervised proactive pathway to full licensure, and special admission status for attorneys licensed in other states." "CBE Recommends Scoring Adjustments for February Bar Exam," State Bar (April 21, 2025). February applicants have been granted a fee waiver available through the July 2026 exam. Furthermore, the Supreme Court of California approved for February exam takers who withdrew from or were unsuccessful on the February exam to apply for the Provisional License Program after its launch on Sept. 1, 2025. Cal. Sup. Ct., "Supreme Court Approves Expanding Provisional Licensure program," (June 11, 2025). The program is currently scheduled to sunset Dec. 31, 2027. Id.

As a result of the unsuccessful exam administration under the State Bar Board of Trustees, the Supreme Court of California has proposed rule amendments to give greater power to the CBE, recognizing its higher expertise in overseeing decisions made in connection with the California Bar Exam. Among other things, the amended rules include greater control to the CBE over exam development and administration such as a review of potential questions, the establishment of standards for selecting experts to review the exam, the development of standards for accrediting third party vendors who may administer and proctor future exams, and a mandated cost benefit analysis for any proposed changes to the Bar Exam. Cal. Sup.Ct., Order No. S290966-"Proposed Amendments to Title Nine of the Rules of Court Involving State Bar Matters," (May 29, 2025). In accordance with Section 6046.6 of the Business & Professions Code, the CBE must provide at least two years' notice before making substantial exam modifications. CA Bus. & Prof. Code § 6406.6 (2025). This aims to ensure fairness and transparency throughout the Bar Exam process for all candidates.

Following a 45-day public comment period, the Board of Trustees and the CBE held a joint meeting to discuss the proposed amendments by the California Supreme Court. "Board and CBE Consider Supreme Court's Admissions' and Exam Oversight Amendments," State Bar (Aug. 14, 2025). At the meeting, the Board and CBE discussed the proposed amendments, as well as the Provisional Licensing Program and future exam administrations. Id. The meeting resulted in the Board and CBE approving the Court's recommended revisions on rules primarily relating to CBE oversight and future changes to the Bar Exam. Id. A joint statement was released from Board Chair Brandon Stallings and CBE Chain Alex Chan to express their gratitude to the California Supreme Court for its leadership, emphasize both organizations shared commitment to serve all California Bar Exam applicants, and convey their collective aim to uphold high professional standards to promote the ethical practice of law. Id.

While the State Bar has shown effort to remedy the shortcomings in the administration of the newly adapted Bar Exam, the underlying goal of transitioning away from standard testing practices remains with the push towards remote testing and new exam material in affiliation with the Kaplan agreement. The results of the July 2025 exam will surely provide insight on what key concerns must still be addressed, and it will take time to determine whether all of the Board's and CBE's efforts will fulfill their intended goals while providing Bar examiners with a smooth experience.