ChatGPT-4 Understands Academic Attrition’s Impact on Bar Passage, but Does Anyone Else?

Rory Bahadur and Kevin Ruth†

In a recent article, Access Lex cites in part to our research and concludes:

Some researchers have expressed concerns about certain schools using attrition and transfer policies to inflate bar passage rates (Bahadur et al. 2021), an interesting thesis worthy of attention considering the implications for academic and bar success professionals whose efforts to prepare law students for the bar exam could be undermined if certain institutions are found to take shortcuts to bar passage improvement.  However, our findings do not indicate that this is happening on a national scale.  Although attrition rates are positively associated with pass differential on average; transfer-in rates are negatively associated.  (Transfer-out rates do not appear to have a meaningful impact on pass differential.)[1]

Access Lex perhaps misinterprets what our research posits.  Our research posits only the following regarding academic attrition:

Ultimately, our sole assertion is that before an individual school’s bar performance is measured, the impact of academic attrition and transfer students on that particular school’s bar performance should be examined.  Our exact statement is “substantial changes in class composition [due to academic attrition and transfer] between matriculation and bar passage — should not be discounted as an explanation for why a school performs better on the bar exam than its entering credentials predicted.”[6]  We note the italicized language refers to an individual school and what might be causing observed bar passage at that school, and in the research, we do not perform any statistical correlative analysis but analyze each school individually.[7]

We feel it is important to disentangle this lingering and distracting confusion between causation and correlation.  We were surprised that ChatGPT-4 understood the difference in a way that was clear and concise in only three question and answer sequences.  We recreate here the transcript of the ChatGPT-4 query in this regard.

Question 1 to ChatGPT-4:

Bar exam pass rates are measured as number of passers divided by number of takers.  Does academic attrition affect bar passage rates and if yes, explain how confounding variables make this correlation of academic attrition and bar pass rates difficult to detect on a national scale?

ChatGPT-4 Response to Question 1:[8]

Yes, academic attrition (the rate at which students leave their studies) could potentially affect bar exam pass rates, particularly if a significant proportion of students who are at risk of failing the bar exam drop out of law school before they have the chance to take the exam.  This could artificially inflate the pass rate because the pool of students taking the exam might be more academically strong on average than the overall student body.

However, there are several confounding variables that can make it difficult to detect a clear, causal relationship between academic attrition and bar pass rates on a national scale:

(1) Institutional Differences: Law schools vary greatly in their admission criteria, academic rigor, student demographics, and institutional resources (like bar exam preparation services), all of which can influence both academic attrition rates and bar pass rates.

(2) Selection Bias: Law schools might also have different policies for addressing students who are at risk of failing the bar exam. Some schools may offer additional support to these students to help them pass, while others might encourage them to drop out.  This selection bias could make it seem as though academic attrition is having a larger effect on bar pass rates than it actually is.

(3) Geographical Variation: Bar exams are administered by individual states, and their difficulty can vary from state to state. So a school's location can also have a significant impact on its bar pass rate, independent of its academic attrition rate.

(4) Economic and Social Factors: Economic conditions, student mental health, personal circumstances, and other external factors can impact both a student's decision to leave school (academic attrition) and their likelihood of passing the bar exam. These factors can be hard to measure and control for in analyses.

(5) Time Lag and Cohort Effects: The effects of academic attrition on bar pass rates may not be immediate and could be influenced by cohort effects. For example, a particularly strong or weak class could skew the results for a specific year.

All of these factors make it difficult to draw clear, causal conclusions about the relationship between academic attrition and bar pass rates on a national scale.

Question 2 to ChatGPT-4:

Could you list more confounding variables relevant to this query?

ChatGPT-4 Response to Question 2:[9]

Absolutely, there can be other confounding variables that make it difficult to establish a clear relationship between academic attrition and bar exam pass rates.  Here are a few additional variables:

(1) Curriculum Differences: The curriculum offered by different law schools may vary significantly, affecting students' preparedness for the bar exam and potentially influencing both attrition and pass rates.

(2) Faculty Quality and Staff-Student Ratio: The quality of teaching, faculty's dedication and availability, and the staff-student ratio at different law schools can also affect students' academic success and their readiness for the bar exam.

(3) Student Motivation and Resilience: Students' individual motivation levels and resilience to stress can play a significant role in whether they decide to leave their studies and how they perform on the bar exam.

(4) Pre-Law Academic Background: Students' academic performance before attending law school (such as undergraduate grade point average (GPA) or Law School Admission Test (LSAT) scores) can influence both their likelihood of dropping out and their success on the bar exam.

(5) Financial Resources and Support: Financial factors, such as the availability of scholarships, financial aid, or familial financial support, can affect a student's decision to continue in law school and their ability to dedicate time to bar exam preparation.

(6) Support Systems: The presence or absence of strong support systems, including family, social connections, mentorship programs, and mental health resources, can significantly affect a student's ability to handle the pressures of law school and the bar exam.

(7) Language and Cultural Factors: Students for whom English is not their first language, or those from diverse cultural backgrounds, may face additional challenges that could influence their persistence in law school and success on the bar exam.

(8) Law School Ranking: The reputation and ranking of the law school could also play a role, as higher-ranked schools may have lower attrition rates and higher bar pass rates.

These variables highlight the complexity of studying academic outcomes like attrition and bar pass rates.  The interaction of these factors can be complex and multi-directional, which further adds to the challenge of identifying clear, causal relationships.

Question 3 to ChatGPT-4:

Are there even more confounding variables involved than the ones you listed?

ChatGPT-4 Response to Question 3:[10]

Absolutely, understanding and examining the complexity of academic attrition and bar exam pass rates involves a vast array of potential confounding variables.  Here are some additional [confounding variables]:

(1) Learning Environment: This includes the overall culture of the law school, levels of competitiveness among students, and the balance between theoretical and practical teaching.

(2) Work-Life Balance: The ability to manage studies along with personal responsibilities like family commitments, part-time work, or caring for a relative can affect attrition rates and bar exam performance.

(3) Access to Study Materials and Resources: The availability and quality of study resources, such as textbooks, libraries, online resources, and prep courses, can also impact both academic attrition and bar exam success.

(4) Health Factors: Chronic illnesses, mental health conditions, or other health-related issues can impact a student's ability to continue in their studies and prepare effectively for the bar exam.

(5) Extracurricular Activities: Participation in moot courts, law reviews, internships, or other law-related activities can influence a student's academic performance and preparedness for the bar exam.

(6) Legal Market Conditions: The job market prospects for law graduates can impact both attrition and bar pass rates. If the job market is tough, some students might drop out, while others might be more motivated to pass the bar exam.

(7) Distance Learning or In-Person Classes: The mode of learning can also impact attrition and bar exam pass rates, especially in light of the increased use of online learning due to the COVID-19 pandemic.

(8) Policy Changes and Educational Reforms: Changes to educational policy, bar exam formats, or the law school curriculum can have a significant impact on both attrition rates and bar exam pass rates.

These are just a few additional examples.  The true number of potential confounding variables is vast, and many of these factors interact with each other in complex ways.

The italicized text in the response to question 1 indicates that causation and correlation are different entities and why the search for correlation on a national scale is not relevant to our documented impact of attrition on bar pass rates.  Our research addresses only the causation aspect of the relationship between academic attrition and bar passage rates.  We conclude that national level regression analysis is unsuitable for discerning the impact of academic attrition on bar passage because numerous confounding variables impact the analysis resulting in an underestimation or obscuring of the mathematically undeniable causative effect of academic attrition on bar passage rates.[11]  Our research does not involve correlation.  Any other interpretation of our research is not tenable.

It is mathematically undeniable that academic attrition rates impact bar passage at an individual school.[12]  That is the entirety of our research findings regarding attrition.  Any suggestion that we posit anything more than this reality is erroneous.  Similarly, related attempts to refute claims we did not make about national trends and institutional motivations using regression analysis mischaracterize the nature of our research.

Regarding transfer students, the Access Lex article also ignores the established reality that some schools only transfer in high GPA students who are very likely to pass the bar[13] while other schools are less discerning regarding the GPAs of the students transferring in.[14]  Further, some schools also lose the vast majority of their better students, who are most likely to pass the bar, via transfer, before those students write the bar examination, depressing the schools’ bar passage rates.[15]  Jerry Organ reiterates,

[I]t is not just that transfers vary widely across law schools, but that a number of law schools lose their better students (diminishing bar passage performance) while others gain people capable of passing the bar (and likely replacing the attrited students who are less likely to pass the bar) [and] a large number of schools are not really participating in the transfer market at all. . . . This is not something that should be analyzed on a national basis.[16]

Similarly, attrition patterns at various schools are so variable, even among schools with similar matriculant credentials, that it should not be analyzed nationally, but the impact of attrition on each school should be examined individually by comparing what bar passage would be if the academically attrited students were not attrited.[17]  In fact, “global statistical observations obfuscate the true relationship between attrition and bar passage at individual schools.”[18]

And while ignoring these ambiguities in the realities of attrition and the transfer market is problematic, the more significant problem is that it introduces ambiguity where there is none.  By failing out its weakest students before they take the bar examination, an institution improves its bar passage rate or differential.  The fact that numerous confounding variables render this reality difficult to detect with regression analysis, whether described as multilayered or sophisticated, is not relevant to this reality.

In our capitalist environment, commercial entities profit from selling bar success services.  Theories that obfuscate or camouflage the simple reality of academic attrition’s impact on bar passage rates increase the perceived importance of those bar success services.  But more importantly, this unnecessary ambiguity is a harmful distraction from, or an attempt to gentrify with academic nuance, the brutal game of improving bar results some institutions play by failing out their weakest students before they take the bar exam.  This is especially important considering U.S. News just increased the impact of bar passage on schools’ rankings.[19]

Deemphasizing the plain and obvious effect academic attrition has on an individual school’s bar passage rate with inapplicable national scale regression analyses minimizes an obvious and cost-free tool for institutions[20] to improve bar passage rates.  This deemphasis of attrition simultaneously overemphasizes the efficacy and relevance to bar passage rates of the bar improvement tools that are sold by bar preparation vendors.

And, as we note repeatedly, continuing to analyze bar performance as indicative of value added or “the single best outcome measure to consider in assessing whether a law school is maintaining a ‘rigorous program of legal education’”[21] buttresses systemic racism[22] and distracts from the reality that the bar examination does not test competence to practice law.[23]  We live in a world where multitudes of honest people make a living by perpetuating the notion that the bar exam tests competency and selling effective materials to improve bar performance.  However, the bar exam as a test of competence to practice law is a stubborn untruth which is often skillfully and subtly advocated for.[24]

Circling back to ChatGPT-4, if the bar exam tested competence to practice law, then we should close all law schools and save the public some money, because ChatGPT-4 aced the bar examination, performing at the 90^th percentile, so it is presumably supremely competent to practice law.[25]  Ironically, all the scientific and statistical studies, subtly promoting and advocating for the continued use of the bar exam and the validity of Standard 316, are therefore suggesting our industry is ready to be replaced by a $20/month subscription to OpenAI.[26]

†Rory Bahadur is the James R. Ahrens Chair of Tort Law at Washburn University School of Law and Dr. Kevin Ruth holds a Ph.D. in mathematics from the University of Miami and specializes in games of chance.  The authors would like to thank Jerry Organ, Bakken Professor of Law and Co-director of the Holloran Center for Ethical Leadership in the Professions, University of St. Thomas School of Law for generously sharing his expertise on attrition and transfer and reaffirming the flaws in trying to discern the impact of attrition and transfer using national regression analyses.  We would also like to thank Professor David Rubenstein, James R. Ahrens Chair of Constitutional Law, at Washburn University School of Law for insisting that we familiarize ourselves with AI and employ it in this article.

[1]Jason M. Scott & Joshua Jackson, Are Law Schools Cream-Skimming to Bolster Their Bar Exam Pass Rates? 22 (Access Lex Inst., Working Paper No. 22-03, 2023), https://ssrn.com/abstract=4100637.

[2]Correspondence from Jerry Organ, Bakken Professor of L., Univ. of St. Thomas Sch. of L., to author (May 23, 2023) (on file with author) (“If a school has 100 students enter the school as 1Ls and [only] 10 [academically] attrit at the end of the first year, and one could have imagined (in the absence of attrition), that perhaps 85 of the 100 would pass the bar.  But if 10 who are most likely to fail the bar exam are attrited, then the graduating class is 90 and it well may be that the school sees 85 out of 90 pass the bar.  This means the pass rate increases from 85% to 94%.”).

[3]Rory Bahadur, Kevin Ruth & Katie Tolliver Jones, Reexamining Relative Bar Performance as a Function of Non-Linearity, Heteroscedasticity, and a New Independent Variable, 52 N.M. L. Rev. 119, 200–02 (2022).

[4]Rory Bahadur & Kevin Ruth, Quantifying the Impact of Matriculant Credentials & Academic Attrition Rates on Bar Exam Success at Individual Schools, 99 U. Det. Mercy L. Rev. Online 6, 23–25 (2021) (demonstrating how bar passage rates at the top credentialed Florida schools changed significantly when academically attrited students were added back into the denominator of the takers/passers fraction).  We were not able to examine the impact of academic attrition on the 2019 bar exam results because of the anomalous nature of the ABAs recording of academic attrition in 2019.  See id. at 23.

[5]Bahadur et al., supra note 3, at 200–03.

[6]Id. at 158 (emphasis added).

[7]Id. at 160–99.

[8]Text generated by ChatGPT-4, OpenAI, May 22, 2023, https://chat.openai.com/chat.

[9]Id.

[10]Id.

[11]See Rory Bahadur & Kevin Ruth, Bad Math Bar Sauce and the ABA as a Shill for the NCBE, 66 How. L.J. 323 (2022) [hereinafter Bad Math]; see also Bahadur et al., supra note 3.

[12]Bahadur et al., supra note 3; Bahadur & Ruth, supra note 4.

[13]See, e.g., Rory Bahadur, Blinded by Science? A Reexamination of the Bar Ninja and Silver Bullet Bar Program Cryptids, 49 J.L. & Educ. 241, 253–54, 254 n.56 (2020) (“[T]he only students admitted as transfer students [to Florida International University (FIU)] were students, who at other law schools, had achieved a [grade point average (GPA)] of 3.0 or above at the end of their first year at their previous law school. . . . Up until the Fall of 2019, the FIU transfer policy was worded as follows: ‘FIU Law only accepts transfer applicants on a conditional basis.  All transfer offers of admission are conditioned upon your maintaining at least a 3.0 cumulative GPA until you are enrolled at FIU Law.  If you do not satisfy the GPA condition, then FIU Law reserves the right to rescind your offer of admission.’  These students have very strong bar passage indicators based on the fact that law school GPA is more predictive of bar success than [Law School Admissions Test] scores or [undergraduate GPA].  This is the current policy, after 2019: ‘Students with outstanding academic records may be admitted based on first semester grades, conditioned upon your maintaining at least a 3.0 cumulative GPA through the second semester. If you do not satisfy the GPA condition, then FIU Law reserves the right to rescind your offer of admission.’” (internal citations ommitted)).

[14]See Jerry Organ, Organ: The 2021 Law School Transfer Market, TaxProf Blog (Dec. 22, 2021), https://taxprof.typepad.com/taxprof_blog/2021/12/organ-the-2021-law-school-transfer-market.html [https://perma.cc/XK6Q-LW9J].  Jerry Organ cautions about the use of transfer data on a national scale because of variations in the transfer pool nationally.  Id.  Our recent research also notes the following regarding any national modeling of the relationship between transfer and bar passage:

It is impossible to accurately assess the impact of transfers in and out, unless we have data on each individual student, to which school the student transferred, and what their Bar result was at that institution. . . . Some transfer students transfer for reasons unrelated to law school performance or ranking, to be closer to home, for example.

Bad Math, supra note 11, at 382.

[15]See Bad Math, supra note 11, at 380–82.  Using Touro Law as an example of this and describing the inequity of not accounting for this at this individual school:

1. In 2018, Touro lost 18 students to transfer at the end of the first year out of a matriculant pool of 177 students admitted the year before;
2. In 2019, Touro lost 24 students to transfer out of a matriculant pool of 209 students admitted the year before;
3. In 2020, Touro lost 24 students to transfer out of a matriculant pool of 197 students admitted the year before;
4. In 2021, Touro again lost 24 students to transfer out of a matriculant pool of 208 students admitted the year before.

Id. (internal citations omitted).

The transfer students were largely students who were denied admission to schools with higher LSAT and [undergraduate GPA] matriculating credentials but admitted to Touro and excelled way beyond their matriculating credentials suggested in their first year and were at or near the top of the class by the second semester of the first year.  These top students then transferred to other schools based on their first-year grades, and their Bar success will be attributed to the schools they transferred into.  Despite this reality which obviously reduces the bar passage rate at Touro as currently measured by the ABA, the ABA ignores this completely for the purposes of Standard 316.  How can this reality and its impact on bar passage be ignored in good conscience by the ABA and by the publishers of studies that ignore transfer numbers and unjustifiably rank Touro as one of the lowest Bar performing schools?  And, how can Bar performance be measured at the schools which receive large numbers of these top transfer students without accounting for this influx of large numbers of students likely to pass the Bar based on their 1L grades at the institutions they transferred from?

Access schools with diverse student populations like Touro end up being punished by the ABA for giving students with lower matriculating credentials an opportunity to enter the legal profession.  After one year of law school, when these students overperform their entering credentials, they transfer to the very same, less diverse schools that would have denied them admission.  When these students pass the Bar at these “higher performing schools,” it is these schools that get the credit in terms of Bar passage rates and no credit is given to a school like Touro.

Id. at 381–82 (internal citations omitted).

[16]Email from Jerry Organ, Bakken Professor of L., Univ. of St. Thomas Sch. of L., to author (May 23, 2023, 9:48 AM) (on file with author).

[17]See Paul Caron, Organ: Attrition Analysis for 2018, 2019, 2020 — With A Focus On Ethnicity, TaxProf Blog (Dec. 22, 2020), https://taxprof.typepad.com/taxprof_blog/2020/12/attrition-analysis-for-2018-2019-2020-with-a-focus-on-ethnicity.html#:~:text=Overall%20First%2DYear%20Attrition%20has%20Declined%20for%20Three%20Years&text=That%20downward%20trend%20in%20overall,the%202019%2D20%20academic%20year [https://perma.cc/XK6Q-LW9J] (explaining the variability in academic attrition among various law schools and demonstrating the impropriety of analyzing academic attrition via regression analysis on a national level).  See also Bad Math, supra note 11, at 251 n.99 (for a tabular demonstration revealing how academic attrition affects bar passage rates when attrition is examined on the individual school level even though the impact is not observable via a regression analysis).

[18]Bahadur & Ruth, supra note 4, at 6 (emphasis omitted).

[19]See Karen Sloan, Revamped US News Law School Rankings Yield Major Shifts, Reuters (May 11, 2023, 9:48 AM), https://www.reuters.com/legal/legalindustry/revamped-us-news-law-school-rankings-yield-major-shifts-2023-05-11/ (“Those changes primarily were the result of the publication's overhauled methodology that increased the weight of employment outcomes and bar passage rates and reduced the weight of Law School Admission Test scores and undergraduate grade-point averages.”).

[20]Except for loss of tuition revenue.

[21]Scott & Jackson, supra note 1, at 3 (quoting Revisions to Standard 316 Frequently Asked Questions, Am. Bar Ass’n, https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/council_reports_and_resolutions/may19/3-19-may-316-faq.pdf (last visited Aug. 30, 2023)).

[22]Bad Math, supra note 11, at 385; see also Nachman N. Gutowski, Stop the Count; The Historically Discriminatory Nature of the Bar Exam Requires Adjustments in How Bar Passage Rates Are Reported, If at All, 21 Seattle J. Soc. Just. 589, 609–11 (2023).

[23]Bad Math, supra note 11, at 385.

[24]Id. at 384–85 (“One harm caused by the ABA’s conduct is that it implies that Bar performance is a tangible and measurable metric when it is not.  But the greater harm is that the ABA’s conduct reaffirms the falsity that the Bar examination is valid because it subtly suggests that there are tangible transferable methodologies a school can and should employ to increase its performance on the examination.  And we (the ABA) have determined it is so reliable an indicator of institutional efficacy and lawyer competency that we condition accreditation on the metric.  This is essentially very skillful advocacy for the continued use of the bar examination.”).

[25]Debra Cassens Weiss, Lastest Version of ChatGPT Aces Bar Exam with Score Nearing 90^th Percentile, Am. Bar Ass’n J. (Mar. 16, 2023, 1:59 PM), https://www.abajournal.com/web/article/latest-version-of-chatgpt-aces-the-bar-exam-with-score-in-90th-percentile; cf. Eric Martinez, Re-Evaluating GPT-4's Bar Exam Performance (June 23, 2023), https://ssrn.com/abstract=4441311 (suggesting that ChatGPT-4’s performance on the bar examination needs to be re-evaluated because it was not as impressive as widely thought).

[26]Elyse Betters Picaro, How Much is ChatGPT Plus and What Does the ChatGPT Subscription Get You?, Pocket-lint (May 14, 2023), https://www.pocket-lint.com/chatgpt-plus-price-features-how-to-subscribe/.