Understanding the NCBE's Mean Scaled Scores
I’ve been addressing lots of questions over the past few weeks from graduates and soon-to-be graduates, but one of the more frequently asked questions I get as they begin preparing for the February 2023 bar exam is this: Are the February exams harder than the July exams?
The simple answer is “no”—at least according to the National Conference of Bar Examiners, the group that creates almost all the various components of the bar exam administered in nearly every jurisdiction.
According to the NCBE, MBE scaled scores are calculated based on a statistical process known as “equating”—a common technique used on standardized examinations.
What this means is that the raw scores on the current MBE exam (e.g., the number of correct answers out of the 175 multiple-choice questions that are scored) are statistically adjusted to account for differences in difficulty as compared with past examinations. This adjustment—called “equating”—makes it possible to compare scaled scores across test administrations because any particular scaled score will represent the same level of knowledge and performance from one test date to another.
The purpose of “equating”—again, according to the NCBE—is to help ensure that no examinee on one particular administration of the bar exam is unfairly penalized or rewarded for taking a more or less difficult form of the test. Because the adjustment of scores during equating is examination-specific (i.e., based on the level of difficulty of the current examination as compared to previous examinations), it isn’t possible to determine in advance of the test how many questions an examinee must answer correctly to achieve a specific scaled score.
The purpose of “equating”—again, according to the NCBE—is to help ensure that no examinee on one particular administration of the bar exam is unfairly penalized or rewarded for taking a more or less difficult form of the test.
The NCBE doesn’t release scaled-to-raw score conversions. But common wisdom among the bar preparation community, however, is that achieving the mean scaled MBE score across administrations is generally accomplished by correctly answering between 62% and 67% of the multiple-choice questions. That means, to be average, an examinee needs to answer around 109 to 114 of the 175 multiple-choice questions correctly.
And, generally speaking, being average on the July bar exam is usually good enough to pass the exam. However, what about being average on the February exam? Not so much.
The reason is that the equating and scaling of the MBE scores is only one-half of the equating and scaling story. How are the essays and performance tests on the Uniform Bar Exam scaled and equated?
Here’s where the calculations get a bit wonky, in my opinion.
UBE jurisdictions use the equipercentile method to scale essays to the MBE. This means that when the mean scaled MBE goes down, so does the mean scaled essay scores. And when this happens, the overall average combined score goes down, too, since the MBE and written scaled scores each make up one-half of the combined exam score.
The equipercentile method to scale essays to the MBE is a bit more complicated than what I’m describing below, but it should be close enough for purposes of this illustration.
For the July 2022 MBE, the national mean scaled score was about 140.3 (which was about 8 points higher than the February 2022 national mean MBE scaled score of 132). If an examinee were to score an average percent correct score on the July 2022 MBE, that score would equal around a scaled mean score of 140.3. But if an examinee were to score an average percent correct score on the February 2022 MBE, that examinee’s score would be 8 points lower at 132.
Since the scaled written score is equated to the scaled MBE scores, an average written score on the July 2022 exam would be scaled 140.3 as well, whereas an average written score on the February 2022 exam would be a scaled 132.
By scaling written raw scores to the scaled MBE scores, the distribution of written scores will be the same as the distribution of MBE scores—with a very similar average, a very similar minimum and maximum, and a very similar distribution of scores. The best essay score assumes the value of the best MBE score; the second best is set to the second best, etc. At the other end of the spectrum, though, the worst essay score assumes the value of the worst MBE score; the second worst is set to the second worst, etc. Ultimately, this leads to the average scaled written score equaling the average scaled MBE score.
Because both scaled MBE and scaled written scores are lower on the February 2022 exam, a hypothetical examinee on the February 2022 exam who separately scored average on the MBE and average on the written portion on the exam would have earned a combined score of 264 (132 + 132 = 264) and, subsequently, not pass the bar exam in a jurisdiction with a 270 passing score.
However, a hypothetical examinee on the July 2022 exam who separately scored average on the MBE and average on the written portion on the exam would have a combined score of 280.6 (140.3 + 140.3 = 280.6) and would comfortably pass the exam in a 270 jurisdiction.
That’s one of the reasons why the pass rates for the February examinations are typically lower than the July administrations. The national mean scaled scores on the MBE are usually lower in February, and because the essays and performance tests are scaled to the MBE scores, the dip in MBE scores actually has a double impact on an examinee’s combined score.
As far as the national mean scaled scores for July and February administrations go, here are the results for the past five years:
All this is to say that the February exams aren't necessarily "harder" than the July exams. At least based on the NCBE's data, the lower pass rates are the product of the examinee performance.
