David Chen was doing his emergency medicine rotation in February when he heard the news: The notorious US Medical Licensing Examination (USMLE) Step 1 test will move from a three-digit score to pass/fail as early as 2022.
Chen, now an internal medicine resident in California who was then a student at the University of Washington School of Medicine, had long advocated for the change. In March 2019, he and his colleagues wrote a piece that was published in Academic Medicine, arguing that an emphasis on Step 1 scores damaged educational quality, harmed student well-being, and exacerbated disparities among future physicians.
After the perspective piece was published, Chen received both supportive and angry messages. He then watched the debate grow increasingly contentious, as many argued that Step 1 scores are an essential metric for residency evaluation. Still, Chen welcomed the USMLE’s announcement and believes that the switch to pass/fail will ultimately prove a positive one. “It opens up space for innovation and actually training people to be physicians rather than rather than test-takers,” he said.
COVID-19 continues to add uncertainty to the composition and timing of licensing exams. With no clear roadmap, students and institutions are scrambling. Piling onto those concerns, residency directors must now prepare to sort through a sea of applicants without the help of Step 1 scores.
“I worry for residency program directors making these choices,” said Andrew Elder, FRCPE, an internal medicine physician at the University of Edinburgh Medical School in the United Kingdom and a visiting professor at Stanford University Medical School. “It’s a supply-and-demand problem. There are too many doctors wanting the job, and how in heaven’s name do they narrow down the field?”
At their core, questions surrounding the Step 1 change are much bigger than test scores. Although some critics are fearful, supporters see the move as the first step toward a more equitable future. Getting to that future, however, will require creative thinking, says Bryan Carmody, MD, associate program director for the pediatric residency program at the Children’s Hospital of the King’s Daughters in Norfolk, Virginia. “Step 1 falling is the big domino,” Carmody said. “Having that domino topple makes an opportunity for change. It doesn’t by itself make life better, but it paves the way.”
As discussions about the future ensue, arguments about Step 1 have morphed into a debate about the entire future of medical education.
The Step 1 exam was originally designed as a stepping stone toward board certification, says Kenneth Olivier, MD, program director for the Department of Radiation Oncology at the Mayo Clinic, in Rochester, Minnesota. But the test’s scores soon became intensely important as residency programs began using them as a quick way to assess large pools of candidates.
Each year, the number of applicants far exceeds the number of residency slots available in the United States. Some programs receive hundreds of candidates for just a handful of openings, and directors have only weeks to make decisions. Over the past few decades, residency programs have come to depend on Step 1 scores to rapidly winnow down growing applicant lists. That increasing dependence has fed a system-wide feedback loop.
A test-preparation industry has emerged to help people maximize scores. Medical schools have set aside weeks during which there is no clinical training so students can study without being interrupted. Those changes have led to a rise in median scores, which has made it even harder — and more important — to do well. The result has been widespread misery and anxiety among medical students, all for the sake of a test that critics say measures esoteric biological minutiae instead of testing the clinical skills that matter most in a patient setting.
That kind of unwarranted misery is one reason why detractors like Carmody and Chen suggest Step 1’s switch to pass/fail is long overdue. Yet where they see opportunity, residency program directors foresee problems. It would take 40 hours a week for 3 months straight to review 1000 residency applications, even if they only spend 30 minutes on each one, according to Lisa Willett, MD, director of the Tinsley Harrison Internal Medicine Residency Program at the University of Alabama at Birmingham School of Medicine, who addressed the issue in a perspective article for The New England Journal of Medicine.
Willett argues that without Step 1 as a data point for filtering applications, directors are likely to rely more heavily on results from the USMLE Step 2 Clinical Knowledge (CK) exam, which is still scored. Doing so would simply push concerns forward to a test taken during the busy third year of medical school.
Furthermore, moving Step 1 to pass/fail could lead to a system that increasingly favors students who already have a leg up. “Students from larger, established or elite medical schools and students with personal connections may have advantages over students from newer, smaller, international, or osteopathic schools and those who are less well connected,” Willett wrote, noting the potential for added burdens, not fewer. “Program directors, seeking new sources of data, may require applicants to complete visiting rotations, secondary applications, or specialty-specific standardized tests, which will create new stressors, new expenses, and wider gaps for disadvantaged students.”
Decreased emphasis on test scores could also lead to increased emphasis on an applicant’s interpersonal strengths and weaknesses, as gauged by personal references and interviews. That, too, could exacerbate inequities, says Edinburgh’s Elder. Some students may not be able to afford visits to multiple schools in multiple states in order to make a good impression, which could take them out of the running for positions. This could add to existing implicit biases, a well-known problem during the interview process.
“With the scoring system or the pass/fail system, each is potentially unfair in different ways,” Elder said. “I do not think there’s any magical way to do this, and, whatever you do, you might disadvantage some people.”
Bigger Changes Ahead?
To overcome these obstacles, some experts are thinking creatively about how the pass/fail change could usher in bigger, system-wide reforms that would involve individuals at every level of medical education: students, medical schools, residency programs, and test administrators.
One idea for reducing the load on residency programs would be to limit the number of programs to which an individual student could apply. With enough pressure from residency programs, Carmody says, the Association of American Medical Colleges could conceivably use their Electronic Residency Application Service (ERAS) to institute a cap on applications or tax excess applications, although he concedes that the idea may meet with legal challenges. It would also likely meet resistance from various groups, such as couples looking to match into residency together, international medical graduates (IMGs), osteopathic doctors, and others who apply to many different programs.
Alternatively, Carmody says limits could be set on the number of interviews each applicant can accept. That restriction would be technically challenging, insofar as some programs use systems other than ERAS to schedule interviews. Still, these are the kinds of solutions that could reduce the need for metrics such as Step 1 scores to weed people out, says Carmody, who was writing about residency application reform well before the USMLE’s announcement.
Carmody has discussed other ideas to reduce pressure on residency directors, such as allowing for college-like early-decision applications, requiring candidates to rank choices in order of preference when they apply, and letting directors see where else candidates have applied. One less logistically dense solution would be for programs to require school-specific essays or other time-consuming tasks, he says. This could induce students to apply to fewer places.
Finding ways to reduce the number of applicants per program would bring the United States more in line with the United Kingdom, where programs operate on a smaller scale, Elder says. The UK system offers insight into alternate ways of evaluating applicants. For example, when students finish medical school in the United Kingdom, they take a situational judgment test, which assesses skills through work-based scenarios. Research suggests that these assessments are good predictors of performance in the UK’s Foundational Program, a 2-year training program for new doctors.
Step 1 scores, on the other hand, primarily predict performance on other tests. “I don’t personally believe that they are strong predictors of who are going to be the best doctors,” Elder says. “There are many, many other qualities a doctor needs other than knowledge to be a high-performing clinician.”
To help identify applicants with those qualities, Carmody says, residency programs may also demand that medical schools start providing more meaningful and honest evaluations of their students, prioritizing interpersonal details that go beyond an ability to memorize facts. “We have to move away from the one-size-fits-all kind of thinking,” he said.
As pass/fail scoring for Step 1 becomes reality, it is particularly important for students to speak up about how they think residency programs should evaluate them, Carmody says. That includes IMGs and students from non-elite institutions, who often see a good Step 1 score as their ticket to high-profile residencies in the United States. After all, the switch to pass/fail suggests that advocacy works. “Change is possible,” Carmody wrote in blog post. “But you have to speak up.”
Changing Step 1 scoring was never the only goal, says David Johnson, chief assessment officer at the Federation of State Medical Boards, which, along with the National Board of Medical Examiners, administers the USMLE. Instead, the hope is that the move will usher in systemic changes to medical education that will better emphasize the skills that matter most.
“If a school is literally having to take 6, 8, 10 weeks of dedicated downtime so a student can maximize a score, what other subject matter is being put by the wayside and not being given the kind of weight that it deserves?” he asked. The change to pass/fail “creates an opportunity for a more appropriate, more holistic experience on the part of medical students.”
Putting Pressure on Administrators
Some residency programs have already stopped weeding applicants by test scores, offering a window into the possible future of a scoreless Step 1. Two years ago, Olivier at the Mayo Clinic started ignoring Step 1 results when assessing residency candidates. He had become frustrated by what he saw as a lack of correlation between performance on the test and clinical skill. “You don’t sit in front of your doctor and say, ‘Do I have (A) diabetes, (B) hypertension, or (C) something else?” he said. “It’s not how patients interact with doctors.”
Now, he evaluates candidates on the basis of other factors first, including third-year grades and narratives written by medical school directors. In those narratives, he looks for key words like “team player” or “great with patients.”
A change in Step 1 scoring, which has been an initial standard for consideration, will also place an increased emphasis on thorough vetting during later parts of the application process, Olivier’s experience suggests. He asks unexpected questions during interviews and observes responses carefully. He also relies on reports from residents in his program, who spend time with applicants when they visit. The goal is to ascertain how well future physicians will work with patients and other physicians.
The official move to drop Step 1 scores, Olivier says, should also put more pressure on medical school deans to provide real information about their students and will push residency directors to find new ways of assessing the fit and competency of candidates. Reduction in unnecessary stress and anxiety for medical students is an added benefit, he says. Overall, by no longer relying on numbers, more emphasis will be placed on clinical assessments of future doctors, Olivier believes.
“Step 1 going pass/fail is beneficial to all patients in the United States,” he said. Instead of spending months learning answers to exam questions, he hopes medical education will shift toward training students to actually care for people.
Emily Sohn is a freelance journalist in Minneapolis. Her stories have appeared in the Washington Post, NPR, Nature, National Geographic, and many other publications. She can be reached on Twitter @tidepoolsinc.