University of Virginia School of
Systems Leadership Community
March 13, 2013
Medical Education Building, Simulation Center Conference Room, 4:00 p.m.
Present (underlined) were: Behm (GI), Bray (Endo-Repro), Bryant (CPD & MSI), Chen (Ethics/Prof), Clark (HemOnc), Cohen (MBB), Dalkin (Endo-Repro), Dent (CV), Douvas (HemOnc), Durieux (C2S), Gay (Pul), D. Innes (HemOnc), Kalantari (Ren), Knight, LeGallo, Lorenz (M&I), Lowson (Pul), McCollum (MSI), McDaniel (CPD), Marks (CPD), Moyer (Pul), Nadkarni (SIM), Nathan (co-chair) (MBB), Nunez, Noramly (co-chair), Pearson (M&I), Rosner (Ren), Russell (MSI), Shah (GI), Tiourirne (MBB), Waggoner-Fountain, Worden (MBB); also Alexander, Bradley, Caelleigh, Canterbury, Casola, Graham, A. Innes, Jackson, Knight, Lieb, Martindale, Rogers, Roy, Tomlin, White, Yoon.
Convener: Bart Nathan
Assessment Oversight Committee. The committee has developed a philosophy of assessment for the NxGen curriculum: the goal is developmental assessment based on milestones. Such a process requires assessment questions that move students to higher order and integrated questions. The committee analyzed test items for all formative and summative assessments held on weekend during the 2010-2011 iteration of the Preclerkship Curriculum. It was difficult to recognize/categorize the “integrated” questions. The proportions of recall, higher order/clinical, and integrated questions varied considerably across courses/systems from 70/30/0 at one end of the spectrum to 30/55/15 at the other end. Jim Martindale plans to do this analysis for exam items in each system so that the System Leaders can see the percentages of different levels of questions.
a. Recommendations. The committee has recommendations for the assessment process.
i. Training in writing test items. Everyone responsible for writing test items must learn how to write good items. Chris Peterson has agreed to do a PRL on writing test items and then conduct follow-up workshops.
Robust review process. A robust process for reviewing and improving test items must be created and maintained. The goal should be that all exams are ready 6 months before use.
Number of test items needed. Ten items are needed for each hour of classroom time: 5 practice questions and 5 exam questions.
CPD assessments. The present assessment has no predictive validity with OSCE performance. A better, more objective assessment is needed.
Confusing terminology. The term “formative” is used for pre-class quizzes, for practice quizzes before exams, and for the weekend formative exams. The committee has focused on the weekend formative exams.
Formative and summative questions. The committee discussed whether formative questions should or should not be the same format and level as summative questions. Students are clear that they want the weekend formatives to be just like the summatives so that they can be a true preparation.
Integrated questions. Students need to learn how to answer integrated questions. They get better over several systems, but their first exposure is difficult.
Images and/or audio in exam questions. All questions with images are by definition higher order questions. Where feasible, still images, video, and audio should be used in examination items rather than using written descriptions. The USMLE is beginning to use audio and video (especially video) questions. The current exam technology already allows use of images video in exams. Images appropriate for the classroom are difficult to find and time-consuming to make.
Media database. There are complications about images put into the media database, primarily because their copyright status cannot be monitored well.
Information Mastery as a curriculum thread. Karen Knight reported on the information mastery thread, giving examples of how it is now covered. She discussed competencies for information mastery and where and how instruction is now covered in the curriculum.
a. Limitations in present curriculum: Instruction is offered only a few times in the preclerkship curriculum, students have little time to practice focused skills, and there is no regular provision for faculty observation, practice, and remediation. The area of the curriculum where students can develop higher-level information-management skills is the summer research programs, but only about half of the students participate in summer research.
b. Example of an activity in the preclerkship curriculum. A good example of integrating information-management skills in the clinical context is Bill Wilson’s “Genetics Directed Clinical Letter.”
Next meeting. The next meeting will be Wednesday, April 10, 2013.
* * * * *