University of Virginia School of Medicine
Principles of Medicine Committee
Minutes
09.13.06

 

The sole item on the agenda for this meeting was a workshop by Dr. Jerry Short, Associate Dean for Medical Education Support on assessment, in preparation for the up-coming visit of the LCME site visit team.

Objectives of the workshop:

1. To be able to talk about assessment comfortably with LCME site visitors.
2. To realize that all assessment methods have problems.

Schedule for the workshop:

1. Learn what you would like to know about assessment, testing, and grading.
2. Take 2 tests and discuss issues raised by them.
3. Define assessment and list its purposes.
4. Describe assessment terms:
   1. Summative versus formative assessment
   2. Criterion versus norm referenced assessment
   3. Reliability and validity

 

1. Assessment:  A sample of a student's performance obtained for many different reasons:
   
   Feedback
   Improvement
   Motivation
   Achievement
   Reward
   Punishment
   Prediction
   Documentation
   
   
2. Describe the difference between summative and formative assessment.
   

   Characteristic	Formative	Summative
   When is it used?	During instruction	At the end of instruction
   What does it tell students?	What to improve during instruction	How well student did at end of instruction
   What does it tell instructors?	What to re-teach, revise, during instruction	How well students did How well teacher taught

   
   
3. Describe the difference between criterion-referenced and norm-referenced assessment:
   

   Characteristics	Criterion-Referenced	Norm-Referenced
   What does the test tell you about the student?	What the student can do	How the student compares to other students in norm group.
   How are the scores reported?	Pass/Fail, Competent/Not yet competent, Percent correct	Percentile, standard score, percent correct in comparison to other students in norm group
   Describe a test of each type to assess swimming skills	Swim 50 yards in < 3 minutes.	A swim meet
   What is a weakness of each type of test?	Where does the criterion (pass/fail standard) come from?	What is the norm group? What can the student actually do?
   What are examples of widely used tests of each type?	USMLE result:  P/F for licensure CPX Course exam result as P/F	USMLE result: standard score for comparison to norm group. Course exam for comparison to class
   What other terms are associated with each type of test?	Competency, mastery	Traditional instruction Individual differences

   
   The growing talk of national testing and standards comes in the fifth year of the No Child Left Behind era. That federal law sought to hold public schools accountable for academic performance but left it up to states to design their own assessments. So the definition of proficiency -- what it means for a student to perform at grade level -- varies from coast to coast.
   
   Maryland recently reported that 82 percent of fourth-graders scored proficient or better in reading on the state's test. The latest data from the National Assessment of Educational Progress, known as "the nation's report card," show 32 percent of Maryland fourth-graders at or above proficiency in reading.
   
   Virginia announced last week that 86 percent of fourth-graders reached that level on its reading test, but the NAEP data show 37 percent at or above proficiency.
                                     - Jay Mathews, Washington Post, September 2, 2006
   
   
4. Describe the difference between reliability and validity as applied to assessment.

   	Reliability	Validity
   What do each tell us about an assessment?	How consistent are the results of the test	How well the test measures what it claims to measure
   How are they reported?	Test-retest reliability Scale 0.0 - 1.0	Correlation of test scores with a non-test gold standard
   What is the level you can expect to achieve?	Correlation of .80-.90	Correlation of .40-.50 when test and gold standard have good variability.
   What different types are reported?	Test-retest reliability. Split-half reliability	Predictive Content Construct

   
   
5. Describe the possible interaction between time on task and achievement.
   
   When time allowed is held constant, achievement will vary.
   
   When time is allowed to vary, achievement will be constant.

Robert Bloodgood