University of Virginia School of Medicine
Principles of Medicine Committee
The sole item on the agenda for this meeting was a workshop by Dr. Jerry Short, Associate Dean for Medical Education Support on assessment, in preparation for the up-coming visit of the LCME site visit team.
Objectives of the workshop:
- To be able to talk about assessment comfortably with LCME site visitors.
- To realize that all assessment methods have problems.
Schedule for the workshop:
- Learn what you would like to know about assessment, testing, and grading.
- Take 2 tests and discuss issues raised by them.
- Define assessment and list its purposes.
- Describe assessment terms:
- Summative versus formative assessment
- Criterion versus norm referenced assessment
- Reliability and validity
- Assessment: A sample of a student's performance obtained for many different reasons:
Describe the difference between summative and formative assessment.
When is it used?
At the end of instruction
What does it tell students?
What to improve during instruction
How well student did at end of instruction
What does it tell instructors?
What to re-teach, revise, during instruction
How well students did
How well teacher taught
- Describe the difference between criterion-referenced and norm-referenced assessment:
What does the test tell you about the student?
What the student can do
How the student compares to other students in norm group.
How are the scores reported?
Pass/Fail, Competent/Not yet competent, Percent correct
Percentile, standard score, percent correct in comparison to other students in norm group
Describe a test of each type to assess swimming skills
Swim 50 yards in < 3 minutes.
A swim meet
What is a weakness of each type of test?
Where does the criterion (pass/fail standard) come from?
What is the norm group?
What can the student actually do?
What are examples of widely used tests of each type?
USMLE result: P/F for licensure
Course exam result as P/F
USMLE result: standard score for comparison to norm group.
Course exam for comparison to class
What other terms are associated with each type of test?
The growing talk of national testing and standards comes in the fifth year of the No Child Left Behind era. That federal law sought to hold public schools accountable for academic performance but left it up to states to design their own assessments. So the definition of proficiency -- what it means for a student to perform at grade level -- varies from coast to coast.
Maryland recently reported that 82 percent of fourth-graders scored proficient or better in reading on the state's test. The latest data from the National Assessment of Educational Progress, known as "the nation's report card," show 32 percent of Maryland fourth-graders at or above proficiency in reading.
Virginia announced last week that 86 percent of fourth-graders reached that level on its reading test, but the NAEP data show 37 percent at or above proficiency.
- Jay Mathews, Washington Post, September 2, 2006
- Describe the difference between reliability and validity as applied to assessment.
What do each tell us about an assessment?
How consistent are the results of the test
How well the test measures what it claims to measure
How are they reported?
Scale 0.0 - 1.0
Correlation of test scores with a non-test gold standard
What is the level you can expect to achieve?
Correlation of .80-.90
Correlation of .40-.50 when test and gold standard have good variability.
What different types are reported?
- Describe the possible interaction between time on task and achievement.
When time allowed is held constant, achievement will vary.
When time is allowed to vary, achievement will be constant.