Minutes 09/13/06

Minutes 09/13/06

University of Virginia School of Medicine
Principles of Medicine Committee
Minutes
09.13.06

 

The sole item on the agenda for this meeting was a workshop by Dr. Jerry Short, Associate Dean for Medical Education Support on assessment, in preparation for the up-coming visit of the LCME site visit team.

Objectives of the workshop:

  1. To be able to talk about assessment comfortably with LCME site visitors.
  2. To realize that all assessment methods have problems.

Schedule for the workshop:

  1. Learn what you would like to know about assessment, testing, and grading.
  2. Take 2 tests and discuss issues raised by them.
  3. Define assessment and list its purposes.
  4. Describe assessment terms:
    1. Summative versus formative assessment
    2. Criterion versus norm referenced assessment
    3. Reliability and validity

 

  1. Assessment:  A sample of a student's performance obtained for many different reasons:

    Feedback
    Improvement
    Motivation
    Achievement
    Reward
    Punishment
    Prediction
    Documentation

  2. Describe the difference between summative and formative assessment.

    Characteristic

    Formative

    Summative

    When is it used?

    During instruction

    At the end of instruction

    What does it tell students?

    What to improve during instruction

    How well student did at end of instruction

    What does it tell instructors?

    What to re-teach, revise, during instruction

    How well students did
    How well teacher taught


  3. Describe the difference between criterion-referenced and norm-referenced assessment:

    Characteristics

    Criterion-Referenced

    Norm-Referenced

    What does the test tell you about the student?

    What the student can do

    How the student compares to other students in norm group.

    How are the scores reported? 

    Pass/Fail, Competent/Not yet competent, Percent correct

    Percentile, standard score, percent correct in comparison to other students in norm group

    Describe a test of each type to assess swimming skills

    Swim 50 yards in < 3 minutes.

    A swim meet

    What is a weakness of each type of test?

    Where does the criterion (pass/fail standard) come from?

    What is the norm group?

    What can the student actually do?

    What are examples of widely used tests of each type?

    USMLE result:  P/F for licensure

    CPX
    Course exam result as P/F

    USMLE result: standard score for comparison to norm group.

    Course exam for comparison to class

    What other terms are associated with each type of test?

    Competency, mastery

    Traditional instruction
    Individual differences


    The growing talk of national testing and standards comes in the fifth year of the No Child Left Behind era. That federal law sought to hold public schools accountable for academic performance but left it up to states to design their own assessments. So the definition of proficiency -- what it means for a student to perform at grade level -- varies from coast to coast.

    Maryland recently reported that 82 percent of fourth-graders scored proficient or better in reading on the state's test. The latest data from the National Assessment of Educational Progress, known as "the nation's report card," show 32 percent of Maryland fourth-graders at or above proficiency in reading.

    Virginia announced last week that 86 percent of fourth-graders reached that level on its reading test, but the NAEP data show 37 percent at or above proficiency.
                                      - Jay Mathews, Washington Post, September 2, 2006

  4. Describe the difference between reliability and validity as applied to assessment.

     

    Reliability

    Validity

    What do each tell us about an assessment?

    How consistent are the results of the test

    How well the test measures what it claims to measure

    How are they reported?

     

    Test-retest reliability
    Scale 0.0 - 1.0

    Correlation of test scores with a non-test gold standard

    What is the level you can expect to achieve?

    Correlation of .80-.90

    Correlation of .40-.50 when test and gold standard have good variability.

    What different types are reported?

    Test-retest reliability.
    Split-half reliability

    Predictive
    Content
    Construct


  5. Describe the possible interaction between time on task and achievement.

    When time allowed is held constant, achievement will vary.

    When time is allowed to vary, achievement will be constant.

Robert Bloodgood