Measurement characteristics of a concept classification exam using multiple case examples: A Rasch analysis

Nicholas B. Jennings, Marion K Slack, Lea E. Mollon, Terri L Warholak

Research output: Contribution to journalArticle

Abstract

Objective: To determine if an exam using multiple cases to test research design concepts measured only one cognitive skill, concept classification, and to determine if item difficulty varied according to the research design used for the case. Methods: The exam consisted of 50 multiple choice items associated with five example abstracts: a randomized controlled trial, pretest-posttest, crossover, retrospective cohort, and descriptive designs. A Rasch analysis was conducted to determine dimensionality (i.e., measured a single skill). Items were stratified by design to explore the relationship between item difficulty and study design. Overall difficulty was assessed using an item-person map. Results: The exam was administered to 101 students; the mean was 88.4% (mean score = 44.2; SD = 3.5). The Rasch analysis indicated the exam primarily measured one cognitive skill, presumably concept classification. The stratified analysis indicated that overall no single research design was more difficult than other designs; however, the type of research design and item topic interacted so that an easy item for one design could be difficult when associated with a different study design. Conclusions: The exam appeared to function more like a mastery exam documenting that most students performed well rather than as an exam for ranking students by ability. That item topic interacted with study design to affect item difficulty, indicates that items on the same topic are needed to test basic design concepts across study designs.

Original languageEnglish (US)
JournalCurrents in Pharmacy Teaching and Learning
DOIs
StateAccepted/In press - 2015

Fingerprint

Research Design
Students
Randomized Controlled Trials

Keywords

  • Concepts
  • Multiple choice exam
  • Rasch analysis
  • Research design

ASJC Scopus subject areas

  • Pharmacy
  • Pharmacology, Toxicology and Pharmaceutics(all)

Cite this

@article{e31fe38596904e989b8eb07ffc9f1ce2,
title = "Measurement characteristics of a concept classification exam using multiple case examples: A Rasch analysis",
abstract = "Objective: To determine if an exam using multiple cases to test research design concepts measured only one cognitive skill, concept classification, and to determine if item difficulty varied according to the research design used for the case. Methods: The exam consisted of 50 multiple choice items associated with five example abstracts: a randomized controlled trial, pretest-posttest, crossover, retrospective cohort, and descriptive designs. A Rasch analysis was conducted to determine dimensionality (i.e., measured a single skill). Items were stratified by design to explore the relationship between item difficulty and study design. Overall difficulty was assessed using an item-person map. Results: The exam was administered to 101 students; the mean was 88.4{\%} (mean score = 44.2; SD = 3.5). The Rasch analysis indicated the exam primarily measured one cognitive skill, presumably concept classification. The stratified analysis indicated that overall no single research design was more difficult than other designs; however, the type of research design and item topic interacted so that an easy item for one design could be difficult when associated with a different study design. Conclusions: The exam appeared to function more like a mastery exam documenting that most students performed well rather than as an exam for ranking students by ability. That item topic interacted with study design to affect item difficulty, indicates that items on the same topic are needed to test basic design concepts across study designs.",
keywords = "Concepts, Multiple choice exam, Rasch analysis, Research design",
author = "Jennings, {Nicholas B.} and Slack, {Marion K} and Mollon, {Lea E.} and Warholak, {Terri L}",
year = "2015",
doi = "10.1016/j.cptl.2015.09.010",
language = "English (US)",
journal = "Currents in Pharmacy Teaching and Learning",
issn = "1877-1297",
publisher = "Elsevier Inc.",

}

TY - JOUR

T1 - Measurement characteristics of a concept classification exam using multiple case examples

T2 - A Rasch analysis

AU - Jennings, Nicholas B.

AU - Slack, Marion K

AU - Mollon, Lea E.

AU - Warholak, Terri L

PY - 2015

Y1 - 2015

N2 - Objective: To determine if an exam using multiple cases to test research design concepts measured only one cognitive skill, concept classification, and to determine if item difficulty varied according to the research design used for the case. Methods: The exam consisted of 50 multiple choice items associated with five example abstracts: a randomized controlled trial, pretest-posttest, crossover, retrospective cohort, and descriptive designs. A Rasch analysis was conducted to determine dimensionality (i.e., measured a single skill). Items were stratified by design to explore the relationship between item difficulty and study design. Overall difficulty was assessed using an item-person map. Results: The exam was administered to 101 students; the mean was 88.4% (mean score = 44.2; SD = 3.5). The Rasch analysis indicated the exam primarily measured one cognitive skill, presumably concept classification. The stratified analysis indicated that overall no single research design was more difficult than other designs; however, the type of research design and item topic interacted so that an easy item for one design could be difficult when associated with a different study design. Conclusions: The exam appeared to function more like a mastery exam documenting that most students performed well rather than as an exam for ranking students by ability. That item topic interacted with study design to affect item difficulty, indicates that items on the same topic are needed to test basic design concepts across study designs.

AB - Objective: To determine if an exam using multiple cases to test research design concepts measured only one cognitive skill, concept classification, and to determine if item difficulty varied according to the research design used for the case. Methods: The exam consisted of 50 multiple choice items associated with five example abstracts: a randomized controlled trial, pretest-posttest, crossover, retrospective cohort, and descriptive designs. A Rasch analysis was conducted to determine dimensionality (i.e., measured a single skill). Items were stratified by design to explore the relationship between item difficulty and study design. Overall difficulty was assessed using an item-person map. Results: The exam was administered to 101 students; the mean was 88.4% (mean score = 44.2; SD = 3.5). The Rasch analysis indicated the exam primarily measured one cognitive skill, presumably concept classification. The stratified analysis indicated that overall no single research design was more difficult than other designs; however, the type of research design and item topic interacted so that an easy item for one design could be difficult when associated with a different study design. Conclusions: The exam appeared to function more like a mastery exam documenting that most students performed well rather than as an exam for ranking students by ability. That item topic interacted with study design to affect item difficulty, indicates that items on the same topic are needed to test basic design concepts across study designs.

KW - Concepts

KW - Multiple choice exam

KW - Rasch analysis

KW - Research design

UR - http://www.scopus.com/inward/record.url?scp=84949034862&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84949034862&partnerID=8YFLogxK

U2 - 10.1016/j.cptl.2015.09.010

DO - 10.1016/j.cptl.2015.09.010

M3 - Article

AN - SCOPUS:84949034862

JO - Currents in Pharmacy Teaching and Learning

JF - Currents in Pharmacy Teaching and Learning

SN - 1877-1297

ER -