Measuring student learning with item response theory
Young-Jin Lee, David J. Palazzo, Rasil Warnakulasooriya, and David E. Pritchard
We investigate short-term learning from hints and feedback in a Web-based physics tutoring system. Both the skill of students and the difficulty and discrimination of items were determined by applying item response theory (IRT) to the first answers of students who are working on for-credit homework items in an introductory Newtonian physics course. We show that after tutoring a shifted logistic item response function with lower discrimination fits the students’ second responses to an item previously answered incorrectly. Student skill decreased by 1.0 standard deviation when students used no tutoring between their (incorrect) first and second attempts, which we attribute to “item-wrong bias.” On average, using hints or feedback increased students’ skill by 0.8 standard deviation. A skill increase of 1.9 standard deviation was observed when hints were requested after viewing, but prior to attempting to answer, a particular item. The skill changes measured in this way will enable the use of IRT to assess students based on their second attempt in a tutoring environment.
©2008 The American Physical Society
URL: http://link.aps.org/abstract/PRSTPER/v4/e010102
DOI: 10.1103/PhysRevSTPER.4.010102
Showing posts with label IRT. Show all posts
Showing posts with label IRT. Show all posts
2007-10-30
Bao - arxiv.org 2007
Dynamic Models of Learning and Education Measurement
arxiv.org posting
Lei Bao
(Submitted on 6 Oct 2007)
Pre-post testing is a commonly used method in physics education for evaluating students' achievement and or the effectiveness of teaching through a short period of instruction. A popular method to analyze pre-post testing results is the normalized gain first brought to the physics education community in wide use by R. Hake. In his analysis with thousands of students' pre-post test results, it has been observed that students having very different pretest scores tend to have similar normalized gains when going through similar types of instruction, i.e., classes with traditional instruction often have systematically lower gains than classes with research-based collaborative types of instruction. This feature allows researchers to investigate the effectiveness of instruction using data collected from classes with different initial states. However, the question of why the normalized gain has this feature and to what extend this feature will be valid is not well understood. Recently, there have been debates on what the normalized gain is actually measuring and concerns that the normalized gain lacks a probability framework comparing to other methods such as Item Response Theory (IRT). Motivated by searching for answers to these questions, a theoretical model about the dynamic process of learning have been developed, which leads to an explanatory interpretation of the features of the normalized gain. Further the model also connects well to other models and methods such as IRT and shows that the normalized gain does have a probabilistic framework but one different from what the IRT emphasizes. This paper will report the basic theoretical formalism of the new model and explore its applications in data modeling and analysis.
Comments: Theoretical Models of Education Measurement
Subjects: Physics Education (physics.ed-ph); Data Analysis, Statistics and Probability (physics.data-an)
Cite as: arXiv:0710.1375v1 [physics.ed-ph]
arxiv.org posting
Lei Bao
(Submitted on 6 Oct 2007)
Pre-post testing is a commonly used method in physics education for evaluating students' achievement and or the effectiveness of teaching through a short period of instruction. A popular method to analyze pre-post testing results is the normalized gain first brought to the physics education community in wide use by R. Hake. In his analysis with thousands of students' pre-post test results, it has been observed that students having very different pretest scores tend to have similar normalized gains when going through similar types of instruction, i.e., classes with traditional instruction often have systematically lower gains than classes with research-based collaborative types of instruction. This feature allows researchers to investigate the effectiveness of instruction using data collected from classes with different initial states. However, the question of why the normalized gain has this feature and to what extend this feature will be valid is not well understood. Recently, there have been debates on what the normalized gain is actually measuring and concerns that the normalized gain lacks a probability framework comparing to other methods such as Item Response Theory (IRT). Motivated by searching for answers to these questions, a theoretical model about the dynamic process of learning have been developed, which leads to an explanatory interpretation of the features of the normalized gain. Further the model also connects well to other models and methods such as IRT and shows that the normalized gain does have a probabilistic framework but one different from what the IRT emphasizes. This paper will report the basic theoretical formalism of the new model and explore its applications in data modeling and analysis.
Comments: Theoretical Models of Education Measurement
Subjects: Physics Education (physics.ed-ph); Data Analysis, Statistics and Probability (physics.data-an)
Cite as: arXiv:0710.1375v1 [physics.ed-ph]
Subscribe to:
Posts (Atom)