(Post last updated June 16, 2022)
Review panel summary
The Assessment on Protein Structure & Enzyme Inhibition is an eight-item assessment instrument with a combination of true/false, multiple choice, and open-ended items. The instrument was designed to assess students’ understanding of biochemistry concepts. It has been evaluated with undergraduate students enrolled in introductory biochemistry courses in seven U.S. institutions, including four small, private four-year colleges, two private masters-granting universities, and one public masters-granting university [1]; additionally, one of the seven institutions is classified as a Hispanic-Serving Institution (HSI). Several aspects of validity and reliability have been assessed for the data generated by the instrument. In terms of validity, test content evidence was provided by 30 biochemistry faculty members (five of which were extensively involved in the item generation) in a biochemistry Process Oriented Guided Inquiry Learning (POGIL) community, all with a minimum of five years of teaching experience [1]. Additionally, topics/subject matter essential to be included in an undergraduate biochemistry degree plan were initially selected for instrument items based on recommendations and guidelines from the American Society for Biochemistry and Molecular Biology (ASBMB) [1]. Items were revised based on reviews by two independent raters upon initial collection of data [1]. All eight items on the instrument have also been assigned to levels of Bloom’s taxonomy by a panel of 18 biochemistry faculty members [1]. Following initial collection of data, revisions were made to items based on students’ responses to the open-ended items, as well as instructor feedback, and this was used to provide some evidence in support of response process validity [1]; however, as students were not interviewed in order to clarify details within their open responses, it is difficult to evaluate this evidence. In terms of reliability, a sample of 36 responses were evaluated with Cohen’s kappa to provide evidence for interrater reliability for open response items through two independent raters [1].
Recommendations for use
The Assessment on Protein Structure & Enzyme Inhibition was designed to assess students’ understanding of biochemistry concepts [1]. Items have been scored as correct or incorrect for the true/false and multiple choice; scoring for the open-response items uses a developed rubric [1], which assigns items a score of 0 (incorrect), 1 (partially correct), or 2 (correct). In many instances, the evidence provided in the literature has limited details provided; therefore, the continued generation of evidence of validity and reliability of the data generated via this instrument is recommended.
Details from panel review
The developers of the Assessment on Protein Structure & Enzyme Inhibition instrument gathered and reported some aspects of validity and reliability evidence for the instrument [1]; however, some of the evidence provided is limited since a lack of details have been reported. For example, for interrater reliability, there was limited detail provided on the development of the rubric used to evaluate students’ responses to the open-response items. In terms of internal structure validity, future studies might involve exploratory evaluations of how individual items are related, especially given the group scoring based on Bloom’s levels. In terms of relation to other variables, there is currently no documented evidence regarding how the instrument is related to variables such as performance, achievement, and/or progression in the course content area, which may provide further support for the data from this instrument.
References
[1] Villafañe, S.M., Heyen, B.J., Lewis, J.E., Loertscher, J., Minderhout, V., & Arnold Murray, T. (2016). Design and testing of an assessment instrument to measure understanding of protein structure and enzyme inhibition in a new context. Biochemistry and Molecular Biology Education, 44(2), 179-190. https://doi.org/10.1002/bmb.20931