(Post last updated June 24, 2022)
Review panel summary
The Thermochemistry Concept Inventory (TCI) is a 10-item concept inventory designed to assess students’ understanding of thermochemistry concepts typically introduced in general (introductory) chemistry courses. Various pieces of reliability and validity evidence have been collected for this assessment. Test content validity evidence was provided by surveying chemistry faculty about important topics in thermochemistry. Further, interviews with faculty focused on topic coverage, item feedback, and consensus of correct answers [1]. Student interviews were also conducted to elucidate common alternate conceptions which were then used to generate item distractors [1]. Response process validity (RPV) evidence was provided through semi-structured interviews with students that focused on item wording, option plausibility, and student conceptions matching their selected response option. Students in an Honors general chemistry course outperformed the students in the other general chemistry courses, which can be considered evidence for relation to other variables. Sufficient evidence for the internal structure validity and single administration reliability were presented using Rasch analysis and suggest that most items can appropriately discriminate students within this range of abilities [2].
Recommendations for use
The TCI was developed to assess students’ understanding of thermochemistry in general (introductory) chemistry. Evidence supporting the test content and response process validity is published [1]. Internal structure validity supports the use of a total TCI score with the exception of one item (Item K) which did not have acceptable outfit statistics in the Rasch model analysis. Therefore this item should not be included in the assessment when evaluating the total score in a summative assessment [2]. The TCI items functioned equally well (aside from Item K) when administered as a formative or summative assessment, and in lab or lecture settings [2].
Details from panel review
Internal structure evidence was provided by evaluating the unidimensionality and local independence through principal component analysis. Fit statistics of the Rasch model also indicated an acceptable fit for most items [2]. For single administration reliability evidence, Cronbach’s alpha was below the acceptable value. However, the authors note that alpha may not be the most appropriate value to evaluate the TCI and provide a discussion of reliability at the item-level. The TCI authors discuss the acceptable fit statistics of the Rasch model (except for Item K) as a proxy for evidence of reliability and the option probability curves as a way to evaluate the reliability of responses for the range of student abilities [2]. The authors also provided further response process validity evidence through option probability curves from a Rasch analysis which revealed items’ ability to discriminate between a range of performers.
References
[1] Wren, D., & Barbera, J. (2013). Gathering evidence for validity during the design, development, and qualitative evaluation of Thermochemistry Concept Inventory items. Journal of Chemical Education, 90(12), 1590-1601. https://doi.org/10.1021/ed400384g
[2] Wren, D., & Barbera, J. (2014). Psychometric analysis of the thermochemistry concept inventory. Chemistry Education Research and Practice, 15, 380-390. https://doi.org/10.1039/C3RP00170A