(Post last updated June 24, 2022)
Review panel summary
The Reaction Coordinate Diagram Inventory (RCDI) consists of 21 multiple-choice items with 2 of the items being two-tiered answer-reason items, all 21-items include a 0-100% confidence scale (not confident to confident) [1]. The instrument was designed to assess student’s thinking and confidence in interpreting reaction coordinate diagrams and their correspondence to reaction mechanisms. The instrument has been evaluated with second semester general, organic, and physical/biophysical chemistry students. Several aspects of validity and reliability have been assessed for the data generated by the RCDI. Evidence for test content validity was provided by experts (general and organic chemistry instructors) and students [1] and was based on a theoretical framework presented by the authors [2]. During pilot testing, undergraduate students provided feedback on clarity of the RCDI items and this was used to provide evidence for response process validity [1]. Relation with other variables was supported by comparing the different means scored by three groups of students from different classes. Higher mean scores for more advanced classes with significant differences were identified.
Reliability, item difficulty, and item discrimination of the data generated by the RCDI were examined by using Classical test theory (CTT). In terms of reliability, coefficient alpha has been used to estimate single administration reliability for the whole instrument [1, 2]. Even though the coefficient alpha value reported for the RCDI was in an acceptable range, the designers argued that coefficient alpha is not the most appropriate way to assess the reliability in the context of a concept inventory such as the RCDI. The authors justify this statement by mentioning that concept inventories are designed based on students’ misconceptions and fragmented ideas that are also used when students respond to items. Additionally, coefficient alpha implies an underlying unidimensional construct, which is not the case for concept inventories such as the RCDI. Fergusons’ delta was used to demonstrate that the students could achieve a wide distribution of scores across the range of possible scores on the RCDI. The item difficulty and item discrimination indices showed that two of the items (7 and 8) had values out of the acceptable range [1, 2]. The authors argued that these items should still be retained since they could suggest the prevalence of the misconceptions expressed in these items that are held by the students [1, 2]. Item difficulty and discrimination estimates decreased from general chemistry to organic in a longitudinal study which was an indication that the top-performing and low-performing students answered these items correctly in their later chemistry courses [2].
Recommendations for use
The RCDI was designed to assess students’ understanding and confidence with the kinetic and thermodynamic concepts encoded within RCDs [1, 2]. Literature to date provides support for many aspects of validity and reliability to support the use of the RCDI in assessing students’ conceptual understanding pertaining to RCDs. However, while aspects at the item level are well described, evidence to support the internal structure validity of the instrument for use with a total score [1, 2] would further strengthen the interpretation of RCDI data.
Details from panel review
The designers of the RCDI gathered and reported several aspects of validity and reliability evidence for the instrument [1, 2]. For example the test content was decided by the authors and then the designed items were checked for clarity and correctness with experts and students [1]. Undergraduate student interviews were used to support item modifications and interpretations, providing evidence in support of the response process validity [1]. The authors used Classical Test Theory to analyze the psychometric properties of individual items within the instrument. Evidence for item difficulty, item discrimination, and reliability of the scores were provided [2]. Two of the items (items 7 and 8) had indices out of the acceptable range. The designers used the Sankey diagrams to show changes in student’s responses to individual items over time [2]. The results of a longitudinal study showed no significant change in the RCDI scores or confidence from the beginning to the end of second-semester general chemistry but the instrument captured changes in students’ performance by the end of first-semester organic [2].
References
[1] Atkinson, M.B., Popova, M., Croisant, M., Reed, D.J., & Bretz, S.L. (2020). Development of the reaction coordinate diagram inventory: Measuring student thinking and confidence. Journal of Chemical Education, 97(7), 1841-1851. https://doi.org/10.1021/acs.jchemed.9b01186
[2] Atkinson, M.B, & Bretz, S.L. (2021). Measuring changes in undergraduate chemistry students’ reasoning with reaction coordinate diagrams: A longitudinal, multi-institution study. Journal of Chemical Education, 98(4), 1064-1076. https://doi.org/10.1021/acs.jchemed.0c01419