(Post last updated June 1, 2021)
Review panel summary
The Mulford & Robinson Chemical Concepts Inventory (CCI) is a multiple-choice instrument designed to assess the alternate conceptions of students in high school or first-semester college chemistry [1]. The inventory contains 22 multiple-choice items, half of which are two-tiered. The CCI has been evaluated with students enrolled in general chemistry (including engineering students [4]) at a variety of institutions in the U.S. [1–6]. The reliability of the data collected with this instrument was established using test-retest reliability and single administration reliability [1, 2]. The validity of the data collected with this instrument has been investigated through expert evaluation, response-process interviews, and relations to other variables [1, 3-6]. Conflicting evidence was presented about the quality of the CCI items reported in various studies as evaluated through response-process interviews [1, 3, 6]. In addition, one study reported certain topics were underrepresented as 13 out of 22 items assess two general chemistry topics (conservation of mass-matter and phase change), and the remaining 9 items assess unique topics, which presents additional threats to validity [1].
Recommendations for use
In its current form, it is unclear whether the CCI is measuring alternate conceptions of students in high school or first-semester college chemistry. Despite this concern, no modifications and improvements have been made to the CCI items. If using the instrument as developed, users are cautioned to conduct their own examination for the evidence of validity and reliability for the data generated. Future work may explore the utility of changing the CCI items that showed threats to the response process validity [3] as well as adding different items to improve topic coverage and examining how these items function within the item sets.
Details from panel review
An extensive literature search was carried out by the developers to identify misconceptions associated with the CCI topics to inform writing of the items [1]. Expert evaluation of the instrument, as well as internal structure validity were explored with four expert chemical education researchers who believed that all the questions were at an appropriate level for students taking a general chemistry course [1]. Additional validity evidence was obtained with 30 chemistry instructors and 8 chemistry education graduate students who completed the content analysis of the CCI items [3].
Response-process interviews were conducted with 8 students and showed that the students interpreted the questions on the CCI as intended [1]. However, response-process interviews with 25 students in a different study on a subset of the CCI items identified issues with some of the items (e.g., some items may be measuring recall instead of conceptual understanding) [3]. Similar findings were made by Karch et al. who also questioned what is being measured by some of the CCI items [6]. No work has been done to revise and reevaluate the problematic items. This raises concerns that in its current form the CCI is not measuring what it is intended to measure. The threat to validity is further exacerbated by the issue of topic underrepresentation. Specifically, 13 out of 22 items assess two general chemistry topics (conservation of mass-matter and phase change), whereas the remaining 9 items assess unique topics [2]. Although some of the items on the CCI do have some evidence to support the validity of students’ responses, there is not enough consistency among items covering an individual topic to make inferences about students’ alternate conceptions across general chemistry content. The instrument developers sought to “measure the extent of entering student’s alternate conceptions about topics found in the first semester of many traditional general chemistry courses” (p. 739, ref 1). There is a need for multiple items to measure students’ alternate conceptions about an individual concept or topic area.
Relations to other variables were established across several studies [4-6]. Cracolice and Busby established correlations between CCI scores and scientific reasoning ability, intelligence, attitudes toward chemistry, as well as ACS exam scores [4]. Wheeler et al. established correlations between higher CCI scores and programming skills, solving a chemistry problem, and solving a chemistry problem in Mathematica [5]. Finally, Karch et al. used eye-tracking to characterize the mental tasks that students used when completing some of the CCI items and how those mental tasks elicited changing levels of cognitive load [6].
Two sources of reliability were assessed for the CCI. Test-retest reliability was assessed with a subset of students who were given a second posttest two weeks after they were given the first posttest. Results showed a good test-retest reproducibility (Pearson correlation = 0.79, posttest). Single administration reliability was evaluated across two studies, with each reporting coefficient alpha values between 0.70 to 0.79 [1, 2]. Rasch person reliability was also reported [2].
Three studies reported item difficulty and item discrimination values [1, 2, 7]. Barbera identified that 8 items on the CCI have difficulty and discrimination values outside of the recommended values for a concept inventory [2]. Similar result was reported by Reimer et al., who stated that nearly half of the CCI questions are not in the desired item difficulty and item discrimination window [7]. The Rasch model showed poor fit for only one item in both the pretest and posttest data; these results indicate that the data from this item do not fit the model of an item being easier for higher ability students. This finding does not indicate whether the problem is with the item itself or with the actual knowledge of the students [2].
References
[1] Mulford, D. R., & Robinson, W. R. (2002). An inventory for alternate conceptions among first-semester general chemistry students. Journal of Chemical Education, 79(6), 739–744. https://doi.org/10.1021/ed079p739
[2] Barbera, J. (2013). A psychometric analysis of the Chemical Concepts Inventory. Journal of Chemical Education, 90(5), 546–553. https://doi.org/10.1021/ed3004353
[3] Schwartz, P., & Barbera, J. (2014) Evaluating the content and response process validity of data from the Chemical Concepts Inventory. Journal of Chemical Education, 91(5), 630–640. https://doi.org/10.1021/ed400716p
[4] Cracolice, M. S., & Busby, B. D. (2015). Preparation for college general chemistry: more than just a matter of content knowledge acquisition. Journal of Chemical Education, 92(11), 1790–1797. https://doi.org/10.1021/acs.jchemed.5b00146
[5] Wheeler, L. B., Chiu, J. L., & Grisham, C. M. (2016). Computational methods in general chemistry: perceptions of programming, prior experience, and student outcomes. Journal of College Science Teaching, 2016, 45(3), 83–91.
[6] Karch, J. M., Valles, J. C. G., & Sevian, H. (2019). Looking into the black box: using gaze and pupillometric data to probe how cognitive load changes with mental tasks. Journal of Chemical Education, 96(5), 830–840. https://doi.org/10.1021/acs.jchemed.9b00014
[7] Reimer, L. C., Leslie, J. M., Bidwell, S. L., Isborn, C. M., Lair, D., Menke, E., Stokes, B. J., & Hratchian, H. P. (2019). Aiming toward an effective Hispanic-serving chemistry curriculum. In Growing Diverse STEM Communities: Methodology, Impact, and Evidence. ACS Symposium Series; American Chemical Society: Washington, DC.