Chemical Representations Inventory

(Post last updated July 23, 2021)

Review panel summary

The Chemical Representations Instrument (CRI) is a 19-item, open-ended instrument designed to measure student-teachers’ knowledge of chemical representations. Literature, textbooks, curricula and exams were reviewed to extract topics to be included in the assessment. The CRI has been evaluated with 322 student-teachers from 12 German universities who were in different stages of their university education with an average age of 23 years and 62% females [1]. Several aspects of validity and reliability have been investigated for the data generated by the CRI. Three chemistry professors reviewed the items, which provided expert judgment evidence to support the test content validity evidence. Think-aloud interviews have been used to provide response process validity evidence [1], which helped to improve the comprehensibility of the items by excluding or changing misleading wordings. Results from Rasch analysis suggested that the items measure a unidimensional construct, which supports the internal structure validity evidence [1, 2]. The significant correlations between the person parameters of the Rasch scaling and the comparisons by sex, the amount and level of chemistry instruction at upper secondary course, year, and courses at university provided some validity support in regard to relations with other variables [1]. Finally, coefficient alpha and values from the Rasch model were used to estimate the single administration reliability of the item groupings [1, 2], which supports the consistency of each item within a grouping. The item difficulty was explored using classical test theory and Rasch analysis [1]. Item discrimination was also estimated [1, 2]. The authors stated that most items had acceptable difficulty and discrimination.

Recommendations for use

The CRI was developed to measure student-teacher’s knowledge of chemical representations. Various validity and reliability evidence were collected, which supported the functionality of the items. The articles presenting the CRI were written in English and the instrument was provided in English. However, studies were conducted in Germany. It was not clear whether the original instrument was in German or English. Evidence presented suggests that the instrument should be scored using partial credit scoring rather than right/wrong scoring to better distinguish between students’ knowledge [1].

Details from panel review

The CRI developers collected a variety of validity and reliability evidence which provided supportive psychometric evidence for the CRI scores [1]. In addition, the developers reviewed literature, textbooks, curricula, and exams to extract topics to be included in CRI; this is a good practice because it shows the items were developed based on practice and topics already identified in the literature.

Three chemistry professors reviewed the items with regard to their comprehensibility, difficulty, relevance for university studies, appropriateness of items, phrasing, and correct solutions, which provided test content validity evidence [1, 2]. The developers also provided supportive evidence of response process validity through think-aloud interviews with students, including changing or excluding misleading wording to improve the comprehensibility of the items as a result of the interviews [1]. The Rasch analysis showed that the items measure a unidimensional construct and the results showed satisfactory model fit including all 19 items of the CRI and all 322 participants of the sample [1], which supports the internal structure validity. Significant correlations between the person parameters of the Rasch scaling and the comparisons by sex, the amount and level of chemistry instruction at upper secondary course, year, and courses at university, provided evidence for relations with other variables validity. Coefficient alpha and EAP-PV (expected a posteriori/plausible value) were used to estimate the single administration reliability of the items [1, 2] and the values indicated an acceptable to good reliability of the CRI. The proposed solutions for each item were taken as a basis for formulating the scoring guide and the scoring guide was formulated in a very explicit fashion, the assignment of partial and full credit was an immediate result of the students’ answers, with almost no tolerance for interpretation. Hence, the coding process was completed nearly automatically, based on the scoring guide and by calculating the sum of all subtasks.

Additional psychometric evidence was provided regarding difficulty and discrimination. Both classical test theory and Rasch analysis were used to explore the difficulty of the items. Item difficulty ranging from 0.19 to 0.88 and weighted mean squares (WMNSQ) values ranging from 0.9 to 1.15 suggested that most items were acceptable. The mean value of the person parameters (0.33) suggested that the items of the CRI were a little too easy for the persons being tested [1]. Item discrimination was estimated via classical test theory with corresponding values of the items in the inventory ranging from 0.10 to 0.51, suggesting some items need further research to investigate their item functionality [1]. The panelists appreciated that both classical test theory and Rasch analysis were used in the studies. Further research is needed to investigate the validity and reliability of the scores from the instrument if this is used beyond the current population tested (i.e. German student-teachers).

References

[1] Taskin, V., Bernholt, S., & Parchmann, I. (2015). An inventory for measuring student teachers' knowledge of chemical representations: design, validation, and psychometric analysis. Chemistry Education Research and Practice, 16(3), 460-477. https://doi.org/10.1039/C4RP00214H

[2] Taskin, V., Bernholt, S., & Parchmann, I. (2017). Student teachers’ knowledge about chemical representations. International Journal of Science and Mathematics Education, 15(1), 39-55. https://doi.org/10.1007/s10763-015-9672-z