(Post last updated June 24, 2022)
Review panel summary
The Chemistry Expectations Survey (CHEMX) is a 47-item, Likert-scale, instrument that was designed to measure student expectations about learning chemistry. It was developed based off of the Maryland Physics Expectations Survey (MPEX) that measures expectations about learning physics. The CHEMX contains 25 items based on the MPEX and 22 items that were developed by the authors to incorporate additional aspects of chemistry. The CHEMX is described as having seven constructs: effort, concepts, math link, reality link, outcome, laboratory, and visualization. Evidence in support of test content validity was provided through the expert knowledge the authors held about chemistry in the development of the additional 22 items [1]. The internal structure of CHEMX data was explored through factor analysis. Seventeen items loaded onto three distinct factors (visualization, concepts, and reality link) [1]. The authors justified the internal structure of the remaining four subscales by their acceptable reliability estimates. Single-administration reliability was explored through Cronbach’s alpha and inter-item correlations. Estimates were provided for both a total CHEMX score and for each subscale [1]. Relations to other variables were explored when comparing total scores on the CHEMX between faculty members of different chemistry disciplines. Chemistry Education faculty had significantly different total scores compared to the other chemistry disciplines and the authors aligned this result with the statement that “individuals who closely study student learning should score higher on an instrument designed to measure one dimension of that learning”. While this statement supports the use of CHEMX scores for comparing chemistry faculty, the panel felt that further justification would be needed to support total score comparisons among different student groups [1].
Recommendations for use
The CHEMX was developed as a measure of student expectations about learning chemistry. Evidence for the internal structure only provided support for the subscales of Visualization, Concepts, and Reality Link. Therefore, the panel recommends caution in the interpretation of data from the other subscales as evidence for them was lacking from the factor analysis result [1]. While scores from each of the three subscales are supported, there was no internal structure validity evidence provided that supports the use of a CHEMX total score. Overall, the panel felt more evidence for validity of the data generated by CHEMEX was warranted. Additional insights may be gained through explorations of the test content and the response process validity as well as more detailed internal structure validity.
Details from panel review
The authors provided some evidence for validity and reliability for data collected by the CHEMX [1]. While the authors held expert knowledge of chemistry and cited Johnston’s triangle as support for the newly developed items, evidence supporting the test content validity was not evaluated beyond the research team. The validity evidence provided through relations with other variables only provides support for comparing chemistry faculty CHEMEX scores, not those of the intended population (i.e., chemistry students). During the panel discussion of validity for internal structure, it was unclear what type of factor analysis was performed and no output from the analysis was provided. In addition, the authors collected data with both faculty and student participants and it was unclear what specific dataset was used for the internal structure analysis. Finally, the authors found three factors through the factor analysis and justified the structure of the remaining a priori factors through single-administration reliability, which the panel felt was not sufficient evidence for the internal structure.
References
[1] Grove, N., & Bretz, S.L. (2007). CHEMX: An instrument to assess students’ cognitive expectations for learning chemistry. Journal of Chemical Education, 84(9), 1524-1529. https://doi.org/10.1021/ed084p1524
[2] Mazzarone, K.M., & Grove, N.P. (2013). Understanding epistemological development in first- and second-year chemistry students. Journal of Chemical Education, 90(8), 968-975. https://doi.org/10.1021/ed300655s
[3] Albrecht, B. (2014). Computational chemistry in the undergraduate laboratory: A mechanistic study of the Wittig reaction. Journal of Chemical Education, 91(12), 2182-2185. https://doi.org/10.1021/ed400008d