(Post last updated June 23, 2022)
Review panel summary
The Colorado Learning Attitudes about Science Survey (CLASS-Chem) is a 50 item, 5-point Likert scale, survey that was designed to measure students’ beliefs about chemistry and the learning of chemistry. It has been evaluated with undergraduate students enrolled in a variety of chemistry courses such as introductory chemistry [1], general chemistry I [1, 2], organic I [1], as well as undergraduate students serving as general chemistry peer instructors [3] in the U.S. Several aspects of reliability and validity have been assessed for the data generated by the CLASS-Chem instrument. Test content evidence was established during initial development of the chemistry version of the CLASS-Chem survey by asking 50 chemistry faculty to take the survey and provide feedback on the statements [1]. Response process validity evidence was collected through student interviews [1, 3] which asked participants to explain their reasoning for a selected answer. The interviews allowed the authors to tune the survey to 50 items with unambiguous interpretations [1], although misinterpretations of several items were found in the unique sample of undergraduate students serving as peer instructors [3]. The ability of the instrument to capture differences between student populations that would be expected to have different beliefs (for example, chemistry majors vs. nonmajors) offers validity evidence based on relations to other variables [1]. Validity evidence based on the internal structure of the data using confirmatory factor analysis (CFA) showed adequate model fit for the nine single-factor solutions proposed by the instrument authors [2]. However, a CFA for the correlated nine-factor model failed to converge due to a large number of items overlapping on multiple scales. This result implies that CLASS-Chem data cannot simultaneously measure all nine proposed subscales. A three-factor solution using only unique items from non-highly correlated factors showed adequate model fit [2]. Single administration reliability was estimated using coefficient alpha for the overall CLASS-chem survey [1] and each of the nine subscales measured by the CLASS-Chem [2].
Recommendations for use
The CLASS-Chem was developed with the intention to measure student’s beliefs about chemistry and the learning of chemistry [1]. According to the validity and reliability evidence reported in the literature, there is support that the instrument data can be used to measure undergraduate chemistry student’s beliefs on the subscales of Personal Interest, Confidence, and Atomic-Molecular Perspective of Chemistry simultaneously when the full 50-item survey is administered [2]. Additional validity evidence supports that the 7-item Conceptual Learning scale can be administered as a stand-alone instrument [2]. There is no evidence that any of the other eight scales can be used as stand-alone instruments or that data from the 50-item survey can be used to simultaneously measure all nine subscales. Caution should be used if administering this survey to other samples as some evidence of misinterpretation of items was seen in a sample of peer instructors [3].
Details from panel review
The CLASS-Chem developers [1] and other researchers [2, 3] have reported several aspects of validity and reliability evidence for the instrument . While the CLASS-Chem instrument development provided evidence in support of test content validity and response process validity [1]; further analysis showed some additional aspects of validity and reliability to be lacking. One technique used in the development of CLASS-Chem, reduced basis factor analysis [1], is described as a method used to incorporate the eleven chemistry-specific items into an optimized factor structure. However, this method allowed for individual items to be placed on more than one subscale, thereby complicating the meaning of subscale scores with cross-placed items. Confirmatory factor analysis using the theorized nine-factor solution failed to converge owing to the fact that many items of the 50-item survey overlap into multiple sub-categories [2]. This lack of discriminant validity, coupled with many items lacking a theoretical basis for factor inclusion [2], threatens the validity based on the internal structure of the instrument. No studies in the literature have been able to reproduce evidence based on the internal structure of the 50-item instrument [1, 2]. While a high correlation between students’ responses across different semesters was provided in support of the test-retest reliability of the instrument [1], this type of reliability is typically determined with the same sample of respondents.
References
[1] Adams, K.W., Wieman, C.E., Perkins, K.K., & Barbera, J. (2008). Modifying and validating the Colorado Learning Attitudes about Science Survey for use in chemistry. Journal of Chemical Education, 85(10), 1435-1439. https://doi.org/10.1021/ed085p1435
[2] Heredia, K., & Lewis, J.E. (2012). A psychometric evaluation of the Colorado Learning Attitudes about Science Survey for use in chemistry. Journal of Chemical Education, 89(4), 436-441. https://doi.org/10.1021/ed100590t
[3] Atieh, E.L., & York, D.M. (2020). Through the looking CLASS: When peer leader learning attitudes are not what they seem. Journal of Chemical Education, 97(8), 2078-2090. https://doi.org/10.1021/acs.jchemed.0c00129