(Post last updated June 14, 2022)
Review panel summary
The Scale Concept Inventory (SCI) is a 40-item 5-point Likert-type scale instrument developed by the same authors who developed the Scale Literacy Skills Test (SLST). The developers suggest that the SCI can be used to assess students' misconceptions related to scale. The authors described that during the development of the SLST, evidence of the need to develop a companion test addressing misconceptions was shown. In support of test content validity evidence, the developers describe creating the items from the literature and interviews with students [1]. However, a detailed process of the development was not described in the manuscript and this instrument was not available in the publications, nor in the supplemental information. Relation to other variables validity evidence was collected by correlation studies between the SCI and variables including ACT Math and final exam scores, where the correlations were significant [1, 2]. The panel found no reported reliability evidence.
Some descriptive item statistics are reported in the supplemental materials, but not the items themselves [1]. The panel found it difficult to evaluate this instrument further without a deeper understanding of the items.
Recommendations for use
The SCI has been used in preparatory and general chemistry courses [1, 2]. It has been used as a companion to the Scale Literacy Skills Test (SLST) instrument, and often used as a component of what the authors call the Scale Literacy Score (SLS) [1, 2]. However, the SCI on its own has been shown to correlate to measures of performance, such as final exam scores, at a significant level.
It is unclear that this instrument is needed as a companion to the SLST as intended. It could be used as a stand-alone instrument to assess similar but separate aspects of scale literacy skills and misconceptions of such. It is hard to evaluate how to use this instrument or further provide a recommendation for use without more understanding of the items and their internal structure.
Details from panel review
The authors provided evidence of relation to other variables validity in the form of correlation analyses with a variety of scores including final exam, ACT Math, and others. Correlation analysis between the SCI and the course final exam scores was 0.402 for general chemistry, and 0.250 for preparatory chemistry [1]. All of these correlations were significant, however, no estimates of effect size were reported [1]. The authors used the SCI and the SLST combined into one weighted score (SLS), which was found to be a predictor of final exam scores. Given that the correlations with other variables for the SCI (~0.2-0.3) are lower than those for the SLST (~0.4-0.5), and the correlations do not improve markedly when using the combined SLS (~0.4-0.5) [1 supporting information], there was not enough evidence to support using both instruments. Given the higher correlations, the SLST may be the preferred choice for scale literacy testing.
The limitations of this instrument noted by the panel include the sparse information about test content validity, lack of evidence supporting internal structure validity, as well as a lack of evidence supporting reliability.
References
[1] Gerlach, K., Trate, J., Blecking, A., Geissinger, P., & Murphy, K. (2014). Valid and reliable assessments to measure scale literacy of students in introductory college chemistry courses. Journal of Chemical Education, 91(10), 1538-1545. https://doi.org/10.1021/ed400471a
[2] Trate, J.M., Geissinger, P., Blecking, A., & Murphy, K.L. (2019). Integrating scale-themed instruction across the general chemistry curriculum. Journal of Chemical Education, 96(11), 2361-2370. https://doi.org/10.1021/acs.jchemed.9b00594