(Post last updated June 24, 2022)
Review panel summary
The Particulate Nature of Matter Assessment (ParNoMA) consists of 20 multiple choice items and includes 5 different concepts about particles (size, weight, composition, phase change, and energy) [1, 2]. The instrument was designed to assess a student's conceptual understanding regarding the particulate nature of matter (PNM), especially as it relates to phases of matter and phase change topics. The instrument has been evaluated with middle/high school students and students enrolled in college general chemistry [1-3]. The instrument is used as a pre- and post-test. Student scores are reported as the percent of the number of questions the participants correctly answered out of the total score [1, 2]. The authors did not differentiate the student scores at different academic levels [1, 2].
Several aspects of validity and reliability have been assessed for the data generated by the ParNoMA. This instrument is based on a previously unpublished assessment developed by the authors. Evidence in support of the test content validity was reviewed by a small sample of experts [1]. Faculty and graduate students provided feedback regarding the correctness of the answers to the items at the development stage. During pilot testing undergraduate students provided feedback on the clarity of the items and this was used to provide some evidence for the response process validity of ParNoMa data [1, 2]. Evidence in support of the internal structure of ParNoMa data was reported through factor analysis. The analysis of the 20 items [3] suggested three factors (size of particles, composition of molecules, and particle motion and energies) according to the eigenvalues and the shape of the scree plot. Each factor consisted of 4-9 questions [3]. This three-factor finding was different from the initial design of the instrument of five factors. The authors of the study mentioned that the results were conceptually consistent with the initial design and justified the results of factor analysis by combining factors [3]. The instrument was used in evaluating the effectiveness of a treatment (intervention) showing significant gain score between the researched groups.
In terms of evidence supporting reliability, coefficient alpha was used to estimate the single administration reliability for the total score of the ParNoMA [1-3] but not for the subscales. A small sample of students were asked to justify their responses after they completed the pretest and ¾ of the students’ responses matched their pencil-and-paper test, providing further response process validity evidence.
Recommendations for use
The ParNoMA was designed to assess students’ conceptual understanding regarding the PNM (phases of matter and phase change topics) [1]. As developed, this instrument was designed to be used to assess students’ conceptual understanding of the PNM using a total score, however additional evidence of internal structure is suggested, as was acknowledged by the authors [3]. Less confidence would be warranted for interpreting the subscales, with the exception of the grouping (size of particles, composition of molecules, and particle motion and energies) found in a later study [3]. However, to date no evidence in support of the reliability of any subscales has been shown. Studies provide some evidence that the ParNoMa produces valid and reliable data from the sample of college students [2, 3].
Details from panel review
The ParNoMA developers gathered and reported several aspects of validity and reliability evidence for the instrument [1-3]; however some of the evidence was found to be lacking or is not supported by the reported details. For example, the test content and the response process validity is not well supported by the reported details to assess this evidence or for alignment of the instrument with the theories mentioned by the authors. For the internal structure validity, the authors of the study mentioned that the results were conceptually consistent with the initial design and justified the results of factor analysis by combining factors [3] though evidence was lacking to support this action. Coefficient alpha has been used to estimate single administration reliability for the whole instrument [1-3] but not for the original subscales nor those found through factor analysis, which leaves a lack of support for the interpretation of the data generated by the instrument's subscales.
References
[1] Yezierski, E.J., & Birk, J.P. (2006). Misconceptions about the particulate nature of matter. Journal of Chemical Education, 83(6), 954-960. https://doi.org/10.1021/ed083p954
[2] Bridle, C.A., & Yezierski, E.J. (2012). Evidence for the effectiveness of inquiry-based, particulate-level instruction on conceptions of the particulate nature of matter. Journal of Chemical Education, 89(2), 192-198. https://doi.org/10.1021/ed100735u
[3] Tang, H., & Abraham, M.R. (2016). Effect of computer simulations at the particulate and macroscopic levels on students’ understanding of the particulate nature of matter. Journal of Chemical Education, 93(1), 31-38. https://doi.org/10.1021/acs.jchemed.5b00599