(Post last updated July 21, 2021)
Review panel summary
The Metacognitive Activities Inventory (MCAI) is a twenty-seven item, 5-point Likert-scale instrument designed to assess students’ metacognitive skillfulness during chemistry problem solving. It has been evaluated with students enrolled in general chemistry [1, 2] and in an introductory engineering course [3] in two U.S. institutions. Several aspects of validity and reliability have been assessed for the data generated by MCAI. Test content evidence was provided by faculty and graduate students at the development stages and by graduate and senior undergraduate students during pilot testing. During pilot testing undergraduate students provided feedback on the clarity of the instrument and this was used to provide some evidence for response process validity [1]; however, the details provided to assess this evidence are limited. The relation between total MCAI scores and grade point average [1] provides some validity support based on relations with other variables. Another evidence based on relation to other variables is provided by the relation between ability, MCAI scores and problem solving ability [2]. Developers of the MCAI provided information regarding internal structure validity but the results of the factor analysis fail to show evidence for the internal structure of the instrument [1]. In terms of reliability, coefficient alpha has been used to estimate single administration reliability for the whole instrument [1, 3].
Recommendations for use
The MCAI instrument was designed to assess students’ metacognitive skillfulness during chemistry problem solving [1]. According to the validity and reliability aspects reported in the literature, this instrument can be used to assess metacognitive skills using a total score. However, in many instances the evidence provided is limited or does not have enough details; therefore it is suggested that additional evidence is gathered, especially for the internal structure validity to support the use of a total score for the MCAI and its relation with other measures such as performance.
Details from panel review
The MCAI developers gathered and reported some aspects of validity and reliability evidence for the instrument [1, 2]; however some of the evidence is limited since there is a lack of details that have been reported. For example, for internal structure validity, the developers could not show evidence for the intended factors. From theory, the developers were expecting to group items based on the components of metacognition: planning, evaluating, and monitoring; however, the factor analysis did not support this structure. The developers stated that the interdependency of metacognitive skills could affect this structure; therefore total scores were reported. However, in the absence of evidence for a one-factor solution, more support is needed to warrant the use of composite MCAI score (i.e., a total score). Another example of limited evidence is for response process validity. Student feedback on item clarity was collected [1]; however, it is unclear how this feedback was used in evaluating the items.
In terms of relation to other variables, there is evidence that metacognitive skills, as measured by MCAI, are related to performance [1] and to problem solving skills [2]. IRT analysis was used to measure ability and this relation was significantly correlated with MCAI scores [2]. The instrument was evaluated using both classical test theory and item response theory.
References
[1] Cooper, M. M., & Sandi-Urena, S. (2009). Design and Validation of an Instrument To Assess Metacognitive Skillfulness in Chemistry Problem Solving. Journal of Chemical Education, 86(2), 240. https://doi.org/10.1021/ed086p240
[2] Cooper, M. M., Sandi–Urena, S., & Stevens, R. (2008). Reliable multi method assessment of metacognition use in chemistry problem solving. Chemistry Education Research and Practice, 9(1), 18–24. https://doi.org/10.1039/B801287N
[3] Elliott, L., Aqlan, F., Zhao, R., & Janney, M. (2020). Assessment of Metacognitive Skills in Design and Manufacturing. 2020 ASEE Virtual Annual Conference Content Access Proceedings, 34191. https://doi.org/10.18260/1-2--34191