(Post last updated 09 June 2023)
Review panel summary
The Bonding Representations Inventory (BRI) consists of 23 multiple choice items (7 one-tiered items and 8 two-tiered item pairs) [1]. This concept inventory was designed to measure students’ ‘misconceptions’ about bonding and bonding representations related to four themes: periodic trends, electrostatic interactions, the octet rule, and surface features [1]. The BRI has been administered in high-school level, advanced placement, and undergraduate general chemistry courses at various high schools and post-secondary institutions across the United States. At each institution, the BRI was administered both post-instruction and post-assessment of content related to bonding and Lewis structures [1].
Evidence based on test content was collected during development of the BRI. The generation of items and distracters was informed by interview data collected from students enrolled in high school chemistry or undergraduate general chemistry. Additionally, once the developers had created an alpha-version of the concept inventory, each of the items were reviewed by a panel of experts (five general chemistry instructors). Evidence based on response process was collected for the alpha-version of the BRI. This evidence was collected via cognitive interviews with two students enrolled in an undergraduate general chemistry course, although descriptions of these interviews were not provided by the developers. Expert reviews, cognitive interviews, and item analysis of data were all used to inform the modification or removal of items from the alpha-version of the BRI, resulting in the final version of the concept inventory.
Evidence of single administration reliability was assessed through the calculation of coefficient alpha for each course level in which BRI data was collected (i.e., high school, advanced placement, and undergraduate general chemistry). The alpha value for the advanced placement students’ data was found to be acceptable, while the alpha values for the high school and undergraduate general chemistry data were lower than the typically recommended cutoff for the alpha metric. While the developers chose to report alpha values for their data, they acknowledged that alpha may be an inappropriate indicator of reliability for concept inventories, “owing to the fragmented knowledge being measured” [1].
Evidence based on relation to other variables was assessed by comparing the performance of high school students, advanced placement students, and undergraduate general chemistry students on the BRI. The developers of the instrument proposed that “[b]ecause the BRI was administered to students in various stages of understanding about the construct of bonding, it would be expected that students enrolled in higher-level courses ought to perform better on the inventory.” Kruskal-Wallis H tests were used to compare the relative rankings of the data and significant effects were found for class type on total BRI scores. Additionally, post-hoc Mann-Whitney U tests revealed that both the undergraduate general chemistry and the advanced placement students significantly outperformed the high school students, and the undergraduate general chemistry students performed significantly better than the advanced placement students, as theorized by the developers.
Item difficulty and item discrimination of the data generated by the BRI were also examined. Data collected from high school, advanced placement, and undergraduate general chemistry students showed that item difficulty fell within the acceptable cutoff range for all but eight, three, and five items, respectively. Overall, no items were uniformly difficult or uniformly easy across all three groups of students. Regarding discrimination, data collected from the three aforementioned groups showed acceptable values for Fergusons’ delta, which indicates that the measure can distinguish students’ understanding across the full range of scores. Additionally, item discrimination was investigated to assess how well each item could discriminate between the top-performing students and the lowest-performing students. Data collected from high school, advanced placement, and undergraduate general chemistry students showed that item discrimination fell above the acceptable cutoff for all but seven, three, and nine items, respectively. Like item difficulty, there were no items with poor item discrimination values across all three groups of students.
Recommendations for use
The BRI has been used to assess students’ understanding of bonding and bonding representations in high school, advanced placement, and undergraduate general chemistry courses across the country. Additionally, the concept inventory has been used to identify differential levels of understanding among these groups.
Future users of the concept inventory should be aware that validity evidence based on response process is currently limited for the BRI. When reporting on the limitations of their study, the developers of the BRI highlighted the fact that cognitive interviews intended to collect evidence based on response process were conducted with two undergraduate general chemistry students only and did not include high school and advanced placement students. As highlighted by the developers, future researchers who are interested in using the BRI with these populations may want to conduct additional interviews to determine if these students hold “additional misconceptions not measured by the inventory,” or if “the wording of the items confused [the] students” [1]. This type of validity evidence would greatly strengthen the inferences that can be made from BRI data.
Details from panel review
Regarding evidence based on test content, it is unclear what feedback the expert panel of general chemistry instructors provided on the items, and how that feedback was used to inform the modification or removal of items from the alpha version of the BRI. Regarding evidence based on response process, it is similarly unclear how data collected via cognitive interview data collected from the two undergraduate chemistry students was used to inform the modification or removal of items from the alpha version of the BRI.
For item difficulty, data collected from high school chemistry students showed that values fell within the acceptable cutoff range of 0.3-0.8 for all but eight items (Items 5,6,9,11,12,13,14,18), data collected from advanced placement students showed that values fell within the acceptable cutoff range for all but three items (Items 1,20,21), and data collected from undergraduate general chemistry students showed that values fell within the acceptable cutoff range for all but five items (Items 1,12,18,20,21). Turning to discrimination, data collected from the three aforementioned groups showed acceptable values for Fergusons’ delta (> 0.9). Additionally, item level discrimination was investigated to assess how well each item could discriminate between the top 27% of students and the lowest 27% of students. For item level discrimination, data collected from high school chemistry students showed that values fell above the acceptable cutoff of 0.3 for all but seven items (Items 5,6,9,11,12,13,18), data collected from advanced placement students showed that values fell above the acceptable cutoff for all but three items (Items 1,2,20), and data collected from undergraduate general chemistry students showed that values fell above the acceptable cutoff for all but five items (Items 1,2,9,10,12,18,19,20,21). The developers of the BRI highlighted that the “overall distributions of the discrimination−difficulty points shift across the three courses, suggesting that as students gain knowledge in chemistry, they [are] able to answer more items correctly. Hence, the corresponding shift in difficulty and higher discrimination indices [are] observed, suggesting that the students have less fragmented understanding" [1].
References
[1] Luxford, C.J. & Bretz, S.L. (2014) Development of the Bonding Representations Inventory To Identify Student Misconceptions about Covalent and Ionic Bonding Representations. J. Chem. Educ. 91(3), 312-320.