(Post last updated July 21, 2021)
Review panel summary
The Implicit Information from Lewis Structures Instrument (IILSI) is a one question instrument that provides students with seventeen options to select as to whether they think the option can be determined from Lewis structures. It has been evaluated with students in the general chemistry sequence [1, 2], the organic chemistry sequence [1, 2], and with upper level undergraduate and graduate students [1] at two universities in the United States. Several aspects of validity and reliability have been investigated for the data generated by the IILSI. The options that are included in the IILSI are based on open-ended question responses in interviews and surveys which provide a solid foundation for the instrument. Evidence in support of test content validity was established by multiple graduate students and faculty agreeing that it measured its intended purpose and that the options to be chosen represent the information that can be determined from Lewis structures. During development of the IILSI, two additional open-ended questions were added to determine if any answer choices were unclear to the students and to allow students to add any additional options they felt were absent from the list [1]. Additionally, the results of students not being able to connect structures with properties, which were found in the data generated by IILSI, were consistent with interview data from previous studies [1]. These provide evidence for response process validity. In looking at evidence to show relation to other variables, students who have not had instruction on the specific option at the time of data collection, such as resonance, do not choose the option and those that have had instruction do choose the option [1]. Along these lines, results from IILSI also show a relation between the options students choose and the type of curriculum used in the course [2]. As for reliability, the authors note that test-retest reliability, as normally defined, would not be the best way to establish reliability [1]. However, they did show consistency in initial IILSI scores for two first-semester general chemistry samples from different fall semesters exposed to the same instructor and instruction [1].
Recommendations for use
The IILSI was developed to determine students’ perceptions of the information that can be determined from Lewis structures [1]. There was a large amount of validity evidence to support that the IILSI does meet its intended purpose. However, due to the inherent nature of the instrument, reliability evidence is lacking, although the instrument does seem to provide consistent results across semesters [1]. Currently, the IILSI data generated has only been used to look at each option for an entire group of students as opposed to looking at the number of options students correctly choose [1, 2]. Overall, it is recommended that additional evidence be investigated to ensure the IISLI functions in samples beyond that in which it was developed.
Details from panel review
The developers of the IILSI provided an extensive amount of validity evidence to support that the instrument is measuring students’ perceptions of the information that can be determined from Lewis structures [1]. Additionally, the authors followed best practices in developing the instrument based soundly on results from previous qualitative studies [1]. Evidence for test content validity was established through two separate means: 1) four graduate students and a faculty member agreed that it measured its intended purpose and 2) five graduate students and two faculty agreed the options to be chosen represent the information that can be determined from Lewis structures [1]. The authors established evidence for response process validity during the development of the IILSI by including additional questions at the end to ensure that there were no options that the students felt were unclear and to allow students to add additional information they felt could be determined from Lewis structures that were not already included [1]. Additional evidence for response process validity included that the data generated from the IILSI resulted in the same findings as the previous qualitative studies, which showed that students were not able to connect structures with properties [1]. The IILSI was also able to provide evidence of relation to other variables including exposure to instruction on the topic and the type of curriculum used to teach the topic [1, 2].
While the authors provide many means of validity evidence for the IILSI, the evidence provided to establish reliability is limited. The authors note, and the panel agrees, that test-retest reliability, which is the typical means of establishing reliability with an instrument such as this, would not be appropriate due to the impact it might have on students’ future responses. Instead, the authors report the consistency of responses from administration of the IILSI to two similar groups pre-instruction at the start of general chemistry. In comparing the groups, the authors found no significant difference [1]. Additionally, while the majority of the evidence comes from students at a research university in the southeastern United States, the authors did compare the responses to students at the end of first-semester general chemistry at a comprehensive university in the southeastern United States and found no significant difference [1]. While these are not traditional means of establishing reliability evidence for an instrument, the panel does agree that the IILSI provides consistent results for similar samples. The question remains as to the reliability of the instrument outside of these very specific groups.
References
[1] Cooper, M. M., Underwood, S. M., & Hilley, C. Z. (2012). Development and validation of the implicit information from Lewis structures instrument (IILSI): Do students connect structures with properties? Chemistry Education Research and Practice, 13(3), 195–200. https://doi.org/10.1039/C2RP00010E
[2] Underwood, S. M., Reyes-Gastelum, D., & Cooper, M. M. (2016). When do students recognize relationships between molecular structure and properties? A longitudinal comparison of the impact of traditional and transformed curricula. Chemistry Education Research and Practice, 17(2), 365–380. https://doi.org/10.1039/C5RP00217F