Why do we need CHIRAL?
Instructors and researchers in the chemistry education community share a common goal: We all want to know what students know. And not just what they know, but how they think, how they feel, and any information that can help us understand the educational process a little better so that we can grow as teachers and researchers.
One problem that many of us face, however, is having the confidence that our assessments, both formative and summative, are actually eliciting this information in a valid (measures what we intend it to measure) and reliable (consistent, minimizing measurement error) way. Sometimes this is because we do not have the relevant expertise in the complicated methods required to “validate” an instrument. Other times, we might simply not have the time to dedicate to a thorough review of available assessments regarding a certain concept or trait.
Whichever is the case, we feel strongly that having a collection of assessments/inventories that compiles all of this in one convenient place will be a valuable tool for the chemistry education community. But we don’t just think this, we have already asked members of the community to chime in on the project’s goals. After in-depth conversations with researchers, lab coordinators, college instructors, and high school teachers, we have gleaned an understanding of what these various members of the chemistry education community know about evidence-based assessments, what they would like to learn, and how a tool like CHIRAL can be useful for them.
With insight into what the chemistry education community wants out of this tool, we have designed CHIRAL to not only be a repository for available assessments, but also a place to become a true consumer of the information available in relevant literature. Our hope is that we can make these research-based assessments not only more convenient to identify, but also more accessible for those who do not have an in-depth training in the field of psychometrics.
We extract information to fairly represent the majority of inventories used by the chemistry education community
Identifying relevant assessments and all the evidence that supports their validity and reliability is extremely time-consuming. Instrument developers and researchers are trying to adhere to ever-shifting standards for what evidence needs to be presented and how it needs to be presented. Additionally, those outside of the chemistry education community frequently contribute instruments used by those in the field, bringing in additional standards and journals where the source instruments are found.
Further complicating matters: different versions of instruments are published; some assessments are released fully while others are not; pertinent information is not always included in journal publications or discussions… it is no wonder why most of the chemists we spoke to found that just finding a potential inventory is a daunting task.
The CHIRAL team has developed a process to systematically search the literature and identify all instruments used by researchers and practitioners and organize the wealth of information that is available for each. This process is not trivial and constantly evolves as new types of information come to light. Throughout the process, we make our best attempt to find and display all relevant information one would need to decide whether or not to implement an assessment in their teaching or research.
Additionally, our intentions are not to advocate for certain assessments over others, just to present all the information that is currently available in a consistent format. Important to the consumer of this information is that just because one inventory has some information that another appears to be missing does not imply that the former is “better” than the latter.
Panel review by experts helps consumers make data-informed decisions
Identifying the inventories and the evidence that supports their validity and reliability is helpful, but a key component of CHIRAL is to take this a step further by implementing expert reviews. In our market research, we discovered that many in the chemistry community do not feel confident they would know what they “should” look for in the evidence for validity and reliability. CHIRAL’s solution to this is to provide a summary review by a panel experts that will clarify what evidence is available and the quality of that evidence to ease this burden.
Even having all the information collected in one place, inventories are frequently used in several different publications that present different types of evidence. One of the key purposes of these reviews is to make a user-friendly summary of all this evidence so that instructors and researchers can make data-informed decisions about which assessments they will implement in their classes and/or research projects.
Panel review by experts helps consumers make data-informed decisions
How to cite CHIRAL
MLA citation (general website)
Barbera, J.; Harshman, J.; and Komperda, R. (Eds.). The Chemistry Instrument Review and Assessment Library (CHIRAL), https://chiral.chemedx.org. Accessed <insert date>.
Example:
“To identify assessment instruments aligned with the evaluation of student attitude, a search of the Chemistry Instrument Review and Assessment Library (CHIRAL) [1] was conducted. This search generated a number of potential instruments including…”
- Barbera, J.; Harshman, J.; and Komperda, R. (Eds.). The Chemistry Instrument Review and Assessment Library (CHIRAL), https://chiral.chemedx.org. Accessed 31 May 2022.
MLA citation (specific instrument review)
Anonymous Review Panel. “Review of <insert full instrument name>.” Barbera, J.; Harshman, J.; and Komperda, R. (Eds.). The Chemistry Instrument Review and Assessment Library (CHIRAL), <insert date of CHIRAL review>, <link to specific instrument page on CHIRAL site>.
Example:
“The Attitude toward the Subject of Chemistry Inventory (ASCI) [2] was selected as the measure of student attitude for this project. Selection of the ASCI was based, in part, on 1) the measures alignment with the variables of interest for the project and 2) the CHIRAL review panel summary [3] of the available evidence supporting the validity and reliability of ASCI data with students enrolled in a general chemistry setting.”
- Bauer, C.F. (2008). Attitude towards chemistry: A semantic differential instrument for assessing curriculum impacts. Journal of Chemical Education, 85(10), 1440-1445.
- Anonymous Review Panel. “Review of Attitude toward the Subject of Chemistry Inventory (Version 1).” Barbera, J.; Harshman, J.; and Komperda, R. (Eds.). The Chemistry Instrument Review and Assessment Library (CHIRAL), 02 June 2021, https://chiral.chemedx.org/instrument/attitude-toward-subject-chemistry-inventory, .