(Post last updated June 16, 2022)
Review panel summary
The Test of Scientific Literacy Skills (TOSLS) is a 30 question multiple-choice instrument designed to assess scientific literacy skills in undergraduate STEM courses [1]. Though the instrument has been used in earth science and chemistry settings [2, 3], the majority of the reliability and validity evidence comes from undergraduate biology courses [1, 2]. Evidence supporting test content validity was established through an iterative process that involved a robust literature review of both policy documents and student misconceptions of scientific literacy, survey responses from numerous biology faculty members, and review of the instrument by external biology education researchers [1]. After the second revision of the instrument content, point biserial data were used to identify items that needed to be further revised and/or omitted to improve the quality of the instrument [1]. Evidence for response process validity was established through numerous interviews (both cognitive process and reflective) and a focus group with biology students (both majors and non-majors) throughout the iterative instrument development process [1]. Evidence for internal structure validity employed exploratory factor and principal components analysis to reveal a one factor (scientific literacy) solution to the final version of the TOSLS [1]. Single administration reliability evidence for the TOSLS includes KR-20 values for both pre-semester and post-semester administrations of the instrument with biology undergraduate students [1]. Item difficulty and item discrimination values for items on the TOSLS are also computed based on responses from both project-based and lecture-based nonmajors biology courses [1].
Recommendations for use
The TOSLS is designed to measure scientific literacy skills [1]. Based on the reliability and validity evidence, the TOSLS can be used to assess the scientific literacy skills in biology courses. However, due to the very limited reliability and validity evidence from other STEM domains, additional reliability and validity testing is recommended prior to implementation in non-biology courses. Future studies are encouraged to keep in mind the internal structure validity of the one factor (scientific literacy) solution for the instrument [1] when scoring and reporting data.
Details from panel review
All of the reliability and validity evidence for the TOSLS data has been collected with biology students [1]. The evidence for internal structure validity of the instrument indicates a one factor solution, however the original presentation of the instrument claims to have two categories [i.e., (1) understanding the methods of inquiry that lead to scientific knowledge and (2) organize, analyze, and interpret quantitative data and scientific information] that evaluate nine different literacy skills [1]. In another study consisting of 727 students from college biology, chemistry, and earth science courses, the relation between TOSLS scores and SAT math scores, SAT reading scores, previous term GPA, and year in school (junior/senior vs. freshmen/sophomore) provide some evidence for relation to other variables validity [2]. However, some of these relations were viewed as tenuous by the panel because they utilized the claimed two factor TOSLS solution, one for each of the categories proposed in the original instrument; however, that structure has not been supported by factor analysis. Further research is needed to help clarify the internal structure of the instrument across all STEM domains.
References
[1] Gormally, C., Brickman, P., & Lutz, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE–Life Science Education, 11(4), 364-377. https://doi.org/10.1187/cbe.12-03-0026
[2] Shaffer, J.F., Ferguson, J., & Denaro, K. (2019). Use of the test of scientific literacy skills reveals that fundamental literacy is an important contributor to scientific literacy. CBE–Life Sciences Education, 18(3), ar31. https://doi.org/10.1187/cbe.18-12-0238
[3] Bouldin, R.M., & Folchman-Wagner, Z. (2019). Chemistry of sustainable products: Filling the business void in green-chemistry curricula. Journal of Chemical Education, 96(4), 647-651. https://doi.org/10.1021/acs.jchemed.8b00619