Skip to main content

Test Of Scientific Literacy Skills

TOSLS

OVERVIEW
Overview
Listed below is general information about the instrument.
Summary
Original author(s)

  • Gormally, C., Brickman, P., & Lut, M.
Original publication

  • Gormally, C., Brickman, P., & Lut, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates' evaluation of scientific information and arguments. CBE Life Sciences Education, 11(4), 364-377.
Year original instrument was published 2012
Inventory
Number of items 28
Number of versions/translations 2
Cited implementations 3
Language
  • English
Country United States
Format
  • Multiple Choice
Intended population(s)
  • Students
  • Undergraduate
Domain
  • Cognitive
Topic
  • Science Literacy
Evidence
The CHIRAL team carefully combs through every reference that cites this instrument and pulls all evidence that relates to the instruments’ validity and reliability. These data are presented in the following table that simply notes the presence or absence of evidence related to that concept, but does not indicate the quality of that evidence. Similarly, if evidence is lacking, that does not necessarily mean the instrument is “less valid,” just that it wasn’t presented in literature. Learn more about this process by viewing the CHIRAL Process and consult the instrument’s Review (next tab), if available, for better insights into the usability of this instrument.

Information in the table is given in four different categories:
  1. General - information about how each article used the instrument:
    • Original development paper - indicates whether in which paper(s) the instrument was developed initially
    • Uses the instrument in data collection - indicates whether an article administered the instrument and collected responses
    • Modified version of existing instrument - indicates whether an article has modified a prior version of this instrument
    • Evaluation of existing instrument - indicates whether an article explicitly provides evidence that attempt to evaluate the performance of the instrument; lack of a checkmark here implies an article that administered the instrument but did not evaluate the instrument itself
  2. Reliability - information about the evidence presented to establish reliability of data generated by the instrument; please see the Glossary for term definitions
  3. Validity - information about the evidence presented to establish reliability of data generated by the instrument; please see the Glossary for term definitions
  4. Other Information - information that may or may not directly relate to the evidence for validity and reliability, but are commonly reported when evaluating instruments; please see the Glossary for term definitions
Publications: 1 2 3

General
Original development paper ✔
Uses the instrument in data collection ✔ ✔ ✔
Modified version of existing instrument ✔
Evaluation of existing instrument ✔ ✔

Reliability
Test-retest reliability
Internal consistency ✔
Coefficient (Cronbach's) alpha
McDonald's Omega
Inter-rater reliability
Person separation
Generalizability coefficients
Other reliability evidence

Vailidity
Expert judgment ✔
Response process ✔
Factor analysis, IRT, Rasch analysis
Differential item function
Evidence based on relationships to other variables ✔ ✔
Evidence based on consequences of testing
Other validity evidence

Other information
Difficulty ✔
Discrimination ✔
Evidence based on fairness
Other general evidence
Review
DISCLAIMER: The evidence supporting the validity and reliability of the data summarized below is for use of this assessment instrument within the reported settings and populations. The continued collection and evaluation of validity and reliability evidence, in both similar and dissimilar contexts, is encouraged and will support the chemistry education community’s ongoing understanding of this instrument and its limitations.
This review was generated by a CHIRAL review panel. Each CHIRAL review panel consists of multiple experts who first individually review the citations of the assessment instrument listed on this page for evidence in support of the validity and reliability of the data generated by the instrument. Panels then meet to discuss the evidence and summarize their opinions in the review posted in this tab. These reviews summarize only the evidence that was discussed during the panel which may not represent all evidence available in the published literature or that which appears on the Evidence tab.
If you feel that evidence is missing from this review, or that something was documented in error, please use the CHIRAL Feedback page.

Panel Review: Test of Scientific Literacy Skills (TOSLS)

(Post last updated June 16, 2022)

Review panel summary

The Test of Scientific Literacy Skills (TOSLS) is a 30 question multiple-choice instrument designed to assess scientific literacy skills in undergraduate STEM courses [1]. Though the instrument has been used in earth science and chemistry settings [2, 3], the majority of the reliability and validity evidence comes from undergraduate biology courses [1, 2]. Evidence supporting test content validity was established through an iterative process that involved a robust literature review of both policy documents and student misconceptions of scientific literacy, survey responses from numerous biology faculty members, and review of the instrument by external biology education researchers [1]. After the second revision of the instrument content, point biserial data were used to identify items that needed to be further revised and/or omitted to improve the quality of the instrument [1]. Evidence for response process validity was established through numerous interviews (both cognitive process and reflective) and a focus group with biology students (both majors and non-majors) throughout the iterative instrument development process [1]. Evidence for internal structure validity employed exploratory factor and principal components analysis to reveal a one factor (scientific literacy) solution to the final version of the TOSLS [1]. Single administration reliability evidence for the TOSLS includes KR-20 values for both pre-semester and post-semester administrations of the instrument with biology undergraduate students [1]. Item difficulty and item discrimination values for items on the TOSLS are also computed based on responses from both project-based and lecture-based nonmajors biology courses [1].

Recommendations for use

The TOSLS is designed to measure scientific literacy skills [1]. Based on the reliability and validity evidence, the TOSLS can be used to assess the scientific literacy skills in biology courses. However, due to the very limited reliability and validity evidence from other STEM domains, additional reliability and validity testing is recommended prior to implementation in non-biology courses. Future studies are encouraged to keep in mind the internal structure validity of the one factor (scientific literacy) solution for the instrument [1] when scoring and reporting data.

Details from panel review

All of the reliability and validity evidence for the TOSLS data has been collected with biology students [1]. The evidence for internal structure validity of the instrument indicates a one factor solution, however the original presentation of the instrument claims to have two categories [i.e., (1) understanding the methods of inquiry that lead to scientific knowledge and (2) organize, analyze, and interpret quantitative data and scientific information] that evaluate nine different literacy skills [1]. In another study consisting of 727 students from college biology, chemistry, and earth science courses, the relation between TOSLS scores and SAT math scores, SAT reading scores, previous term GPA, and year in school (junior/senior vs. freshmen/sophomore) provide some evidence for relation to other variables validity [2]. However, some of these relations were viewed as tenuous by the panel because they utilized the claimed two factor TOSLS solution, one for each of the categories proposed in the original instrument; however, that structure has not been supported by factor analysis. Further research is needed to help clarify the internal structure of the instrument across all STEM domains.

References

[1] Gormally, C., Brickman, P., & Lutz, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates’ evaluation of scientific information and arguments. CBE–Life Science Education, 11(4), 364-377. https://doi.org/10.1187/cbe.12-03-0026

[2] Shaffer, J.F., Ferguson, J., & Denaro, K. (2019). Use of the test of scientific literacy skills reveals that fundamental literacy is an important contributor to scientific literacy. CBE–Life Sciences Education, 18(3), ar31. https://doi.org/10.1187/cbe.18-12-0238

[3] Bouldin, R.M., & Folchman-Wagner, Z. (2019). Chemistry of sustainable products: Filling the business void in green-chemistry curricula. Journal of Chemical Education, 96(4), 647-651. https://doi.org/10.1021/acs.jchemed.8b00619

Versions
Listed below are all versions and modifications that were based on this instrument or this instrument were based on.
Instrument is derived from:
Name Authors
Scientific Reasoning Test

  • Sundre, D. L., & Thelk, A. D.
Quantitative Reasoning Test

  • Sundre, D. L., & Thelk, A. D.
Name Authors
Performance Assessment Of Undergraduate Research

  • Harsh, J.A.
Citations
Listed below are all literature that develop, implement, modify, or reference the instrument.

  1. Gormally, C., Brickman, P., & Lut, M. (2012). Developing a test of scientific literacy skills (TOSLS): Measuring undergraduates' evaluation of scientific information and arguments. CBE Life Sciences Education, 11(4), 364-377.

  2. Shaffer, J.F., Ferguson, J., & Denaro, K. (2019). Use of the test of scientific literacy skills reveals that fundamental literacy is an important contributor to scientific literacy. CBE Life Sciences Education, 18(3), .

  3. Bouldin, R.M., & Folchman-Wagner, Z. (2019). Chemistry of Sustainable Products: Filling the Business Void in Green-Chemistry Curricula. Journal of Chemical Education, 96(4), 647-651.