(Post last updated 20 December 2024)
Review panel summary
The General, Organic, and Biological Chemistry Knowledge Assessment (GOB-CKA) instrument was developed with the purpose of providing formative assessment for pre-nursing students enrolled in a GOB course about topics of relevance to their future nursing career. The 45-item assessment contains 38% general chemistry items, 13% organic chemistry items, and 49% biological chemistry items. It is designed to be completed within a 50-minute lecture period. The items contain one correct answer and three distractors which were created from misconceptions that students presented during response process think-aloud interviews when the instrument was being developed. Student language was used verbatim for these distractors.
The instrument was first developed by conducting open-ended interviews with practicing nurses, nurse educators, and GOB chemistry instructors. The interviews were analyzed, and topics were identified. These topics were then classified as Important (direct application to nursing and clinical practice), Foundational (not directly important for clinical practice but facilitate understanding of important topics), and Not Important (no direct application and not significant for practice). A national survey was conducted where 264 nurse educators and 16 GOB instructors were presented the topics and were asked to label the topics as Important, Foundational, and Not Important. There was a 95% agreement among respondents.
After identifying the Important and Foundational topics, open-ended questions were developed, and expert (four GOB instructors and nine nursing educators) responders were sought to establish evidence based on test content. In addition, 20 pre-nursing students were interviewed to establish response process validity. Sixty-seven open-ended questions were tested during this process. Each student was presented with 7-10 of these questions in an interview.
The 67 open-ended questions were turned into multiple choice questions after the interviews, and the instrument was piloted with 458 pre-nursing students at two different universities. The instrument was presented in a paper and pencil format and students responded using a scantron. All students finished within 45 minutes. The item-level data was analyzed using item difficulty and item discrimination. There were 20 items that were deemed as presenting difficulty and discrimination outside of the desirable range, these items were removed. A beta test with the remaining 47 items revealed two additional items that presented difficulty and discrimination outside of the desirable range; these items were also removed. The final instrument contains 45 items displaying good to very good discrimination and an average of 0.48 item difficulty.
Evidence based on relationships to other variables was measured by performing a correlation analysis between the GOB-CKA and the GOB-ACS examination. A correlation coefficient of 0.72 was found between these two assessments, indicating that students who perform well on the GOB-CKA are likely to perform well on the GOB-ACS exam.
Recommendations for use
The authors recommend that this instrument be used as a formative assessment in GOB pre-nursing courses, as an indication of a student’s understanding of foundational and practical knowledge of chemistry relevant to nursing practice. Given average item discrimination and difficulty, this assessment is meant to be adequately challenging for most students. Given the evidence of correlation to student performance on the GOB-ACS exam, this assessment may also be used to formatively assess student preparation for this commonly used final examination.
Details from panel review
The GOB-CKA assessment instrument was developed using a variety of validity measures such as evidence based on test content, response process, and relation to other variable(s). As discussed in the panel summary, these sources provide users with some validity evidence to support data. However, evidence based on internal structure validity was not explored. This evidence would provide users support for scoring of this instrument (e.g., with a total score). In addition, the authors conducted t-tests to provide evidence that data from two different universities, or from two different semesters, could be used together. More robust evidence of this practice could be reported by performing measurement invariance testing or differential item functioning analyses.
Only one aspect of reliability was explored. Coefficient alpha was reported as evidence of single administration reliability, reporting a single value of 0.76 to indicate good reliability of data. However, this statistic operates under the assumption that the model being tested is a tau-equivalent model, or that all items share the same true score or have equal factor loadings. These assumptions were not reported; therefore, it is unclear whether this reliability measure is appropriate for the data reported by the GOB-CKA.
References
[1] Brown, C. E., Hyslop, R. M., and Barbera, J. (2015), Development and Analysis of an Instrument to Assess Student Understanding of the GOB Chemistry Knowledge Relevant to Clinical Nursing Practice, Biochem. and Mole. Bio. Educ., 43(1), 13-19