This week, the Council of Chief State School Officers (CSSO) held its annual National Conference on Student Assessment in Atlanta where education professionals, including individuals and leaders from federal, state, and district agencies, as well as test publishers and administrators, educational consultants, university faculty and other experts gathered to discuss the critical role of assessment in the education system. Underscoring our mission to advance science, HumRRO experts gave presentations and served as a discussant at NCSA.
In the session, “Changing How We Assess Science: Perspectives from the National and State Levels,” Emily Dickinson, Ph.D., senior staff scientist at HumRRO, and her co-presenters updated attendees on revisions to the National Assessment of Educational Progress (NAEP) science framework.
The revisions, overseen by the National Assessment Governing Board (NAGB), are informed in part by research designed to understand the landscape of state science content standards and curricula. Dickinson shared a summary of a 2021 HumRRO study comparing content outlined in the NAEP science framework with state content standards that were not part of the Next Generation Science Standards (NGSS).
Using computer algorithmic and human review by staff and national science experts, the HumRRO study found substantial differences between NAEP content and standards for states that are not direct adopters of NGSS. Such gaps may result in assessments that don’t adequately measure the true state of science achievement in the United States.
When examining the content covered by the full set of states’ science standards, the study found many state statements do not overlap in content with any NAEP statement:
- At grade 4, 31% of all state content statements reviewed by HumRRO experts and external science experts were rated as not overlapping with a NAEP content statement.
- At grade 8, 32% of all state content statements reviewed by HumRRO experts and external science experts were rated as not overlapping with a NAEP content statement.
- At grade 12, 55% of all state content statements reviewed by HumRRO experts and external science experts were rated as not overlapping with a NAEP content statement
Experts rated the least amount of content overlap between NAEP and states’ standards at grade 12. As with a prior NAEP-to-NGSS comparison conducted by AIR, experts also rated the least amount of overlap in content between NAEP and states’ standards for the physical science domain, especially at grades 8 and 12.
“Science experts noted that states whose standards are based on the National Resource Council (NRC) K–12 Science Framework have more in common with NAEP than states whose standards are not based on the NRC framework,” said Dickinson.
At the conference, participants also had the opportunity to discuss lessons learned from assessment results over the past several years, especially in light of disruptions and changes caused by the pandemic. HumRRO Principal Scientist Art Thacker, Ph.D., presented “The Application of Virtual Meetings with Standard Settings,” exploring the various aspects of deciding to use a virtual standard setting process, the development and implementation of the available procedures and tools, and the outcomes of using a virtual process.
Standard setting is an essential activity for any assessment program where performance level cut scores are required before reporting student performance on the assessment. Due to the pandemic, it was necessary to determine ways to conduct these as virtual meetings while establishing procedures, protocols, and tools for conducting valid standard settings virtually. During his presentation, Thacker shared lessons learned from these programs with a focus on the future application of virtual standard settings.
Thacker also served as a discussant in the session, “Implications of Psychometric Issues and Policy Decisions on SY20-21 Summative Assessments.” In the 2019-2020 and 2020-2021 school years, sweeping changes to statewide summative assessments and accountability were made in response to the pandemic. These changes impacted students, schools, and districts in a variety of ways and led to many foreseeable and some unanticipated consequences. Thacker provided his expertise and context to the discussion on implications of technical adjustments to policy decisions and their effect on accountability and state trends.
By sharing their expertise and knowledge, Dickinson and Thacker continued HumRRO’s tradition of honoring our mission as a nonprofit to give back to science and the profession.
