High quality educational assessments play a key role in ensuring that the U.S. remains competitive in today’s globally interconnected world by helping address critical questions such as which children are ready for the next grade, how well their teachers are performing, how entire schools and districts are performing, as well as whether students are prepared for college and careers.
But how can we be sure that the data generated by educational assessments are accurate and informative? When assessment results are used to make potentially life-changing decisions about students, teachers, and school systems, we must be able to trust the information they provide.
HumRRO has a well-established reputation for our technical rigor and objective perspective, making us the vendor of choice for conducting comprehensive quality assurance (QA) efforts. As a prime example, for over 15 years, HumRRO served as the primary QA lead for the National Assessment of Educational Progress (NAEP) — the only nationwide assessment of what American students know and can do in a variety of subject areas. NAEP scores are considered the “gold standard” against which states compare their own assessments.
Over two dozen states have trusted us to evaluate the quality of their assessments, to verify scores, or to ensure that test content accurately reflects educational benchmarks. We conduct psychometric processing for approximately 150 tests annually, impacting over 15 million students each year. In aggregate form, these assessment data inform teacher performance evaluations, educational curriculum and instruction design efforts, school-level needs, and, ultimately, student learning.
Elements of Effective Quality Assurance
The complexity of large-scale assessment systems and the high-stakes decisions that flow from them can result in a very contentious environment. This situation leaves no room for anything but the highest standards of excellence. Our decades of QA experience and strong technical expertise allow us to develop innovative, effective, and customized QA strategies that reflect our clients’ unique assessment goals, educational contexts, and accountability pressures.
Flexibility
We are routinely the go-to vendor for conducting quick turnaround special studies related to NAEP on a variety of cutting‑edge topics, including investigations of optimal score linkage methods and cross-validation strategies. Our propensity to operate in a nimble and efficient manner also serves us well when responding to requests to provide immediate assistance to help a state or testing company deal with apparent irregularities in the test administration process, such as scoring issues or computer interruptions during testing.
Efficiency
Educators and parents demand accurate and timely assessment results. In response, we conduct our score checking QA work in “real time” so that any issues are resolved prior to score reporting. As an example, we often complete third-party checking analyses in a matter of days, allowing students to receive their scores as quickly as possible. We improved the efficiency of the NAEP assessment by creating new data transmission and stakeholder communication protocols, which led to increased transparency and measurable improvements in coordination among the NAEP Alliance contractors.
Comprehensiveness
Nothing replaces a “boots on the ground” perspective when doing good QA work. This in-depth approach allows us to obtain a thorough assessment of item development, test administration, alignment to standards, data processing, and scoring/reporting procedures. As an example, we conducted up to 80 site visits each year where we rigorously evaluated all aspects of the NAEP’s assessment process. In addition, we have provided similar comprehensive QA services of assessment programs at the state level.
Validity
Our flexible and tailored approach, coupled with our technical expertise, lends itself to building and establishing validity evidence specific to each assessment and unique client need. As one piece of evidence, we conduct alignment studies to ensure that assessment content is appropriately aligned with state-mandated educational benchmarks. As a natural extension to our validity work, we assess the impact of test administration disruptions (such as computer malfunctions during testing) on the interpretability and validity of the resulting scores.
Utility
HumRRO works in parallel with testing contractors to perform true QA, as opposed to audits. Although some clients have requested us to conduct after-the-fact audits, to document errors that have occurred, our real-time QA processes discover and correct issues before they become errors on students’ score reports. We also routinely visit schools to investigate the utility of test information for educators.
Accuracy
Our psychometric experts are the nation’s premier QA authority. We use state-of-the-art Item Response Theory (IRT)-based methods to conduct item calibration, equating, and scaling analyses of high-stakes tests, and to conduct third-party score verification. We also
rigorously assess the accuracy of various scoring algorithms created by states to classify students and schools based on assessment data.
