Two HumRRO scientists recently partnered to present a well-received session focused on “Transforming High-Stakes Testing” at the 2020 Beyond Multiple Choice (BMC) conference, held virtually in early November. Consistent with the conference theme, Beth Bynum, Ph.D., and Andrea Sinclair, Ph.D., described assessment innovations HumRRO is pioneering that move beyond the common reliance on multiple choice formats in high-stakes testing contexts.
Bynum began the session by highlighting the benefits of scenario-based assessments such as Situational Judgment Tests (SJTs). These benefits include the potential to measure skills and abilities, such as judgment, decision making and interpersonal skills, with greater fidelity than multiple-choice methods allow, plus encouraging criterion-related validity results and relatively small to moderate subgroup differences. Drawing from HumRRO’s experience deploying SJTs in high-stakes selection and credentialing contexts, she went on to describe several lessons learned, which range from when and how to best involve SMEs in the development process to factors to consider when identifying optimal response formats, scoring rubrics, and equating strategies.
Delving into the exciting world of game-based assessments, Sinclair described a multiyear effort to create game-like performance tests to measure conscientiousness. For example, one game measuring perseverance involved a word search that shuffled the grid of letters once users found the words, forcing them to find the same words again. “We were curious to see if individuals would persevere in finding words in the puzzle after a random shuffle or if they would skip to a new puzzle,” Sinclair explained.
After showing examples of these game-based assessments and sharing lessons learned, she offered suggestions about how future work in this arena might generate more refined modeling techniques as well as rigorously constructed score composites and pattern-based scoring strategies.
Now in its third year, the Beyond Multiple Choice Conference is quickly becoming a unique, not-to-miss event where experts from across the high-stakes assessment field convene to learn about and share cutting-edge innovations. “I really enjoyed the diversity of perspectives represented at this conference,” Sinclair said, “and I look forward to future BMC events.”
