Journal of Educational Measurement Special Issue on AI Edited by HumRRO’s David Dorsey and Hillary Michaels

Co-authored by Educational Testing Services’ Kadriye Ercikan and Daniel F. McCaffrey, the article reviews AI-based scoring methodologies in educational assessments. Ercikan and McCaffrey give recommendations for defining construct, task, and evidence models to guide task and assessment design that optimize the development and implementation of AI-based automated scoring of constructed response items and support the validity of inferences from and uses of scores.

Steve Ferrara of Cognia and Saed Qunbar of Workera argue that automated scoring engines should be transparent and construct relevant—at least as much as is presently feasible. The co-authors address the current limitations on evidence and validity arguments for scores from automated scoring engines from the points of view of the Standards for Educational and Psychological Testing. The authors address issues such as construct relevance, construct representation, fairness, and emerging principles in AI.

Performance Assessment Analytics’ Mark D. Shermis also discusses the challenge of automated essay scoring validity, particularly in the absence of a commonly held definition and theory of good writing. At best, he notes, the algorithms attempt to measure select attributes of writing and calibrate them against human ratings with the goal of accurate prediction of scores for new essays. Sometimes these attributes are based on the fundamentals of writing, such as fluency, but quite often they are based on locally developed rubrics that may be confounded with specific content coverage expectations. This lack of transparency makes it difficult to provide systematic evidence that machine scoring is assessing writing versus slices or correlates of writing performance.

Matthew S. Johnson, Xiang Liu, and Daniel F. McCaffrey of Educational Testing Services provide definitions of fairness and discuss ways in which modeling decisions and the predictive methods used in automated scoring can lead to biased, and thus unfair automated scores. They then introduce two simple methods for evaluating bias, evaluate their statistical properties through simulation, and apply these methods to a large-scale reading assessment.

A. Corinne Huggins-Manley of the University of Florida, and Brandon M. Booth and Sidney K. D’Mello of the University of Colorado argue that a more flexible approach to fairness arguments that occurs outside of and complementary to validity arguments is required to address many of the views on fairness that a set of assessment stakeholders may hold. The article focuses on two contributions: introducing the argument-based fairness approach to complement argument-based validity for both traditional and AI-enhanced assessments; and applying it in an illustrative AI assessment of perceived hireability in automated video interviews used to prescreen job candidates.