[fusion_builder_container hundred_percent=”no” hundred_percent_height=”no” hundred_percent_height_scroll=”no” hundred_percent_height_center_content=”yes” equal_height_columns=”no” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” status=”published” background_position=”center center” background_repeat=”no-repeat” fade=”no” background_parallax=”none” enable_mobile=”no” parallax_speed=”0.3″ video_aspect_ratio=”16:9″ video_loop=”yes” video_mute=”yes” border_style=”solid” type=”legacy”][fusion_builder_row][fusion_builder_column type=”1_1″ layout=”1_1″ center_content=”no” target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” background_position=”left top” background_repeat=”no-repeat” hover_type=”none” border_style=”solid” border_position=”all” animation_direction=”left” animation_speed=”0.3″ last=”true” border_sizes_top=”0″ border_sizes_bottom=”0″ border_sizes_left=”0″ border_sizes_right=”0″ first=”true” background_blend_mode=”overlay” min_height=”” link=””][fusion_text hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” animation_direction=”left” animation_speed=”0.3″]
HumRRO Vice President, David Dorsey, Ph.D., and HumRRO Director of Education, Hillary Michaels, Ph.D., leveraged their unique expertise and experience to edit the Fall 2022 Special Issue of the Journal of Educational Measurement.
The issue, Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment, discusses artificial intelligence (AI)-enabled assessments—tools used in organizational and workforce development and, increasingly, in education that have intrigued scientists and practitioners. While AI presents the lure of efficiency and effectiveness, Dorsey and Michaels caution that users must maintain high standards for validity and fairness, which can be challenging in the complicated world of AI-enabled assessment.
With that in mind, Dorsey and Michaels curated a series of original research articles from practitioners and researchers that tackle the challenges of AI validity arguments in educational assessment domains. Specifically, the articles explore how AI impacts Evidence-Centered Design (ECD) and development from assessment concept and coding to scoring and reporting. They also present information on ways to ensure that bias is not built into these systems. Lastly, they discuss future horizons, many that are almost here, for maximizing what AI offers while minimizing negative effects on test takers and programs.[/fusion_text][/fusion_builder_column][/fusion_builder_row][/fusion_builder_container][fusion_builder_container hundred_percent=”no” hundred_percent_height=”no” hundred_percent_height_scroll=”no” hundred_percent_height_center_content=”yes” equal_height_columns=”no” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” status=”published” background_image=”https://www.humrro.org/dev/corpsite_wp/wp-content/uploads/2019/02/humrro-cubes-pattern-lightest.png” background_position=”center center” background_repeat=”repeat” fade=”no” background_parallax=”up” enable_mobile=”no” parallax_speed=”0.3″ video_aspect_ratio=”16:9″ video_loop=”yes” video_mute=”yes” border_style=”solid” margin_bottom=”40px” type=”legacy”][fusion_builder_row][fusion_builder_column type=”1_1″ layout=”1_1″ center_content=”no” target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” background_position=”left top” background_repeat=”repeat” hover_type=”none” border_style=”solid” border_position=”all” padding_top=”40px” padding_right=”40px” padding_bottom=”20px” padding_left=”40px” animation_direction=”left” animation_speed=”0.3″ last=”true” border_sizes_top=”0″ border_sizes_bottom=”0″ border_sizes_left=”0″ border_sizes_right=”0″ first=”true” background_blend_mode=”overlay” min_height=”” link=””][fusion_text hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” animation_direction=”left” animation_speed=”0.3″]
It was an honor to work with some of the top educational measurement specialists in the field to wrestle with tough issues around AI-enabled assessments and validity arguments. This topic will surely remain important for the field going forward, so our hope was to begin a conversation and start an exploration of related topics.
— Hillary Michaels, Ph.D.
[/fusion_text][/fusion_builder_column][/fusion_builder_row][/fusion_builder_container][fusion_builder_container type=”flex” hundred_percent=”no” hundred_percent_height=”no” hundred_percent_height_scroll=”no” align_content=”stretch” flex_align_items=”flex-start” flex_justify_content=”flex-start” hundred_percent_height_center_content=”yes” equal_height_columns=”no” container_tag=”div” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” status=”published” border_style=”solid” box_shadow=”no” box_shadow_blur=”0″ box_shadow_spread=”0″ gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ background_position=”center center” background_repeat=”no-repeat” fade=”no” background_parallax=”none” enable_mobile=”no” parallax_speed=”0.3″ background_blend_mode=”none” video_aspect_ratio=”16:9″ video_loop=”yes” video_mute=”yes” pattern_bg=”none” pattern_bg_style=”default” pattern_bg_opacity=”100″ pattern_bg_blend_mode=”normal” mask_bg=”none” mask_bg_style=”default” mask_bg_opacity=”100″ mask_bg_transform=”left” mask_bg_blend_mode=”normal” absolute=”off” absolute_devices=”small,medium,large” sticky=”off” sticky_devices=”small-visibility,medium-visibility,large-visibility” sticky_transition_offset=”0″ scroll_offset=”0″ animation_direction=”left” animation_speed=”0.3″ filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″][fusion_builder_row][fusion_builder_column type=”1_1″ layout=”1_1″ align_self=”auto” content_layout=”column” align_content=”flex-start” valign_content=”flex-start” content_wrap=”wrap” center_content=”no” column_tag=”div” target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” order_medium=”0″ order_small=”0″ hover_type=”none” border_style=”solid” box_shadow=”no” box_shadow_blur=”0″ box_shadow_spread=”0″ z_index_subgroup=”regular” background_type=”single” gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ lazy_load=”avada” background_position=”left top” background_repeat=”no-repeat” background_blend_mode=”none” sticky=”off” sticky_devices=”small-visibility,medium-visibility,large-visibility” filter_type=”regular” filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″ transform_type=”regular” transform_scale_x=”1″ transform_scale_y=”1″ transform_translate_x=”0″ transform_translate_y=”0″ transform_rotate=”0″ transform_skew_x=”0″ transform_skew_y=”0″ transform_scale_x_hover=”1″ transform_scale_y_hover=”1″ transform_translate_x_hover=”0″ transform_translate_y_hover=”0″ transform_rotate_hover=”0″ transform_skew_x_hover=”0″ transform_skew_y_hover=”0″ animation_direction=”left” animation_speed=”0.3″ last=”true” border_position=”all” first=”true” min_height=”” link=””][fusion_title title_type=”text” rotation_effect=”bounceIn” display_time=”1200″ highlight_effect=”circle” loop_animation=”off” highlight_width=”9″ highlight_top_margin=”0″ title_link=”off” link_target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” content_align=”left” size=”3″ text_color=”var(–awb-color6)” text_shadow=”no” text_shadow_blur=”0″ gradient_font=”no” gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ style_type=”default” animation_direction=”left” animation_speed=”0.3″ sep_color=”var(–awb-color7)” margin_bottom=”35px”] The JEM Special Issue includes the following articles: [/fusion_title][fusion_content_boxes layout=”icon-on-side” columns=”1″ heading_size=”2″ iconspin=”no” icon_align=”left” animation_direction=”left” animation_speed=”0.3″ hide_on_mobile=”small-visibility,medium-visibility,large-visibility” title_color=”var(–awb-color5)” body_color=”var(–awb-custom13)” icon=”fa-file-alt fas” link_type=”text” circlecolor=”var(–awb-color4)” circlebordersize=”0″ margin_bottom=”0″ title_size=”19px”][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Optimizing Implementation of Artificial-Intelligence-Based Automated Scoring: An Evidence Centered Design Approach for Designing Assessments for AI-Based Scoring.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12332″ link_target=”_blank”]
Co-authored by Educational Testing Services’ Kadriye Ercikan and Daniel F. McCaffrey, the article reviews AI-based scoring methodologies in educational assessments. Ercikan and McCaffrey give recommendations for defining construct, task, and evidence models to guide task and assessment design that optimize the development and implementation of AI-based automated scoring of constructed response items and support the validity of inferences from and uses of scores.
[/fusion_content_box][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Validity Arguments for AI-Based Automated Scores: Essay Scoring as an Illustration.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12333″ link_target=”_blank”]
Steve Ferrara of Cognia and Saed Qunbar of Workera argue that automated scoring engines should be transparent and construct relevant—at least as much as is presently feasible. The co-authors address the current limitations on evidence and validity arguments for scores from automated scoring engines from the points of view of the Standards for Educational and Psychological Testing. The authors address issues such as construct relevance, construct representation, fairness, and emerging principles in AI.
[/fusion_content_box][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Anchoring Validity Evidence for Automated Essay Scoring.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12336″ link_target=”_blank”]
Performance Assessment Analytics’ Mark D. Shermis also discusses the challenge of automated essay scoring validity, particularly in the absence of a commonly held definition and theory of good writing. At best, he notes, the algorithms attempt to measure select attributes of writing and calibrate them against human ratings with the goal of accurate prediction of scores for new essays. Sometimes these attributes are based on the fundamentals of writing, such as fluency, but quite often they are based on locally developed rubrics that may be confounded with specific content coverage expectations. This lack of transparency makes it difficult to provide systematic evidence that machine scoring is assessing writing versus slices or correlates of writing performance.
[/fusion_content_box][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Psychometric Methods to Evaluate Measurement and Algorithmic Bias in Automated Scoring.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12335″ link_target=”_blank”]
Matthew S. Johnson, Xiang Liu, and Daniel F. McCaffrey of Educational Testing Services provide definitions of fairness and discuss ways in which modeling decisions and the predictive methods used in automated scoring can lead to biased, and thus unfair automated scores. They then introduce two simple methods for evaluating bias, evaluate their statistical properties through simulation, and apply these methods to a large-scale reading assessment.
[/fusion_content_box][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Toward Argument-Based Fairness with an Application to AI-Enhanced Educational Assessments.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12334″ link_target=”_blank”]
A. Corinne Huggins-Manley of the University of Florida, and Brandon M. Booth and Sidney K. D’Mello of the University of Colorado argue that a more flexible approach to fairness arguments that occurs outside of and complementary to validity arguments is required to address many of the views on fairness that a set of assessment stakeholders may hold. The article focuses on two contributions: introducing the argument-based fairness approach to complement argument-based validity for both traditional and AI-enhanced assessments; and applying it in an illustrative AI assessment of perceived hireability in automated video interviews used to prescreen job candidates.
[/fusion_content_box][fusion_content_box linktext=”Read Article” animation_direction=”left” animation_speed=”0.3″ title=”Validity Arguments Meet Artificial Intelligence in Innovative Educational Assessment: A Discussion and Look Forward.” link=”https://onlinelibrary.wiley.com/doi/10.1111/jedm.12330″ link_target=”_blank”]
Dorsey and Michaels conclude the special issue with an overall discussion and point to future emerging trends in AI that might shape our approach to validity and building validity arguments.
[/fusion_content_box][/fusion_content_boxes][/fusion_builder_column][fusion_builder_column type=”1_1″ layout=”1_1″ align_self=”auto” content_layout=”column” align_content=”flex-start” valign_content=”flex-start” content_wrap=”wrap” center_content=”no” column_tag=”div” target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” order_medium=”0″ order_small=”0″ hover_type=”none” border_style=”solid” box_shadow=”no” box_shadow_blur=”0″ box_shadow_spread=”0″ z_index_subgroup=”regular” background_type=”single” gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ lazy_load=”avada” background_position=”left top” background_repeat=”no-repeat” background_blend_mode=”none” sticky=”off” sticky_devices=”small-visibility,medium-visibility,large-visibility” filter_type=”regular” filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″ transform_type=”regular” transform_scale_x=”1″ transform_scale_y=”1″ transform_translate_x=”0″ transform_translate_y=”0″ transform_rotate=”0″ transform_skew_x=”0″ transform_skew_y=”0″ transform_scale_x_hover=”1″ transform_scale_y_hover=”1″ transform_translate_x_hover=”0″ transform_translate_y_hover=”0″ transform_rotate_hover=”0″ transform_skew_x_hover=”0″ transform_skew_y_hover=”0″ animation_direction=”left” animation_speed=”0.3″ last=”true” border_position=”all” first=”true” min_height=”” link=””][fusion_text hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” animation_direction=”left” animation_speed=”0.3″ margin_top=”-10px”]
“For those interested in AI-enabled assessments—practitioners and researchers alike—it is critical that we keep validity arguments front and center,” said Dorsey. “For this special issue, we sought to find contributions that showed a way forward for ensuring that our ability to produce robust validity arguments keeps pace with the ever-increasing power of AI.”
[/fusion_text][/fusion_builder_column][/fusion_builder_row][/fusion_builder_container][fusion_builder_container type=”legacy” hundred_percent=”no” hundred_percent_height=”no” hundred_percent_height_scroll=”no” align_content=”stretch” flex_align_items=”flex-start” flex_justify_content=”flex-start” hundred_percent_height_center_content=”yes” equal_height_columns=”no” container_tag=”div” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” status=”published” class=”border-radius-10 post-contact-card” border_sizes_top=”0px” border_sizes_right=”0px” border_sizes_bottom=”0px” border_sizes_left=”0px” border_style=”solid” margin_bottom=”65px” padding_top=”50px” padding_right=”30px” padding_bottom=”50px” padding_left=”30px” box_shadow=”no” box_shadow_blur=”0″ box_shadow_spread=”0″ gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ background_color=”#003E74″ background_image=”https://www.humrro.org/corpsite/wp-content/uploads/2020/08/updated-blue-background.jpg” background_position=”center center” background_repeat=”no-repeat” fade=”no” background_parallax=”none” enable_mobile=”no” parallax_speed=”0.3″ background_blend_mode=”none” video_aspect_ratio=”16:9″ video_loop=”yes” video_mute=”yes” absolute=”off” absolute_devices=”small,medium,large” sticky=”off” sticky_devices=”small-visibility,medium-visibility,large-visibility” sticky_transition_offset=”0″ scroll_offset=”0″ animation_direction=”left” animation_speed=”0.3″ filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″][fusion_builder_row][fusion_builder_column type=”1_1″ layout=”1_1″ center_content=”no” target=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” background_position=”left top” background_repeat=”no-repeat” hover_type=”none” border_style=”solid” border_position=”all” margin_bottom=”0″ animation_direction=”left” animation_speed=”0.3″ last=”true” border_sizes_top=”0″ border_sizes_bottom=”0″ border_sizes_left=”0″ border_sizes_right=”0″ first=”true” background_blend_mode=”overlay” min_height=”” link=””][fusion_builder_row_inner][fusion_builder_column_inner type=”1_3″ layout=”1_3″ center_content=”no” hover_type=”none” target=”_self” hide_on_mobile=”medium-visibility,large-visibility” border_style=”solid” border_position=”all” box_shadow=”no” box_shadow_blur=”0″ box_shadow_spread=”0″ margin_bottom=”0″ background_type=”single” gradient_start_position=”0″ gradient_end_position=”100″ gradient_type=”linear” radial_direction=”center center” linear_angle=”180″ background_position=”left top” background_repeat=”no-repeat” background_blend_mode=”none” animation_direction=”left” animation_speed=”0.3″ filter_type=”regular” filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″ last=”false” border_sizes_top=”0″ border_sizes_bottom=”0″ border_sizes_left=”0″ border_sizes_right=”0″ first=”true” min_height=”” link=””][fusion_imageframe image_id=”5129|full” custom_aspect_ratio=”100″ lightbox=”no” link=”https://www.humrro.org/corpsite/profile/rod-mccloy/” linktarget=”_self” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” max_width=”170px” align_medium=”none” align_small=”none” align=”center” style_type=”dropshadow” hover_type=”none” borderradius=”50%” caption_style=”off” caption_align_medium=”none” caption_align_small=”none” caption_align=”none” caption_title_tag=”2″ animation_direction=”left” animation_speed=”0.3″ filter_hue=”0″ filter_saturation=”100″ filter_brightness=”100″ filter_contrast=”100″ filter_invert=”0″ filter_sepia=”0″ filter_opacity=”100″ filter_blur=”0″ filter_hue_hover=”0″ filter_saturation_hover=”100″ filter_brightness_hover=”100″ filter_contrast_hover=”100″ filter_invert_hover=”0″ filter_sepia_hover=”0″ filter_opacity_hover=”100″ filter_blur_hover=”0″]https://www.humrro.org/corpsite/wp-content/uploads/2021/07/DavidDorsey_350x350.jpg[/fusion_imageframe][/fusion_builder_column_inner][fusion_builder_column_inner type=”2_3″ layout=”3_4″ center_content=”no” hover_type=”none” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” background_position=”left top” background_repeat=”no-repeat” border_style=”solid” margin_bottom=”0″ animation_direction=”left” animation_speed=”0.3″ last=”true” border_position=”all” border_sizes_top=”0″ border_sizes_bottom=”0″ border_sizes_left=”0″ border_sizes_right=”0″ first=”false” background_blend_mode=”overlay” min_height=”” link=””][fusion_text hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” animation_direction=”left” animation_speed=”0.3″]
For more information, contact:
David Dorsey, Ph.D.
Vice President
[/fusion_text][fusion_button link=”https://www.humrro.org/corpsite/profile/david-dorsey/” target=”_self” alignment=”left” hide_on_mobile=”small-visibility,medium-visibility,large-visibility” sticky_display=”normal,sticky” color=”custom” button_gradient_top_color=”#4c9c2e” button_gradient_bottom_color=”#4c9c2e” button_gradient_top_color_hover=”#96c121″ button_gradient_bottom_color_hover=”#96c121″ border_radius_top_left=”25px” border_radius_top_right=”25px” border_radius_bottom_right=”25px” border_radius_bottom_left=”25px” size=”medium” text_transform=”none” stretch=”default” icon=”fa-angle-right fas” icon_position=”right” icon_divider=”no” animation_direction=”left” animation_speed=”0.3″]Contact David[/fusion_button][/fusion_builder_column_inner][/fusion_builder_row_inner][/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]
