HumRRO President and CEO Suzanne Tsacoumis, Ph.D., delivered a keynote address at the International Personnel Assessment Council Annual Conference on Monday, sharing her insights on technological advances in high-stakes assessments and their applications.
“Our profession has well-established guidelines for developing valid assessments for use in high-stake situations,” she told attendees. “However, technological advances, such as rich media, automated scoring, and artificial intelligence (AI), have introduced new approaches that weren’t even on the radar screen when the industry settled on best practices.”
While technology-rich assessments can enhance efficiency and improve user engagement, Tsacoumis stressed the importance of validity. She demonstrated examples of HumRRO’s rich media simulations used in recruitment, and selection and promotion.
In addition, HumRRO is engaged in cutting-edge research developing assessments using AI and machine learning without compromising item quality. For example, natural language processing (NLP) can be applied to job analysis by predicting the important knowledge, skills, abilities, and other characteristics (KSAO) from job description text. “By using NLP for automated item generation, we can train a machine learning model on a large amount of human-written text and construct text by determining the most plausible terms following pre-existing text,” Tsacoumis noted. Research conducted by her HumRRO colleagues suggests that study participants often cannot distinguish an item written based on an NLP model from one written by a human.
Tsacoumis concluded her address with the following considerations when evaluating AI-based assessments:
- Resulting scores should be considered fair and unbiased.
- Content should be job-related.
- Scores should predict future job performance or other relevant outcomes.
- Scores should be reliable and related to the job.
- Development process should be well documented.
“Although there is great value in embracing the power of technology advances, it is critical we continue to adhere to professional and legal guidelines and standards. We must find the right balance between technology and human input,” Tsacoumis said.
