Computer adaptive testing (CAT) has emerged as a dominant form of test administration to improve test security and obtain more precise estimates of examinee ability. Relative to static test forms where all items appear in a fixed order, CATs offer a dynamic assessment experience—adjusting and aligning content to an examinee’s ability level in real time until a pre-determined level of measurement precision is reached. Thus, CATs routinely result in a more secure, accurate, and efficient assessment process.
Moving from a fixed form to a CAT also frees test developers from having to build intact test forms. Instead, once developers build an item bank, the key question becomes, “Is the item bank sufficient to support CAT in a real testing scenario?” And if it is, “How do we configure an adaptive algorithm to make best use of the item bank to meet our test administration goals and objectives?”
To help answer those questions and support the potential benefits of using CAT with your testing program, HumRRO has developed an online tool to contribute to the exploration of CAT for measurement professionals. Known as Flight Plan—so named because it can help you explore CAT options on a real item bank before you launch a testing program—this new tool allows non-programmers to begin simulating whether their assessment program can support a CAT in a user-friendly, engaging way.
Complementing the consultation HumRRO routinely provides around assessment strategy, Flight Plan is a purely web-based application, making it accessible and easy to use yet also highly functional as it sits on a complex adaptive algorithm that pulls together some of the most recent advances in CAT science.
Traditional Barriers to Implementing CATs
The Flight Plan tool was designed to help test administrators overcome common obstacles to implementing CATs. For example, the process of tailoring ability estimates for each test taker often requires large item pools, including items that span a wide range of difficultly—especially when the testing population is expected to vary considerably on the targeted ability. A lack of item coverage at specific ability levels also increases the risk of item overexposure, which is exacerbated when those deficits occur at ability levels common among test takers.
Another challenge is, when a bank contains items with marginal psychometric properties, the CAT engine can take considerable time to reach an acceptable level of measurement precision.
Finally, some CATs adapt on both the examinee ability and on content constraints typically defined in a test blueprint. In these cases, the algorithm must pay attention to the psychometric properties of the items, and it also must use the metadata associated with the items to find the “best” items to meet both psychometric and test blueprint considerations.
Assessing the likelihood of such challenges occurring typically involves developing and running custom simulations via analytic packages such as R or SAS, requiring complex programming skills.
Enter Flight Plan
Because Flight Plan requires no programming, users can begin simulating the item bank parameters needed to support a CAT in three easy steps: 1) reading in a test blueprint, 2) uploading an item bank, and 3) configuring the CAT engine. The tool is also fully customizable, allowing users to specify constraints, such as item features or blueprint characteristics, that reflect the unique nature of their assessment program.
With the click of a button, Flight Plan uses these inputs to generate a variety of reports, including:
The tool also builds in flexibility through the output it generates. For example, users can specify how provisional ability estimates are generated, such as Owen’s sequential method or the Expected a posteriori (EAP) method, following each item response. They can also choose multiple ways to compute information as examinees progress through the test as well as a variety of exposure control options, including a random approach, Sympson-Hetter, and a constraint on a-parameters.
Considering these options helps decision-makers better understand the types of choices they must make prior to determining whether a CAT solution will work in their unique assessment context. While the current version of the tool focuses on forced-choice items, a version that also integrates multiple-choice content is in the works.
Charting a Data-Driven Assessment Strategy
HumRRO has a long history of drawing on the synergy created by our deep expertise in diverse assessment domains—including education, credentialing, talent management, and leadership and career development—to help our clients chart data-driven assessment strategies that reflect their unique goals, history, and context.
Tools like Flight Plan enhance this process by providing concrete, tangible ways to explore various options. We are opening up access to this tool in the hope that navigating it sparks creative thinking and reflection about whether your assessment program is ready for a CAT-based solution. And if the simulation results indicate that it’s not prudent to launch CAT now, simply interacting with Flight Plan can help chart the course of how to get there.
We invite you to access this new CAT simulation tool and give it a test drive. Be sure to check out the tutorial, background information, and documentation tabs, and share your questions and reactions by emailing [email protected] or contacting Harold Doran.
