HR and talent management professionals looking to find the right hiring solution face a dizzying array of choices: What competencies should we assess? What type of assessment will work best? How should the assessments be administered? The dozens of off-the-shelf (OTS) assessments currently flooding the marketplace—everything from personality inventories to cognitive ability tests and simulations—don’t make these choices any easier.
Despite the best of intentions, organizations may not even consider what is arguably the most critical choice of all—whether to purchase an existing OTS solution or develop a custom assessment. Here, I define custom assessment as one built from the ground up to meet a client’s unique needs, rather than an existing OTS tool that has been “customized” through surface-level changes. In my experience, many organizations reach for an OTS solution without carefully considering whether a custom assessment that reflects what’s distinct and different about their hiring needs might be a better choice.
To be fair, OTS solutions have some clear strengths. They can be a good choice when an organization wants to measure generic competencies among applicants for relatively less complex jobs, such as sales, clerical, or administrative occupations. Because OTS assessments are typically used across a wide variety of organizations, OTS vendors can provide their clients with normative data and existing validity evidence. And the lack of development costs can be very appealing.
However compelling they seem, these strengths should be viewed in light of several potential drawbacks associated with OTS assessments:
- They often involve per-candidate fees, which can really add up in large-volume hiring situations. For example, a $15 per candidate-fee can easily top $75,000 if the job opening generates more than 5,000 applicants.
- Precisely because they are designed apply across many different industries and occupations, OTS assessments don’t capture what makes a job truly unique—they fail to capture both job-specific knowledge and skills and the rich, contextualized way that more generic competencies (problem solving, conflict management, decisiveness) are applied within a specific job. To put a fine point on it, “problem solving” means something very different to school principals, accountants, and Federal law enforcement officers. Ideally, an assessment solution should reflect that.
- Clients using OTS assessments must critically evaluate how the assessment has been deployed and what type of exposure the items, test materials, and scoring rubrics have experienced, especially when the assessment has been administered in an unproctored way.
Developing a custom assessment is not the silver-bullet solution to all of these challenges. Like OTS tools, they have some drawbacks. Perhaps most obviously, they involve up-front development costs and time. If the assessment targets job knowledge and includes job-specific content and scenarios, subject matter input is needed to develop them. However, their benefits can drastically outweigh these costs:
- Custom assessments capture what really makes a job unique—not only in the competencies/skills required, but in how they are expressed on the job. That makes them ideal for challenging, complex, mission-critical jobs. What better way is to there to test an applicant’s judgement and decision making skills than by having them complete a scenario-based assessment that reflects the realities of their agency and the specific personnel-, business-, and resource-related challenges they face each and every day? In my view, tweaking an OTS with surface-level changes that reflect a customer’s branding, job titles, and industry doesn’t go more than “skin deep,” and fails to capture the true essence of mission critical jobs.
- Applicants are likely to experience custom assessments as engaging and very job relevant because they can mirror the target job with a high degree of fidelity.
- Clients own their custom-built assessments, so there are no licensing and per-candidate fees. This can help them quickly recoup development costs, particularly in the case of large volume hiring.
- Because they are not harnessed to a specific administration method or platform, custom assessments can allow for a variety of flexible administration options—like the assessment itself, the administration method can be “built to suit.”
It’s clear that the choice between OTS and custom solutions is not an easy one, and that there are many factors to consider. But, before reaching for an OTS that seems like it will do the trick, why not pause and consider whether a custom solution might be a better fit?
