The Challenge
Human capital (HC) programs represent a significant investment in an organization’s resources. However, evaluating the programs’ impact on organizational performance can be challenging.
HumRRO completed a large-scale evaluation for the U.S. Office of Personnel Management (OPM), assessing the impact of its HC Program on Governmentwide human capital management (HCM).
This evaluation was intended to meet OPM’s need for program evaluation information and the U.S. Office of Management and Budget’s (OMB’s) Program Assessment Rating Tool (PART) requirements for program impact evaluations.
What We Did
Program Evaluation Questionnaire
HumRRO developed and administered questionnaires to HCM employees in agencies throughout the Federal government. The questionnaires asked agency HCM staff members about their use of and satisfaction with the OPM HC Program’s products and services. HCM staff were also asked to rate the Program’s overall effectiveness and impact on their agency’s HCM.
Program Impact Analyses
To assess the HC Program’s impact on agencies’ HCM implementation results and organizational performance, the questionnaire responses were combined with organization-level data on (a) agencies’ HC implementation results from the HC portion of the President’s Management Agenda (PMA) Scorecard, (b) employees’ perceptions of their agencies’ HCM from the Federal Human Capital Survey (FHCS), and (c) agencies’ organizational performance results from their Performance and Accountability Reports (PARs) and PART scores. The outcome data covered the period from 2002, when the HC Program was first launched, through 2007.
The HC Program’s impact was then analyzed using state-of-the-art methods (e.g., multi-level analysis, growth curve modeling, and hierarchical linear modeling). The results of these analyses represented an enhancement over previous evaluations, because they involved data from multiple sources and were longitudinal, rather than cross-sectional, in nature.
Framework for Evaluating the HC Program on Government-wide Human Capital Management
What We Found
- Familiarity with the HC Program’s 46 products and services was very high.
- The HC Program was rated as more effective in 2005-2007 than it was 3 to 4 years earlier.
- On average, the percent of employees satisfied with the Program’s 46 products and services was much higher than the percent dissatisfied.
- Agencies that most frequently used the Program’s products and services implemented HCM more effectively than agencies that used them less frequently.
- Agencies with more effective HCM implementation were perceived more favorably by employees, according to the results of the 2004 and 2006 FHCS.
- Agencies with favorable FHCS results achieved stronger organizational performance, according to both their self-reported PARs and OMB’s PART ratings of their performance.
