Advanced cloud-based data analysis is now available to everyone – both small and large organizations alike. HumRRO is among those companies taking advantage of these advancements.
September 25th, 2019 – Decades ago, only the largest and most sophisticated companies and institutions had access to “high performance computing” technologies. In practice, this meant that only large government agencies, high tech companies, and a small cadre of elite academic institutions harnessed the power of “supercomputers.” Using these supercomputers, such entities delivered new scientific discoveries and remarkable insights, in areas ranging from genomics and financial risk modeling to emerging uses such as machine learning and autonomous vehicles.
Within the behavioral sciences, measurement specialists also began conceiving of uses of advanced computing power. For example, the idea of “computational psychometrics” appeared, triggering calls to explore how advanced computing power could inform the development of mathematical models, intelligent learning applications, virtual assessments, and computer simulations of large-scale, complex data (Von Davier, Deonovic, Yudelson, Polyak, & Woo, 2019). Vendors also began offering Artificial Intelligence (AI) based methods as a practical means for developing, administering, and scoring operational assessments, which required increasing amounts of computing power (Putka & Dorsey, 2019). In the past, accessing added computing power meant simply upgrading to the latest laptop, desktop, or local server. However, as algorithmic complexity and dataset sizes increased, such solutions simply stopped being affordable or viable. Instead, power users turned to cloud-based systems, either public (e.g., Amazon, Google, Microsoft) or private, to gain a computational advantage.
Cloud-based systems offer a number of improvements, including: 1) scalability – such systems can provision as much computing capacity as needed, 2) dependability – most public cloud data centers are designed for 24/7 operations, without limitations such as power interruptions, and 3) convenience – analyses can be run and accessed from anywhere with Internet connectivity. As a practical example of why such cloud-based features matter—even for a relatively small company like HumRRO—we offer below a brief case example from our own recent work.
A HumRRO Case Example:
- A client asked us to assess a variety of different models for optimizing the validity of scores from an operational test battery. This analysis involved thousands of different model comparisons, based on Bayesian statistics.
- We used an Amazon Web Services (AWS) cloud analytic platform to estimate this large number of models. On traditional computing platforms (e.g., desktops, laptops), each model would take from one hour to multiple days to complete.
- As a result of conducting these analyses on cloud platforms, we were able to compress the total analysis time from approximately 9 months to 2 to 3 weeks. Thus, we reduced the required project time by about 90%.
- Along the way, we were able to collect useful data (runtime for each model, CPU usage metrics) that allowed us to further optimize our analyses.
At HumRRO, we see a bright future for leveraging cloud platforms for high performance computing, allowing us to better serve our customers while advancing science and practice. We are contemplating further applications and uses of these technologies, to include:
- Refining our procedures and solutions to allow for the use of cloud-based analysis in more projects,
- Automating the scaling of computational resources,
- Promoting easier entry-level access to cloud-based tools, infrastructures, and analysis methods,
- Exploring other “use cases,” possibly including the use of analytics as a backend engine for Web applications, creating advanced item banking applications, and exploring new forms of AI-assisted item writing and item analysis.
Anyone involved in modern psychological and educational measurement knows that assessment development and implementation involves a lot of data analysis. At its core, data analysis is computing. So as datasets grow and methods become more complex, analysts will likely turn to high performance computing to maintain efficiency, effectiveness, and competitiveness.
At HumRRO, our mission is to develop and apply state-of-the-art science and technology to improve the performance of individuals and teams within public and private organizations and educational institutions. In the very near future, state-of-the-art-science and technology is going to include an abundance of computing power. Consequently, we are excited to be working at the leading edge of these computing approaches.
References:
Putka, D. J. & Dorsey, D. W. (2019, April). A tour of I-O relevant AI/ML developments. Friday Seminar at the 34th Annual Society for Industrial and Organizational Psychology Conference. Washington, DC.
Von Davier, A. A., Deonovic, B. E., Yudelson, M., Polyak, S., & Woo, A. (2019). Computational psychometrics approach to holistic learning and assessment systems. In Frontiers in Education (Vol. 4, p. 69). Frontiers.
