Validity is a pretty dry topic but it is consistently the subject of the most frequently asked questions about the use of Prevue Assessments. It has therefore earned feature article status. If you can persevere through to the final part of this article you will find the offer of a reward for your dedication.

Validity questions generally break down into the following three topics:

• What is validity?
• Why is it important?
• How is it measured?

What is Validity? (this is the easy part)

In the simplest terms validity confirms an assessment tool or process “measures what it purports to measure”, “is job relevant” and “does assist in making more effective decisions”.

Keep in mind that the reference to assessment tools or processes doesn’t just refer to Prevue Assessments. The need to confirm validity applies to any and all tools and procedures that are used in assessing individuals including the collection of information from job applicants, content of screening and interview questions, any kind of testing used in the hiring process and even background and reference checks.

Why is it Important?

There are two basic reasons why employers should be concerned about validity:

1. First you don’t want to be wasting your time and money using assessment tools or processes that are not effective.  Not only do you want to know they are effective, you should also want to determine their degree of effectiveness – in other words, your return on investment (ROI) on the time and money invested in those tool and processes. Proving validity and ROI are virtually the same process.
2. Second (first for some) the law requires that you be able to demonstrate the assessment tools or processes used in the hiring process are valid.  In the USA this is contained in the Civil Rights Act, EEOC Guidelines and host of other federal, state and local laws.  In Canada, Australia, the UK and other common law jurisdictions the requirement is contained in a wide range of human rights and employment legislation at all levels of government.

How is it Measured? (this is the heavy going part)

Evidence of Validity should be available from the following two sources:

• First, proof of validity should be provided by the supplier of any assessment tool or process you utilize.  For the Prevue Assessments this is comprised of the Prevue Technical Manual together with the Prevue Technical Bulletin, “Measures of Reliability and Validity”.  Copies are available from your authorized Prevue Distributor or by email request to support@prevueassessments.com
• Second is confirmation of validity through validation studies conducted by the user of any assessment tool or process, sometimes referred to as internal validity.  This is what the law expects the user to demonstrate in the course of responding to a complaint filed under any of the legislation noted above or in civil proceedings that involve employment issues.

The specific degree or level of validity is measured by validity coefficients reported as a number between 0 and 1.00. The US Dept. of Labor’s guide on Testing and Assessment confirms that validity coefficients of .21 to .35 are typical of most assessment tools with scores over .35 being very beneficial. You can see in the above-noted Technical Bulletin that the validity coefficients for the Prevue Assessments generally exceed .35.

Validity and ROI are confirmed by conducting one or more of the following types of validation studies:

• Criterion-related validation to demonstrate the relationship between assessment performance and job performance that confirms either concurrent validity or predictive validity;
• Content-related validation to confirm the assessment is relevant to and measures requirements or qualifications for the job; and
• Construct-related validation to exhibit the assessment does measure the constructs it purports to measure and those constructs are important to job performance.

You Can Do It – We Can Help (finally, the reward part – more work for all of us)

The Prevue Technical Manual confirms the Prevue Assessments have been validated with all three types of studies outlined above.  The obligation on employers to confirm Internal Validity we know can be both time consuming and expensive.  Prevue has therefore developed the “Prevue Internal Validation/ROI Program” to assist Prevue users to confirm the internal validity and/or the ROI of their use of Prevue Assessments.  You can obtain an outline of the Prevue Internal Validation/ROI Program from your authorized Prevue Distributor or by email request to support@prevueassessments.com .

For those of you masochists who may be interested in finding out more about this subject we would refer you to chapter 3 of the “Employers Guide to Testing and Assessment” published by the US department of Labor, Employment and Training Administration, at www.onetcenter.org/dl_files/empTestAsse.pdf .

Written By: Ken Danderer July 10, 2010

• Assessments are job suitability tests that are given to job applicants either online or in pencil and paper format before they are hired or promoted.  Assessments are also used for coaching, training and team building activities.
• To be most effective, the assessment should be designed to specifically test for qualities or attitudes that are important for the job by comparing the applicant to a benchmark of the knowledge, skills, abilities and behavior required in the job.
• Tests are based on valid principles of psychology and should be designed and developed by occupational psychologist with expertise in job profiling and test design. Such persons are referred to as Psychometrists.

Ipsative Scores & Normative Scores Explained:

Ipsative means “of the self, and is used in psychology as in the phrase “ipsative measure” to indicate a specific type of measure in which respondents compare two or more options and pick the one which is most preferred , or in other words make a forced choice.  This is contrasted with measures that use Likert-type scales, in which respondents choose the score (e.g. 1 to 5) which best represents the degree to which they agree with a given statement – also referred to in this article as Normative testing.

Ipsative scores may be found in any test where there are forced choices and where each choice is scored.  This means that at the end of the test the scores can represent only the relative strength of that characteristic in that individual.  There must be negative correlations between some of the scores simply as an artifact of the scoring system, in that if one chooses one response as the best that excludes the others.  This scoring system fails to measure the absolute strength of the characteristic.

Q&A

Q: Could you send me a comparison analysis of DiSC vs. Prevue?

A: Not all DiSC products are alike so you need to look at the DiSC product of interest to you.  As a general rule, most DiSC products are not suitable for hiring or other higher stakes decisions because they are Ipsative rather than Normative instruments – the technical manual for the better DiSC products will usually confirm this limitation.  Essentially most DiSC products are feel good assessments that will not offend anyone’s sensibilities when used for team building exercises where assessment results are shared.  Prevue is designed to life the iceberg out of the water so decision makers can really understand what they are dealing with in terms of human resource management.  Prevue is therefore more difficult to work with in Coaching or Training applications but provides reliable information for high stakes decisions.

It is clear that normative tests are far superior to ipsative tests as precise measures of psychological characteristics.  Ipsative scores are only suitable as a basis of discussion.   Since however, it is perfectly possible to use normative tests as a basis for discussion and have, in addition, scores suitable for statistical analysis, there seems no reason to use ipsatively scored tests and they are not recommended.

• Assessments are job suitability tests that are given to job applicants either online or in pencil and paper format before they are hired or promoted.  Assessments are also used for coaching, training and team building activities.
• To be most effective, the assessment should be designed to specifically test for qualities or attitudes that are important for the job by comparing the applicant to a benchmark of the knowledge, skills, abilities and behavior required in the job.
• Tests are based on valid principles of psychology and should be designed and developed by occupational psychologist with expertise in job profiling and test design. Such persons are referred to as Psychometrists.

Q&A

1. What kinds of small businesses find these assessments most helpful? (e.g. size of company, industries)

A: The heaviest use of testing is for the following types of jobs;

• Managerial/supervisory at the entry and mid levels of management
• Technical professionals like programmers
• Sales
• Customer Service
• Call Centres

2. What is the average cost of a new hire for a small business?  This would be based on research you might have done in support of your service.

A:  Cost to hire data is generally only collected by larger companies that have a recruiting department or by recruiting firms. Smaller companies tend to focus more on cost of turnover. In other words, what is the cost of a bad hire? I’ve commented on this subject in question # 5.

3. What problems or challenges do assessments resolve for small business?

A:  Testing is generally focused on two aspects of job performance. The first and most simple is “Skills testing” to measure knowledge on subjects like software programs. The second and somewhat more mysterious is “Job Fit testing” to match applicants to the demands or preferred behaviours of specific positions, usually measured by abilities, interests and personality tests (the Prevue Assessments are Job Fit Assessments).

There is an old saying that “people are hired on skills and fired on fit”. Skills testing assure the candidate actually has the knowledge required for a position while Job Fit testing provides information on whether the candidate will be comfortable addressing the demands of the position in terms of things like ability to assimilate new information quickly, conscientiousness, interest in working with people, capacity to deal with stress, etc.

4. How expensive and time-consuming is it to get an assessment prepared?

A:  With the advent of online testing in the 1990’s the time required to administer a test to job applicants has been reduced to minutes. The cost of testing is usually dependent on the volume of tests used by an employer, and depending on the types of tests that are used and the amount of support required or requested by the employer.  Prevue Distributors may also charge a consulting fee.

5. How many hires or how long do you expect it would take for the assessments to pay for themselves in saved HR costs?

A:  We expect our Prevue Assessments will more than pay for themselves from the first hire made with the use of assessments. Most turnover is caused by lack of job fit – things such as the inability to close sales, reluctance to make sales calls, failure to make timely decisions, failure to complete and deliver reports, failure to show up to work on time. Information from valid and reliable job fit assessments can provide insight into the characteristics that cause these behaviours.

Cost of turnover has been proven to be at least 30% of an employee’s annual salary and anywhere up to 100% of that annual salary, depending on the extent of recruiting and training costs that are related to the position. So if you can reduce turnover by even as little as 5 to 10%, the cost of testing the best 2 or 3 applicants for a position is pretty cheap relative to the return – and that ignores the cost and interruption to the performance of the manager and fellow workers.

6. How do you think the assessments would help an immigrant business owner (or someone whose first language is not English) to improve their business?

A: This is tough one. Although the concept of typing people based on intelligence, personality and interests goes back almost as far as recorded history, modern business testing was only borne in WW II through the efforts of the English and the British to place people in military positions where they would be most effective. Thus most test development has, until recently, been based upon a white English speaking population. That situation and the use of assessments have however undergone significant change over the last 25 years with the introduction of testing in virtually every corner of the globe. Thus, we at Prevue Assessments have developed and validated versions of our Prevue Assessments in French, Spanish, Portuguese, Dutch and Arabic and are scheduled to complete translations to German, Russian, Mandarin, Polish and Japanese.

Testing is definitely finding a market in all cultures and languages so there is no reason why immigrant business owners should not take advantage of these tools in whatever language they or their job applicants prefer.

1.         “While selecting the best people is an important goal in itself, a parallel goal of no less importance is screening out undesirable candidates. The consequences of choosing the wrong people are substantial, as they lead to increased turnover rates, recruitment costs, and training expenses, along with lost productivity and decreases in morale. The high costs associated with replacing poorly performing individuals make it all the more important to identify and select the best performers in the first place.”

2.         “…. there are in fact many widespread selection techniques that provide little more than chance estimates of who will succeed in a given position. Some examples include education level, training and experience ratings, and academic achievement, which are all common selection methods that nonetheless provide minimal predictive utility. Other popular selection methods, such as unstructured interviews, vary considerably in their effectiveness and are far from optimal.”

3.         “Why, then, are ineffective selection techniques so popular, when there is a large body of scientific literature detailing best practices for performance prediction?  … one of the most common reasons for not employing optimal selection methods is that many human resource practitioners and top managers simply do not believe in the real-world effectiveness of empirically validated selection tools. …  Despite the many studies that examine the utility and validity of different selection procedures, the results of this research have not fully permeated the awareness of managers and decision makers.”

4.         “In order to take advantage of the large individual differences in productivity, it is first necessary to identify the top candidates. In this respect, it is clear that hiring the best people requires the use of the best selection procedures. Based on meta-analyses of numerous validation studies … the most effective and efficient method for selecting the top performers involves testing for both cognitive ability and personality (Schmidt & Hunter, 1998).” The paper then examines these two domains in more depth:

Cognitive ability: “… also known as general mental ability, intelligence, or simply IQ, is one of the best predictors of performance across many different domains. Broadly speaking, it reflects an individual’s ability to plan, reason, process information, and control his or her behaviour. Some would argue that it is in fact the best-validated construct in all of psychology, as its ability to predict performance has been repeatedly demonstrated in thousands of studies carried out across 100 years of research (Schmidt & Hunter, 2004).

• Across all job categories, individual differences in cognitive ability account for approximately 25 per cent of the variability in performance.
• The general factor of cognitive ability predicts performance outcomes even better than aptitude tests claiming to assess the specific skills needed for a given job.
• If only one variable could be assessed to predict performance across multiple domains, cognitive ability would certainly be the most useful.
• While there has been some concern that such tests are culturally biased, there are also non-verbal tests of cognitive ability that do not discriminate against respondents from different cultural and linguistic backgrounds.”

Personality Factors: “The second most important variable in performance prediction is personality:

• While researchers have long used a variety of trait dimensions to predict real-world outcomes, the field of performance prediction has benefited greatly from the five-factor model of personality. The five-factor framework, or ‘Big Five’ model, is a taxonomy that describes personality differences across five broad dimensions of variation (Goldberg, 1993). The five dimensions are extraversion, agreeableness, conscientiousness, emotional stability, and openness. These dimensions demonstrate good cross-cultural reliability, are relatively stable across the lifetime, and incorporate the variance captured by most other personality taxonomies.
• Conscientiousness, which describes individuals who are reliable, hard-working, and self-disciplined, is the best personality predictor of workplace performance and academic success.Conscientious individuals have a strong work ethic, and tend to be more effective at pursuing their goals. An individual who is low in conscientiousness will be more easily distracted, less organised, and less productive.
• Following conscientiousness, the most important personality trait for predicting success across multiple domains is emotional stability. Individuals who score highly on this trait experience less negative emotion and generally handle stress better. In contrast, less emotionally stable individuals will have higher levels of chronic stress and anxiety. This trait is particularly important for predicting performance in highly demanding positions, and is also associated with increased health, job satisfaction, and lower rates of job burnout.
• Certain positions can benefit from the examination of other traits as well. In particular, the outgoing, assertive, and talkative nature of extraverts gives them an advantage in domains that require extensive social interaction. Extraversion therefore appears to be a good predictor of success in sales and management positions.
• Openness, finally, is a good predictor of performance in domains requiring innovation and creativity.

In conclusion, Mr. Hirsh notes that “Across a broad number of domains, there are large individual differences in performance and productivity outcomes. In order to capitalise on these differences, however, it is necessary to use the most effective selection methods. Even small improvements in the predictive validity of selection processes can lead to substantial economic benefits. A large body of research now indicates that measures of cognitive ability and personality are powerful and efficient tools for predicting performance. While some form of selection is inevitable for any competitive position, psychological assessment remains an invaluable tool for identifying the top performers and making an informed decision.”

The Hirsch paper does go on to examine faking and bias response, whether an emphasis on individual attributes and abilities is the best strategy for ensuring organizational fit, and the impact of situational factors on the use of these types of assessments. This article will not address these subjects but future articles for this newsletter will likely do so.

We will be happy to provide you with a complete copy of Mr. Hirsh’s article.  You can requisition a copy from our offices at support@prevueassessments.com.

Written by: Kenneth Danderfer

December 5, 2009