Assessing leadership talent

We are currently assessing the leadership capability of the executive team of a well known resources company. There is a constant debate amongst executive assessment providers regarding what to look for and how to measure leadership capability. The issues consistently raised in the assessment research address three key areas; the taxonomy of global leadership competencies, the criterion-related validity of each construct and the effectiveness of various assessment methodologies, in particular, the use of 360-degree assessment.

Many assessment approaches target leadership competencies or capabilities from a behavioural or skills perspective. Research supports that this is one domain most open to individual development; however, this approach ignores the key leadership aspects less malleable. Tubbs and Shultz (2006) suggest that one needs to consider personality and values in addition to leadership competencies.

In summary, some leadership aspects are greater predictors of success than others. For example, a leader’s IQ is a greater predictor of future performance than his or her height. Hence, a key issue relating to leadership assessment is the selection of the most relevant criteria that will predict leadership effectiveness. A recent meta-analysis (Arthur et al., 2003) examined almost 180 research articles to determine the most effective criteria for assessing future job performance via the assessment centre methodology.
The major findings in relation to key abilities are that:

• ‘problem-solving’ accounts for 15 percent variance in job performance,
• ‘influencing others’ accounts for 3 percent variance in job performance,
• ‘planning and organising’ accounts for 1 percent variance in job performance, and
• ‘communication’ (including written communication) accounts for 1 percent variance in job performance.

The researchers describe these as ‘cognitive-like abilities’ and argue, as do others, that such abilities display some of the ‘highest criterion-related validities in the field’. Hence, an assessment methodology that accurately measures these cognitive-like abilities (that collectively account for 20 percent of the variance in job performance) will deliver a higher level of reliability and validity than those omitting one or more of these criteria.

The authors also recommend that the assessment criteria be minimised (i.e. focus on the critical abilities rather than extending these to all abilities), that psychologists and HR professionals be used as opposed to managers (also supported by Lorenzo, 1984) or that trained assessors be used. Soundly designed (i.e. high reliability and validity) and correctly administered psychometric instruments are the most accurate and cost-effective way to assess these abilities.

The major issue with 360-degree assessment relates to ‘fit for purpose’ considerations. Toegel and Conger (2003) successfully argue that 360-degree assessment methodology is less effective as an appraisal tool and more effective in a development application. In essence their research suggests that 360-degree assessment when used as an evaluative or appraisal process (as in succession planning) has very poor validity and reliability (i.e. it is inaccurate due to a range of individual and organisational biases).

The authors’ findings relate less to the method of data collection (i.e. on-line versus face-to-face interviews) and more to the methodology and content design (e.g. values and behaviours versus actual performance outcomes). It also supports the research (Lorenzo, 1984 and Arthur, et al., 2003) highlighting the biases and hence unreliability of using untrained assessors, e.g. most managers and executives, when rating the abilities of others and further reinforces the argument for using psychologists or HR professionals.

Importantly, Toegel and Conger find ‘that interrater reliability for overall performance is not the same for the different reference groups. The consistency among bosses' ratings is the highest (.52), followed by that of peers (.42), and that of direct reports (.26). It seems that supervisors, peers, and direct reports ground their assessments in different criteria’ (2003, page 304). This suggests that conducting a 360-degree assessment as part of the candidate evaluation process will provide highly incongruent, inconsistent and unreliable data. In addition, the authors present a strong argument for the disengagement generated by participants in the 360-degree assessment process when used for appraisal purposes. The succession management research highlights this as a key success factor (Taylor & McGraw, 2004).