... And, reasons why development tests should include not only feedback from professional colleagues but also from personal relations This article is based on joint research by ESADE, Pompeu Fabra University and Case Western Reserve University Effective leaders have a sense of self-awareness, a desire to learn and grow and develop resonant relationships that inspire others. They create a sense of well-being and trust around them and activate people's potential by focusing on the latter's strengths, dreams and unique capabilities. In his research findings published in Frontiers in Psychology, Joan Manuel Batista-Foguet (Director of ESADE's Leadership Development Research Center) and his colleagues shed light on why leadership assessment has to go beyond IQ tests and why development questionnaires to measure leadership effectiveness should include feedback from different observers and contexts. Leadership assessment has to go beyond IQ tests How useful are IQ tests to measure leadership effectiveness? Joan M. Batista: During the First World War (R. Yerkes) and, specially, after the Second World War, psychometrics exploded, and tests became a very popular means to "scientifically" measure human abilities. People were qualified as more or less intelligent based solely on how capable they were of answering tests. For instance, a fireman would be judged as capable of putting out a fire just by his test score. There is plenty of academic work that argues against this narrow test approach (Stephen Jay Gould's is among the most entertaining). Tests can do a lot of damage... There was a point in time in which people believed that intelligence was the exact score on an IQ test. Some have seen their careers fail because of tests, and I'm afraid we're still doing the same nowadays.For example, to access graduate management programs, students are required to take the GMAT (Graduate Management Admission Test) which is supposed to measure a set of verbal and logical competencies required to complete an MSc or MBA program at any university. But, did you know that you can improve your score each time you take the test? The second time is better than the first, and so on. This threat to the validity of the inferences you can make about applicants' competencies is known as "testing". Actually, applicants can demonstrate an increase in their competencies in one area by improving the way they take the exam. Some academies that prepare students for the GMAT or other tests do not teach them content; they teach them how to pass the exam. There was a point in time in which people believed that intelligence was the exact score on an IQ test How can tests be improved? People working in social sciences have a hard time accepting something important. We often try to analyze things that we don't see, concepts such as attitudes, poverty, wealth, health, competency, love. They are abstractions, which we agree to anchor to reality through observable indicators (we often operationalize them by means of the responses to items included in a questionnaire) and call them constructs. However, we need to realize that they are intangible constructs, which means that they are just invented in order for us to understand and communicate with each other. And, as such, people can interpret them in different ways. So, when we refer to constructs, we have to observe them from several points of view (triangulation). This is best illustrated by the "competency" construct which is typically assessed in leadership development programs. How so? In classical performance tests, evaluators usually analyze competencies related to job performance only through professional colleagues (subordinates, peers, bosses, etc.) in the same specific context (work competencies). They always look for convergence among raters, since agreement among them can be interpreted as evidence of the construct's validity and its measurement (i.e., what we measure is a valid inference of that competency's existence). However, when we assess leadership competencies, in general, we normally use questionnaires, which are the basis for the development programs. These questionnaires are based on feedback from others -from the individuals' work and personal environments-, but the diversity of competencies, contexts and raters' perspectives makes the convergence of all these viewpoints meaningless. All perspectives are valid and, eventually, complementary. What do your research findings reveal? In our research, we demonstrate that there are more appropriate rater perspectives and contexts when assessing specific competencies. For example, if you want to know if I have self-control, don't ask my boss whether I have it or not. Ask my subordinates and peers, or, better yet, my wife and children! However, if you want to know about my level of organizational awareness, ask my work colleagues instead of my wife, since she would probably have a hard time assessing this competency. If you want to know if I have self-control, ask my subordinates and peers, or, better yet, my wife and children! The person evaluating you has different criteria when assessing, for instance, flexibility. This will depend on whether the rater is your subordinate, your boss or your friend. Development questionnaires are like a broken mirror: To get a global picture you have to put all the pieces together. If you want to give leaders proper feedback, you need all the possible sources of information about the person's behavior. Besides, this can help them contingently apply the most appropriate leadership style. We know that what friends see, compared to peers or subordinates, is different because their observations come from completely different contexts. You not only act differently based on who you talk to, but the person observing you probably has different criteria in terms of empathy, flexibility and so on compared to other people. What contributions does your research make? The poet, Ramón de Campoamor, was right. When we measure a construct like love, health, empathy or flexibility, context is key. We shouldn't analyze a competence by itself; we should also take into account the context surrounding it because the lens each of us uses to observe reality is different. If you want to give leaders proper feedback, you need all the possible sources of information about the person's behavior For development tests to be effective, empathy, self-awareness, flexibility and so on have to be measured in different contexts. We show that a competency measured by the individual himself/herself, by someone from the latter's personal environment or from the professional domain has very different results. For example, empathy ratings are usually low when assessed by bosses, peers and subordinates compared to partners, friends, classmates and family. What are the takeaways from your findings? If we are aware that not everyone rates the same competence in the same way, we can find environments that are better suited to evaluate each specific competency. For example, for organizational awareness, don't ask your friends; ask the boss directly. For leadership or teamwork, you can ask friends. Adopting this new approach in development questionnaires allows us to customize our 360º surveys for specific environments depending on the competence. And this results in shorter questionnaires for each rater group -nobody likes long questionnaires!- which should improve data quality as well as feedback quality. You may also like: 4 factors to be happy at work

ESADE

Back to home

IQ is just the threshold: Measuring leadership effectiveness requires a variety of viewpoints

03/2019

... And, reasons why development tests should include not only feedback from professional colleagues but also from personal relations


This article is based on joint research by ESADE, Pompeu Fabra University and Case Western Reserve University



Effective leaders have a sense of self-awareness, a desire to learn and grow and develop resonant relationships that inspire others. They create a sense of well-being and trust around them and activate people's potential by focusing on the latter's strengths, dreams and unique capabilities.


In his research findings published in Frontiers in Psychology, Joan Manuel Batista-Foguet (Director of ESADE's Leadership Development Research Center) and his colleagues shed light on why leadership assessment has to go beyond IQ tests and why development questionnaires to measure leadership effectiveness should include feedback from different observers and contexts.


Leadership assessment has to go beyond IQ tests


How useful are IQ tests to measure leadership effectiveness?


Joan M. Batista: During the First World War (R. Yerkes) and, specially, after the Second World War, psychometrics exploded, and tests became a very popular means to "scientifically" measure human abilities. People were qualified as more or less intelligent based solely on how capable they were of answering tests. For instance, a fireman would be judged as capable of putting out a fire just by his test score. There is plenty of academic work that argues against this narrow test approach (Stephen Jay Gould's is among the most entertaining).



Tests can do a lot of damage...


There was a point in time in which people believed that intelligence was the exact score on an IQ test. Some have seen their careers fail because of tests, and I'm afraid we're still doing the same nowadays.


For example, to access graduate management programs, students are required to take the GMAT (Graduate Management Admission Test) which is supposed to measure a set of verbal and logical competencies required to complete an MSc or MBA program at any university. But, did you know that you can improve your score each time you take the test? The second time is better than the first, and so on. This threat to the validity of the inferences you can make about applicants' competencies is known as "testing". Actually, applicants can demonstrate an increase in their competencies in one area by improving the way they take the exam. Some academies that prepare students for the GMAT or other tests do not teach them content; they teach them how to pass the exam.


There was a point in time in which people believed that intelligence was the exact score on an IQ test


How can tests be improved?


People working in social sciences have a hard time accepting something important. We often try to analyze things that we don't see, concepts such as attitudes, poverty, wealth, health, competency, love. They are abstractions, which we agree to anchor to reality through observable indicators (we often operationalize them by means of the responses to items included in a questionnaire) and call them constructs. However, we need to realize that they are intangible constructs, which means that they are just invented in order for us to understand and communicate with each other. And, as such, people can interpret them in different ways.


So, when we refer to constructs, we have to observe them from several points of view (triangulation). This is best illustrated by the "competency" construct which is typically assessed in leadership development programs.



How so?


In classical performance tests, evaluators usually analyze competencies related to job performance only through professional colleagues (subordinates, peers, bosses, etc.) in the same specific context (work competencies). They always look for convergence among raters, since agreement among them can be interpreted as evidence of the construct's validity and its measurement (i.e., what we measure is a valid inference of that competency's existence).


However, when we assess leadership competencies, in general, we normally use questionnaires, which are the basis for the development programs. These questionnaires are based on feedback from others -from the individuals' work and personal environments-, but the diversity of competencies, contexts and raters' perspectives makes the convergence of all these viewpoints meaningless. All perspectives are valid and, eventually, complementary.


What do your research findings reveal?


In our research, we demonstrate that there are more appropriate rater perspectives and contexts when assessing specific competencies. For example, if you want to know if I have self-control, don't ask my boss whether I have it or not. Ask my subordinates and peers, or, better yet, my wife and children!


However, if you want to know about my level of organizational awareness, ask my work colleagues instead of my wife, since she would probably have a hard time assessing this competency.


If you want to know if I have self-control, ask my subordinates and peers, or, better yet, my wife and children!


The person evaluating you has different criteria when assessing, for instance, flexibility. This will depend on whether the rater is your subordinate, your boss or your friend. Development questionnaires are like a broken mirror: To get a global picture you have to put all the pieces together. If you want to give leaders proper feedback, you need all the possible sources of information about the person's behavior. Besides, this can help them contingently apply the most appropriate leadership style.


We know that what friends see, compared to peers or subordinates, is different because their observations come from completely different contexts. You not only act differently based on who you talk to, but the person observing you probably has different criteria in terms of empathy, flexibility and so on compared to other people.


What contributions does your research make?


The poet, Ramón de Campoamor, was right. When we measure a construct like love, health, empathy or flexibility, context is key. We shouldn't analyze a competence by itself; we should also take into account the context surrounding it because the lens each of us uses to observe reality is different.


If you want to give leaders proper feedback, you need all the possible sources of information about the person's behavior


For development tests to be effective, empathy, self-awareness, flexibility and so on have to be measured in different contexts. We show that a competency measured by the individual himself/herself, by someone from the latter's personal environment or from the professional domain has very different results. For example, empathy ratings are usually low when assessed by bosses, peers and subordinates compared to partners, friends, classmates and family.


What are the takeaways from your findings?


If we are aware that not everyone rates the same competence in the same way, we can find environments that are better suited to evaluate each specific competency. For example, for organizational awareness, don't ask your friends; ask the boss directly. For leadership or teamwork, you can ask friends.


Adopting this new approach in development questionnaires allows us to customize our 360º surveys for specific environments depending on the competence. And this results in shorter questionnaires for each rater group -nobody likes long questionnaires!- which should improve data quality as well as feedback quality.


You may also like: 4 factors to be happy at work


More Knowledge
Multisource assessment for development purposes: Revisiting the methodology of data analysis
Batista Foguet, Joan M.; Saris, Willem Egbert; Boyatzis, Richard; Serlavós Serra, Ricard; Velasco Moreno, Ferran
Frontiers in Psychology
Vol.  9, nº 2646, 12/2018, p. 1 - 16
The power of EI competencies over intelligence and individual performance: A task-dependent model
Truninger De Albuquerque, Margarida; Fernández Marín, Xavier; Batista Foguet, Joan M.; Boyatzis, Richard; Serlavós Serra, Ricard
Frontiers in Psychology
Vol. 9, nº 1, 09/2018, p. 30 - 43
Exploring the relationship between two compositions using canonical correlation analysis
Mateu Figueras, Gloria; Daunis Estadella, Josep; Coenders Gallart, Germà; Rosell , Berta; Serlavós Serra, Ricard; Batista Foguet, Joan M.
Metodoloski Zvezki: Advances in Methodology and Statistics
Vol. 2, nº 13, 12/2016, p. 131 - 150
Why multisource assessment and feedback has been erroneously analysed and how it should be
Batista Foguet, Joan M.; Saris, Willem Egbert; Boyatzis, Richard; Serlavós Serra, Ricard
Why multisource assessment and feedback has been erroneously analysed and how it should be
European Academy of Management (EURAM)
Bruselas (Belgium) ,  01/06/2016 - 01/06/2016
Back to home