New Studies in 360-degree Feedback Effectiveness

2006 SIOP presentations by 3D Group shine light on leadership assessments.

Dallas, TX (PRWEB) May 4, 2006 -- 3D Group, industrial/organizational psychologists specializing in 360-degree assessments, will be presenting the results of three new studies of multi-rater feedback at the Society for Organizational and Industrial Psychology’s 21st annual conference in Dallas. The studies provide an important evaluation of the validity and usefulness of these extremely popular employee assessment tools – and reveal some surprising results.

Dale S. Rose, PhD., President and co-founder of 3D Group explained several reasons for conducting the studies. “At one level, we wanted to validate our latest survey instrument now that it has been in the field for a while. At another level, we wanted to investigate something that’s always been a bit of a gray area: the usefulness of narrative comments. Finally, we wanted to look at the value of qualitative comments from a novel point of view: the people actually receiving them.”

We hope it will help guide feedback providers to write comments that are clear and focus on specific, job-related behaviors – and not to shy away from negative observations.

In the first study, Rose and Dr. Greg Robinson looked at 3D Group’s own Leadership Navigator for Executives assessment tool. Introduced in September 2004, this survey instrument was among the first off-the-shelf products to enable 360-degree assessment at the Vice President level, and completed 3D Group’s suite of assessment tools across all levels of leadership.

The survey is tailored to the specific tasks and challenges faced by high-level executives and carefully validated through literature review before release. However, because 3D Group has found that the effectiveness of a given tool is highly dependent on role- and task-specific information, they conducted a statistical analysis of the assessment’s real-world success using a sample of 77 executives and 1046 ratings. The results covered three areas: All 12 competency scales, in the two categories of “Organizational Leadership” and “Leading People,” were found to be highly reliable; the competencies were shown to measure executive competence; and finally, executives found the survey to be highly relevant to their actual jobs.

The second study, conducted by Treena Gillespie, Rose, and Robinson, looked beyond specific tools, jobs, and competencies into one of the less-well-understood areas of 360-degree assessment: narrative comments. “Most corporate executives will use a multi-rater assessment at some point in their career, and practically all of these assessments include space for qualitative, open comments,” notes Rose. “But are these actually useful? Very few people have tried to find out.” This research built on a previous study by Rose and Farrel (2002), in which 25% of comments were found to be distracting or irrelevant using standardized rating criteria.

In the new study, using a sample from surveys done for their clients, the group had subject matter experts rate the narrative comments for content validity and job relevance, and for usefulness (defined by clarity and trait- or-behavior-based focus). “The results clearly showed that usefulness varies by what you ask, and even more by whom you ask,” said Dr. Rose. “Comments from peers proved much less useful than those from superiors or subordinates. It’s not clear whether peers cannot, or will not, provide valuable feedback – but it is clear that peer feedback presents a challenge to practitioners of 360-degree assessment and their clients. However, we were also surprised to find that usefulness did not vary by source and competency area, which is a positive sign for overall effectiveness.”

Finally, the 3D Group team turned to what is arguably the most relevant measure of survey usefulness: the reaction of the people actually receiving feedback. After delivering the feedback results and coaching executives on development, 3D Group consultants asked 44 leaders to determine which of the reviewers’ comments they considered “most useful” and “least useful.” Researchers blind to the survey’s purpose then assessed the comments for their clarity, favorableness, and any basis in specific behaviors or tasks.

The comments executives felt were “most useful” tended to be those rated as clearer and more behaviorally specific than the “least useful” comments. Interestingly, executives also found unfavorable comments to be more useful than favorable ones. In other words, executives appreciated the developmental potential of “bad news,” and were more responsive to constructive criticism than positive reinforcement. According to Rose, the study will have practical as well as academic value. “We hope it will help guide feedback providers to write comments that are clear and focus on specific, job-related behaviors – and not to shy away from negative observations.”

Taken together, Dr. Robinson believes that the three studies make an important contribution to the literature of 360-degree assessment: “360 degree assessment has undergone several stages of refinement, and the techniques are highly sophisticated. At the same time, it’s been around long enough that public perceptions of its value can be a real issue. It’s important to continue this kind of research, both for the technical benefit of practitioners in the field, and for the larger good of the industry.”

About 3D Group
3D Group specializes in 360-degree feedback and other employee assessment tools. They use their expertise in psychometrics – the specialized discipline of psychological testing and measurement – to deliver a wealth of objective, reliable, and easy to understand data that can be utilized in a variety of ways to improve manager and company performance. 3D Group is dedicated to helping businesses and nonprofit organizations of all sizes increase the effectiveness of their people and programs. 3D Group researchers have authored numerous publications related to employee assessment, including the “Benchmark Study of North American 360-Degree Feedback Practices, 2003 Study” and “Current Trends in 360-Degree Feedback.”
More information can be found at: http://www.3dgroup.net

Available for interviews: Dale Rose, Ph.D., President and Co-Founder of 3D Group.

Media contacts:       
Donna Lehman      
770-565-7275

###