How many questions in 360 degree appraisal

Share Button

I’ve just commented on an interesting post from one of our competitors on how many questions you should have in 360 degree appraisals.  The post has interesting feedback on how long people take to complete questions. 

My comment was

“Mark,


Thanks for this – the stats are really useful. We share your avoidance of “quick and dirty” solutions. I guess the really interesting thing for me is whether the 19 minutes spent answering the HR questions gave better feedback to work with than the 14 minutes people spent answering the engineering company questions. We generally find that the narrative feedback is more useful than rating scale scores and use that insight when trying to gauge “how many” questions are needed.”

I strongly believe that the rating feedback serves the narrative feedback.  And “how long does it take to complete” is less important than “how rich is the feedback we get”.  I’d rather have 10 good narrative comments than 20 rating scales.  Great stats though!

Brendan

Share Button

The importance of narrative in 360 degree feedback

Share Button

A brief post today which was prompted by a recent client project where I was handling the 360 degree appraisal debriefs.

There were plenty of respondents, from each of the nominated categories, such as peers, direct reports and line manager (of which there were interestingly 2 of in this case) – all of them providing a lot a rated feedback as they answered all the questions.

However, the challenge arose when it came to looking at the free-text comments in support of the rated questions above – the line managers had made no comments…

Now with one line manager that would be difficult enough, but with two line managers, differing in opinion quite considerably, the ability to understand what was going on was reduced significantly.

It reminded me how important it is to communicate with respondents and train them in how to give good, constructive feedback. It is not enough to simply rate a statement, particularly if it is at one end of the scale or the other – narrative must be encouraged at all times – it allows the recipient to fully understand the impact of their behaviour, through the use of examples and evidence.

Without it, the feedback can become a blur of graphics which tells only half a story.

We can implement checks within the system which provide ‘pop-up’ boxes to encourage people to fill in the free-text comments section, but at the end of the day it must be through clear communication and framing of how the 360 degree feedback process works to best effect, that will establish the good practice of adding narrative.

John

Share Button

360 degree appraisal : getting the communications right

Share Button

Continuing with thoughts on how to best implement a 360, I’d like to add some thoughts on the communication required   I’ve already covered how to design the report, get the questions right, and the rating scale so at this stage all of the key components of the 360 degree appraisal should be in place.  Most importantly, we should be sitting on our strategic intention from our planning phase.

Communicating effectively will reduce anxiety and increase participation.

Normally we would recommend that communication covers : Why are we doing a 360?  What are the outcomes?  What is the process?  At each phase of communication you can touch these points rather than rely on one lengthy email or information document that may not be read. 

How and when to communicate is really dependent on the current knowledge within the company, the abilities of the participants, the culture, and the level of complexity in the process.  As a minimum we would recommend

  • Warm up communications that identify that the 360 is coming, the intention of it and at a high level how it will work
  • With a week to go, we recommend an email that explains the process
  • Within the email invitations to the recipients and/or the respondents of the 360 feedback, again give process information and clarity of purpose
  • Within the 360 degree feedback system itself, ensure that the welcoming page, first page of the questionnaire, and other instructional text is “on message”

Brendan

Share Button

Rating scales in 360 degree feedback

Share Button

Back to my movement along Bowland Solutions’ recommended approach to 360 degree feedback.  I turn my attention to the rating scale.

This can prove an oddly contentious point with our clients.  There are invariably 4 main debates

  • How many rating points
  • Should there be a middle point
  • Is not applicable allowed
  • The words – in particular whether there should be a frequency scale or a commentary scale

Let’s take these points in turn.

How many rating points : we recommend as few as possible.  Keep it simple.  Our standard scale would be 3 points plus not applicable.  So; Strength, Competent, Development Area would be a great scale.  We work with longer scales but we find that they add complexity more than the provide information.

Should there be a middle point : yes.  I know the fear – everyone will score in the middle (the Veblen effect).  But they don’t – really, the don’t.  And often people have a “he’s fine”, “ambivalent”, “mixed feelings” type response that they want to give.  Forcing them to go positive or negative creates a false response.

Is not applicable allowed : yes.  You might call it something else : “no evidence”?  But in 30 or so questions we have to expect some people to just not know.  Now – most important, you should review the responses to questions to ensure that questions are not overloaded with N/A responses – or that one group of respondents are answering in this way.  If they are then the question requires rewording or removing.

The words : We favour scales such as “Strength”, “Development Area” rather than “Often”,”Sometimes”.  The latter, frequency, scales are however effective and really this is just personal preference.  I find that the behaviours read better as a statement rather than as a question and this fits with a statement based rating scale.  What I would avoid at all costs is a numerical rating scale.

The rating scale in 360 degree feedback is less important than people make it – I believe that respondents look at it initially to get a feel for where to place their mark but really they are providing graduated feedback more than specific.

When debriefing though you do tend to use the rating scale as part of the language – and I concentrate on this area when I am designing the rating scale.

Brendan

Share Button

Face-to-face 360 degree feedback debriefs

Share Button

I am often asked when it is appropriate for a recipient to see their 360 degree feedback report for the first time….before the debrief or at the debrief itself?

The encouraging note here is that most people do now recognise the need for a face-to-face debrief, rather than allowing the individual to read their report of their own.

However I feel that allowing the recipient to read the report on their own before the debrief, misses much of the point of having one at all.

It is a very interesting to be able to see how the individual responds to the feedback when they read it for the first time….nodding, wincing, shrugging, etc, it gives a lot of information to the debriefer, which enables them to question such non-verbal responses and help get a true picture of how the recipient truly perceives the feedback.

If they are allowed to read the report beforehand, they will often come to the session with a more measured approach; they will have absorbed much of the suprise, shock or delight, and so the debriefer has lost a critical part of how the individual feels.

John

Share Button

Integrating your HR systems

Share Button

Not the sort of thing we normally concentrate on, but I was reading an article in HR Zone on integrating HR systems.

The article considers the merits of having one overall system (the SAPs of this world) with having best of breed solutions for say your 360 degree or performance appraisal systems.

There is a fledgling solution out their for integrating such services (HRXML).  Such a solution allows some like Bowland Solutions to send packets of information to a central HR service such as Oracle or SAP in a standard way.  This allows the central piece of software to report back in an holistic manner on the individual.

We are capable of generating an HRXML feed – although I have to say we have never been asked to do it by a client. 

I would recommend going the “best of breed” route for software.  We may be biased but you are likely to start with a payroll provider – ‘cos that’s the most important thing!  But there is a world of difference between payroll software and 360 degree or performance review software.  A successful 360 implementation arrives by ensuring that the admin is unimportant – it is the debrief that matters.  Payroll administration is a very different area.  I’d rather have the problem of integrating software than have the problem of having sub-standard software that doesn’t need integrating.

Take a look at the article – it is an interesting read on a dull topic.

Brendan

Share Button

Top Tips When It Comes to 360 Degree Feedback

Share Button

I recently compiled this list on the basis of my experience with a few clients we have successfully supported in getting their own 360 degree feedback programme underway:
 

  1. Support the Process with Trained People

 An area where I often see 360 degree feedback,and other forms of appraisal, fall over is when the process is not fully supported.The most successful are when the raters, recipients and debriefers all receivetraining on the role that they need to play..

  1. If it’s Complicated They Won’t Do It

Implementing any new process in an organisationis fundamentally a ‘change initiative’ and as such you need to take allpossible steps to make the process easy for people to adopt. To overcome this,tailor the system as much as possible to the organisation, including not toomany questions, wording that the user can relate to and plenty of opportunityfor free-text replies. But most of all, make it self explanatory and the systemvery easy to use. 

  1. The Aim is to Move Forwards – Not Just Find Out Where You Are

Always remember why you are doing a 360 degreefeedback. At the end of the day, it is to use the information to developindividuals so that tangible improvements can be made to business performance.  

The best way to do this is to repeat the cycleafter the training, coaching or other such intervention, so that you canaccurately assess the change, improvement and impact  of the process. 

  1. Follow Through

Some companies conduct 360 degree appraisalsand but do nothing with the results. Success is created by reacting quickly tothe training needs, publishing clear learning and development plans and, putquite simply, getting on with making improvements as soon as you have theresults.

If  nothing happens staff engagement will decreasenext time. Do something positive and the benefit of appraisals will soon gathermomentum across the whole organisation.

  1. Tell Everyone What’s Going On

When you introduce or move to an online 360degree system it can raise questions in people’s minds. Why is our companydeciding to do it this way? What’s changed? How’s this going to affect mepersonally?

Have a plan for clear, regular communicationand open channels for everyone to ask their questions. It will improve theoverall success and the benefits you enjoy as a business.

 John

Share Button

360 degree feedback for a doctor

Share Button

Some things come across the desk that leave you wondering.  I got sent a link to IWantGreatCare .  It is a website that offers to let you rate your doctor.  Now we once set up a website to rank tradespeople so Bowland Solutions have been down this route before ourselves.

What really has me pondering is how you rate your doctor on this site.  You select whether you trust them, whether they listen to you, and whether you would recommend them.  You have to move a slider to select from 0% to 100%.  You then get to write a review.

First, I get encouraged that those attributes are deemed important for a doctor – not sure that communication was particularly a big deal a while ago.  But then, you can’t help but start to wonder.  Surely whether the doctor gives good medical advice has to be important.  I know the patient isn’t well placed to judge this sometimes, but that might just be an argument that the website is a bad idea, rather than something to just ignore and rate them any way.

Sometimes when working with a client on 360 degree feedback or even on an annual performance review, we have to caution that there are times that you just can’t get the information you are looking for from a subjective rating.  I wish I liked this website, I really do.  And I really wish it would be useful information.  But somehow, I doubt the data it collects or the information it provides will really add to how we select a doctor.

Brendan

Share Button

360 Degree Feedback Debriefs – Training Tips

Share Button

I recently ran our training programme with a client, “How to conduct effective 360 degree feedback debriefs”, where we train internal HR and other staff how to share a report with the recipient of 360 feedback.

Fundamentally, we see the primary aim is to ensure the recipient understands what the feedback is saying; thereafter it is necessary to get a degree of acceptance from them before helping them determine what actions they wish/need to take as a result of the is feedback i.e. Changes in day-to-day behaviours or undertaking some other focused development activity.

As a consequence of these aims, it places the ‘debriefer’ firmly in the role of someone trying to ‘explore’ the report with the recipient.

Very often I see delegates on the programme, who ask evaluative questions during the role-played debrief sessions, rather than exploratory ones:

“Do you think you are a good leader?”
“Do you feel that you should do this rather than that?”

These questions forgo the chance to review the feedback itself, which is the main aim; better questions are more inquistive in nature:

“Have you had this type of feedback before?”
“What might have prompted someone to give you this feedback, do you think?”

The latter encourage more reflection and help the individual absorb the feedback more readily.

John

Share Button

360 degree appraisal : designing the questions

Share Button

Continuing with my theme of Bowland Solutions‘ approach to the 360 degree appraisal process I turn my attention to designing the questionnaire.

I am not going to concentrate on the creation of the questions themselves.  That is a wholly separate topic around competency frameworks which I intend to write separately on.  Instead, I wish to concentrate on the structure of the questions rather than their content.

We have found the following guidelines useful – but please note they are only guidelines!

  • Between 20 and 40 questions is about right
  • Usually we see these as groups of 3-5 questions per competency
  • Adding a narrative question for each competency is normally the right way to go – we have adopted a question that one of our partners, Peter Hyde, used for a common client.  “Please provide comments/evidence/examples that support your answers above”.  We don’t always stick to this – but it is a good start.
  • Consider whether every group of people can answer every question.  If not, then exclude the questions from that group.  So, peers may answer a subset of the overall question set for example.
  • It can be useful to have a question (or even 2 or 3) at the end of the 360 that asks people to give broader feedback or cover points they would like to make
  • Questions should be brief, clear and unambiguous, and describe an observable behaviour
  • If you have people for whom English is not their first language then we would recommend translation the questions.  You probably don’t need to translate the whole system but the nuance of questions matters.

I’ll cover rating scales in my next post so I won’t go into detail here – but clearly whether you are using a frequency rating scale (e.g. Often, rarely,…) or a observational rating scale (e.g. Strong performer, development need) makes a difference to how you should work the question.

Whether we ask the right questions is obviously an important contributor to the success of the 360.  I would say, though, that if you build the reports first and ensure the competency framework is drawn down from the strategy then the questions should start to write themselves.

Brendan

Share Button

360 degree appraisal : designing the report

Share Button

Continuing along our suggested approach to a 360 degree appraisal project I turn my attention to design.  Design covers many areas, and today I wanted to consider the report design.  I have deliberately started at the end of the exercise as I believe that thinking about the report – having already thought about the strategic intention of the 360 degree feedback – allows you to ‘begin with the end in mind’.

When designing the report, Bowland Solutions have the following biases (we like to call them principles!) in mind.  First, we believe that 360 degree appraisal is a method of consolidating feedback rather than averaging it.  Second, we believe that narrative offers the richest information – the rating scales simply tell you where to look.

With these principles in mind, we use the following checklist as a guide that influences us in designing the reports.

  • What is the purpose of the feedback.  In particular we are heavily influenced if the 360 is linked to an annual performance appraisal.
  • Who will receive the report : the recipient, their manager, HR ?
  • What experience has there been of 360s within the organisation?
  • What is the culture of the organisation?
  • What icons, symbols, other literature exists within the organisation that should influence the design of the report

With this framework we are able to build a highly tailored report.  We do of course use building blocks that have worked in the past and which we consider to be best practice, but a report that works for the company and the recipient is what we are looking to achieve.

We have found that in many cases there is a template for the structure of the report.  We often follow this structure

  • Summary table of where feedback came from
  • Overall, by competency, tabular summary of feedback (not averages)
  • For each competency, a tabular summary of feedback
  • Within each competency, graphical representation of feedback for each behaviour with a summary of the narrative comments received

We use various graphical reports to meet specific client requirements but this structure holds fairly well and supports a organised, clear debrief.

Brendan


Share Button