Kirkpatrick - should we be sulking about it?
I have had no issue with the Kirkpatrick model, as it has always made sense to me. Just recently I have been challenged to reconsider my position. Last week running the Learning Loop, one of the participants said they have started to use Will Thalheimers’ Learning Transfer Evaluation Model, which made me curious about what that model had to offer, that was different to Kirkpatricks.
This week I read a blog from Work Learning Research saying how levels 3 and 4 of Kirkpatrick are misunderstood by most L&D people, that they use learner reactions as if they were valid level 3 and 4 measures. The blog got me ruffled a bit, particularly because the survey results:
1) use a sample size of TWO for vendors
2) there is no visible link with those 250 surveyed and whether they are actually even measuring at levels 3 or 4 (if they are not, then could they know the answer to the question?)
This makes any conclusions, drawn from the research, in my opinion tenuous to say the least.
What made me curious enough to look at Will Thalheimers’ model was any suggestion that it had something new to offer. What I did like was the focus on learning transfer, but then I thought - shouldn’t the focus be on performance? Will has a lot to say and criticise the Kirkpatrick model for, but I am looking for a deeper and better way to do evaluation. What I would like to offer is a deeper explanation of Kirkpatrick’s model and I hope you will make up your own mind how you might apply it.
“Begin with the end in mind” said Steven Covey. I agree wholeheartedly. If you do a thorough analysis, engage with stakeholders and line managers about the outcomes (in performance) required, then evaluation should be straight-forward. But please note if the analysis part is skipped over or done on a superficial level, THE EVALUATION WILL BE DIFFICULT WHICHEVER MODEL YOU USE.
- Level 1 – learner reactions – still an important part of seeing if you have had sufficient engagement and struck the right chord with the objectives (Business focused and learner-centred according to the 5 Secrets of Accelerated Learning (author - Krys Gadd)
- Level 2 – learning achieved – did they learn what they were supposed to and have they met those objectives? This is important to know for L&D and their line managers
- Level 3 – impact on performance – this is what is then observed on the job and in my opinion should not be L&D’s sole remit. If they have analysed the needs correctly and determined the correct outcomes, the line managers should be engaged enough to help imbed the learning as well as measure performance improvements.
- Level 4 – impact on the business- if the stakeholders have been engaged and the outcomes are focused on the business, the stakeholders will be interested in measuring the business impact.
In the graphic below I have overlaid Kirkpatrick model onto Will Thalheimers' model.
Course and Learning Solutions Building @ Michaels & Associates | Employee Training
6yJust measure the right things. Want to measure performance improvement in supply chain? Gather the current metrics, figure out what has to improve, build your training, create assessments that prove the training is effective, then gather the new metrics (usually error rates), and see what improved - or not. Measure the right things.
Instructional Systems Designer | LinkedIn Top Voice | Speaker | eLearning Developer
6yThe problem with some of these models is that they are not data analysis models at all. They don’t seem to have been designed for the workplace either. Level 1 is ambiguous and subjective, it has great applications for ILT where instructors would vary. Level 2 only proves cognition and how are you doing that? With a 10-question quiz? Level 3 I’ve done it, but it’s not easy. Never seen it done and properly documented in real life. Why? You need to add more work to already overwhelmed leaders. Level 4 this has been mostly based on correlations and that doesn’t give us causation i.e. we are selling more since training was effected. I would say mirror the evaluation tactics of your business leaders and consider not using academic models for workplace results. Here’s another big question: If you evaluate and find out your training sucks, are you making it public to your org?
Supporting transformational change, through people and organisational development, to support better health and wellbeing outcomes across the UK.
6yReally interesting article and discussion - we're currently discussing how to evaluate the impact of our internal development programme. I tend to use Kirkpatrick with the same caution as I treat any model I use in training - as a useful lens to guide thinking rather than as a process to follow blindly. The most useful way I've found to use Kirkpatrick is to use it as a tool to scope learning. If I client approaches us asking for training on a particular topic, I'll ask a series of questions to support them to think beyond 'a training course'... What's the impact you are hoping to have on organisational/team results? (Level 4) Therefore what behaviour do you need to see from people to achieve these results? (Level 3) Therefore what do they need to know? (Level 2) How can we create the best learning experience to give them the knowledge and skills? (Level 1) Using the model in reverse is much more helpful in achieving real change, and can help with any evaluation. I also find that clients are then more engaged in evaluation if they've been encouraged to think like this from the outset...
Executive Director JPMorgan Chase - Head Of Operational Learning
6yKirkpatrick... a model for its day and not for now.. it’s always been a challenge for any business to show and demonstrate how training alone has made that significant change in business performance.. We live in a world now where employees are truly the most important asset a business has. So employee engagement and the impact they have day to day on our customers is quite simply the best way to measure success. It’s got to be more simple and not over engineered... we don’t apply Kirkpatrick when we buy a new book from amazon or rate a meal after we’re in Nando’s.. There must be a way we can learn from how we measure customer experience.. what can we bring back to measuring the colleague experience in a similar manner. Kirkpatrick takes to long.. problem has been and gone and a new one has appeared by the time we can measure..
Global learning and development leader.
6yStacey Hatch What are thoughts here?