December 2010 Previous month Next month

Every 3 years, the OECD conducts a series of professional tests to measure the competence of 15-year olds around the world. This week, they've published the latest results from testing 470,000 people in 65 countries, in reading, maths and science.

Here is a table of the top performers:


As you can see, the top performance in the world is from the Shanghai province of China, and next come South Korea and Finland. The two largest countries where SAP employs people - Germany and the United States - do not get anywhere near the top.

SAP and SAP users do not of course recruit 15-year olds. But if we and others in the IT industry want to have a strong industry in the future, we need to be able to recruit competent people. And the 15-year olds of today are our new recruits tomorrow. We talk a lot about sustainability, mostly meaning resources and energy, but we also need to have sustainability of people.

SAP has employees in 120 countries and a very global culture, so I'm sure it can if it needs move operations around and remain successful. I am not suggesting that Germany and US and other places including my home country of the United Kingdom should be better than South Korea or Finland, but could they not be as good?

SAP and most of us in the IT industry want to recruit high performers. The graph below shows the percentage of high achievers in mathematics worldwide, showing the top two levels of a six level analysis by OECD:


This table seems particularly significant to me if companies are looking to employ the best people.  


For more information on the PISA test results, see the results section on the OECD website.


I'm an assessment professional and it's great to see such an impressive set of test results. And also to see politicians and others seem to appreciate the value of the results. For a more assessment-orientated take on the results, see my blog entry on the Questionmark site which looks at some of the reasons why some countries do better than others - including by offering good quality assessments.


I think there's also a lesson here for all of us. Find out the facts and measure our performance. Germany, the US, the UK seem successful, vibrant countries - three of the centers of the knowledge economy. But if our 15-year olds are not as strong, our future will not be as strong.


There may well be parallels in our organizations that assessments or other ways of measuring things could our organization realize the present and improve the future.


PS To help answer comments, here is the rest of the table above for the countries lower down the table:


It surprises me that many people I speak to don’t know about the Kirkpatrick Model of Learning Evaluation. It has been described before in the SAP Community in this Would You Take Training From This Man? by Marilyn Pratt, but it's very well worth knowing about.

Why you should care about how to evaluate training

If you work in a regulated industry, it’s common that there is a legal requirement for organizations to check the quality of their training  – for instance the UK Financial Services Authority states "Firms should ensure that their employees' training needs are assessed at the outset and at regular intervals (including if their role changes). Appropriate training and support should be provided to ensure that any relevant training needs are satisfied. Firms should also review at regular intervals the quality and effectiveness of such training." (my italics)

However whatever industry you work in, you will want to ensure that your company is getting good value for money and resources. For instance the USA alone spends over US$125 billion per year on training. 

Kirkpatrick Model

Professor Kirkpatrick came up with his model in 1959, and it has defined the landscape for evaluating training. It has four levels as shown in the diagram below.


Level 1 of the model measures participants reactions immediately after training : did they like it, was it relevant to their work and how could thetraining be improved?  Level 1 is usually measured by a course evaluation survey or smile sheet, indeed these are often called level one surveys.

Level 2 measures knowledge - did the participants learn? It's all very well discovering the participants liked the learning, but did they actually improve their knowledge and skills. It can be measured in other ways, but it's often measured with objective tests, or by checking the difference on pre and post course tests.

Level 3 measures behavior - whether the participants were able to apply their learning on the job. Essentially you compare the performance pre-training with the performance post-training and measure the difference. Level 3 is often measured by surveys or interviews, or by observational assessments where an observer monitors how someone is doing and fills in a checklist (sometimes now on a mobile device).

Level 4 measures results for the business, that the training has actually impacted the business. This is the hardest to measure, but obviously the most important.

A key characteristic of the model is that the levels build up from one another, and it's important to measure at every level. If participants do not react well to training (level 1), then they are unlikely to learn. And if participants do not learn (level 2), then they are unlikely to improve behavior. And if they do not change behavior (level 3), then results will not improve. Achieving results in level 4 is usually due to success at levels 1, 2 and 3.

For more information on the Kirkpatrick model, refer to Donald L. Kirkpatrick's book "Evaluating Training Programs : The Four Levels" .  There is also a Powerpoint in Questionmark's Learning Cafe that gives a good introductory description.

 Other evaluation models

Professor Kirkpatrick is still alive and I had the honour of meeting him at a Questionmark user conference a few years back. Inevitably there have been further ideas in the area of learning evaluation over the years. Some I'd highlight are:

  • Jack Phillips in the 1990s added an unofficial 5th level : Level 5 as measuring Return on Investment, how much return a company makes on its training.
  • Josh Bersin has developed the Impact Measurement Framework which is an end-to-end model for measuring impact of training, well worth reading about.
  • Robert Brinkerhoff has introduced the Success Case Method which evaluates training by looking at where it's most and least successful. See here for a blog article that explains this.
  • And very recently Dr. Will Thalheimer is defining a Learning Landscape Model which presents a very insightful view of measuring learning. See here for a great video on Youtube that introduces this.

 If you already know of the Kirkpatrick model, I hope this is a useful reminder. If you've not come across it before, I hope you agree it's a great way of thinking about whether training is effective and how training can help business.