Earlier today and under extremely difficult technical conditions, I presented early results from the Certification 5 survey during SAP Inside Track Belgium. At the time of writing we have 399 responses. The pattern of responses to key questions is not changing over time. I'll get to that later.
Here's the presentation which you can download from the link in the header of this blog post.
The thing to understand is that the key theories we proposed in the original Certification 5 paper are largely supported by those who have completed the survey.
So for example we asked:
"If you are SAP Certified, do you believe that the multiple choice question format validated your practical skills?"
62% said no. One respondent added:
"SAP Certification via MCQ is an insult to the disclipine of Computer Science/Information Systems. None of my higher level CompSci classes were passed only by answering multiple choice questions. Higher level computer science classes(including systems analysis) require essay style questions to test knowledge in an objective fashion. The current MCQ certification exams only promotes rote memorization of feature lists and not whether you have the ability to solve a business problem via that solution."
That's possibly the most stinging indictment I have ever read of SAP Certification. The flip side of that question:
"SAP Professional Level Certification is designed to validate advanced SAP skills "by requiring proven project experience, business process knowledge, and a more detailed understanding of SAP solutions" in your focus area. Do you think this can be accomplished solely by multiple choice?"
72% said yes. To the follow up:
"If SAP Professional Certification included a requirement to provide the answer to a practical problem the answer to which could be objectively assessed, do you think this would represent a valuable enhancement to SAP Certification?"
87% said yes
I said I would talk about patterns, When I developed the survey there was always the risk that much of what the C5 had been trying to say would be blown up as not something the market of SAP freelancers and others want. Once we got past the first 100 or so answers I started to monitor the pattern of results and especially to the questions mentioned above. That is because the style of Professional level certification is what now lies at the heart of discussions with SAP and, at some stage, will inform the Master level.
I can confidently say that up to present, there has been very little movement in the numbers. So for example the answers to questions about validation as described by SAP vary in the range 71-76% over time. The answers to the question of practical problem requirements range 83-88%. I consider these to be insignificant in the context of the whole and statistically valid. Other may disagree but I am satisfied that subject to some obvious 'fixing' we have a result that's unlikely to materially change over time.
It is possible these results could change significantly with a larger sample but I am assuming that would not be the case given we have now been running the survey some 6 weeks.
During the SAP Inside Track Brussels event I was asked if I can share more from the survey. The answer at this time is 'no.' It is not that I wish to hide anything but the C5 need to give SAP Education a full debrief including our thoughts on the totality of the results before releasing the full results into the public domain. We believe that is only fair as we expect they will wish to consider and respond to the findings.
Two things I can share which I think are contextually valuable:
36% of respondents were not aware SAP is expanding to Professional and Master levels - that means there is a strong need for communication.
64% of respondents have not read the C5 paper. It was important for us to know this because there is always the possibility that vested interests in the topic might try and flood the survey and so skew results. Given the C5 paper has been downloaded: 4,891 times as at the time of writing then I think it is safe to assume we have succeeded in avoiding obvious survey completion bias. Given also that the number of people who have not read the C5 paper has grown significantly over time (at the 200 response levels it was about 50/50) AND taking into account the patterns we're seeing, then I belief this adds weight to my confidence that what we're seeing is defensible.
Am I pleased with the results survey so far? Of course. It is always gratifying to see theory tested and proven largely correct or at least in the right ball park. Is there much from which SAP can learn? Absolutely. Having a 31% qualitative response rate from respondents is phenomenal. It is clear this topic continues to stir plenty of thought and for that I must thank all those who took the time and effort to run through the 23 questions and then go on to add insights we have so far not been able to see.
One thing that HAS been a disappointment. There are very few responses from the SI community. We know from back channel conversations that SAP Certification is a thorny topic for them. They often do their own thing and, quite frankly, would rather things stayed as they are. There are many reasons for this. The C5 do not share that view, considering it to be the 'lazy way' that fails to address the needs of customers.
We are continuing to run the survey through TechEd so if you are reading this for the first time and would like to participate then please take the survey here.