Second in a series of blog posts on how an SAP LMS (LSO or SuccessFactors Learning) and SAP Assessment Management can help make trustable assessments.
In my previous post in this series, I explained that to trust assessment results, assessments need to be reliable (consistent) and valid (measure what they intend to measure). But how can a typical corporate organization who is seeking to use assessment results for competency checking, regulatory compliance or testing understanding after training or make assessments that are likely to be valid and reliable?
A key part of the answer is that validity and reliability starts with the authoring process. You need good processes at each of these stages:
- Planning assessment
- Authoring items
- Assembling assessment
- Pilot and review
- Delivery
- Analyze results
But if you do not have a repeatable, defensible process for authoring questions and assessments, then however good the other parts of your process are, you will not have valid and reliable assessments.
Here are five key tips that anyone involved in assessments should consider doing.
1. Organize questions in an item bank with topic structure
Putting questions in an item bank structured by hierarchical topics facilitates an easy management view of all questions and assessments under development. It allows you to use the same question in multiple assessments, easily add questions and retire them and easily search questions, for example to find the ones that need update when laws change or a product is retired
Some systems require you to insert questions individually within an assessment without allowing you to bank and organize them. If you only have a handful of assessments or you rarely need to update assessments, such systems can work, but for most organizations, you need an item bank.
2. Use questions that apply knowledge in the job context
It is better to ask questions that check how people can apply knowledge in the job context rather than just whether they have specific knowledge. For less IT-literate workers, you may find that “hotspot” and “drag and drop” question types make it easier for them to respond in the context of them performing their tasks without having to develop “office” style typing skills.
3. Have your subject matter experts directly involved in authoring
One of the critical factors in making successful assessments is to get effective input from subject matter experts (SMEs), as they are usually more knowledgeable, more current and so better able to construct and review questions than generalist learning specialists. If you can harvest or “crowdsource” items from SMEs and have learning specialists review them, your items will be of better quality.
4. Ensure good review of all questions
You need to make it easy for SMEs and others to collaborate in a good review process to ensure that questions will contribute to validity and reliability. Capabilities like tracking changes in questions are useful for this, and also help ensure that assessments are legally defensible.
5. Set a pass score fairly
Setting a pass score fairly is critical to being able to trust an assessment result. You need to know that if someone passes, they are competent and that if they fail, that the remediation or other action you take is justified and worthwhile.
One useful capability within SAP Assessment Management is the ability to make a pass score dependent on topic pre-requisites. For example, if an assessment has three topics, you can require that someone only passes if they score high enough score in each topic as well as on the whole assessment. This ensures that people do not pass tests when they are strong in some topics but weak in crucial ones and can help make an assessment more valid.
If you do all these things well, you will make your assessment results much more trustworthy and help all stakeholders in the process. If you are using SAP LSO or SuccessFactors Learning, then adding SAP Assessment Management to your suite will make it much more practical to perform these and other things to be able to trust your assessments.
In my next post, I'll share five further tips beyond authoring to make your assessments more trustable.
- SAP Managed Tags:
