“Have all the core data in place and be as data-driven as possible.” – Peter Manijak
Certification can be a very powerful element to feature in your learning program. Validating the knowledge you provide while raising your industry profile and building lifelong loyalty among an upskilled talent pool, a formal certification program marks you out as a serious player in your sector.
I Zoomed in to discuss certification with Peter Manijak, a pioneer and innovator in the world of credentialing and certification. Peter has built and managed some of the leading programs in the world today, such as EMC, Hitachi Data Systems and Magento, an Adobe Company. He has also consulted for major companies including AWS. We reached out to him to get his take on the best strategies to adopt – and the main pitfalls to avoid – when your organization sets out to develop or hone its certification program in today’s demanding training market.
We focused our conversation on leveraging the power of data, and how the insights provided can contribute to a more robust, scalable and successful certification program outcome.
Formal certification is a different ballgame to the training certificates which some companies use to drive interest, and Manijak pulls no punches about the difficulty of the task. “People who have no certification background are not likely to understand the effort, rigor, and resources required to produce a world-class global certification program.”
“It can take years to recover from a poorly written exam, and the blowback can be enough to sink a program and badly damage the reputation of an affiliated company or organization”
“Frankly, many training professionals do not understand either, and often struggle to put together a decent assessment or test. It can take years to recover from a poorly written exam, and the blowback can be enough to sink a program and badly damage the reputation of an affiliated company or organization.”
The stakes can be high when developing a formal certification strategy and program, but an effective framework need not take years to develop. With a clear aim, a sound plan and a good grip on the right data throughout the process you can be focused from the get-go, maximizing efficiencies and minimizing time to market.
Measuring what matters
“Establishing the parameters of success is very important, and realism should be a key feature of all your ambitions,” says Manijak. He goes on: “Clear and accurate data are needed to underpin your projections. When reviewing the framework of the program, there is a need to look at the goals, the exams, the tracks, the costs and the feedback. Have all the core data in place and be as data-driven as possible.”
“What matters gets measured. What’s measured, matters”
Taking a data-focused approach to results analysis, in both the pre- and post-release phases, pays dividends. “What matters gets measured. What’s measured, matters,” runs Manijak’s mantra.
Once the program is launched into the market, robust reporting insights and analytics are critical to monitoring, measuring, and managing program performance – both in-house and against the competition. “Review the dashboard to see how each exam is being consumed… How does the program compare to others? Bring in as much industry data as possible,” Manijak advises.
Perception of quality is a key metric in this regard. By definition, if you’re aiming to award a proof of excellence to top people in your field, “the key thing here is to protect the value of the word ‘certification,’” says Manijak. The market must be convinced your certification exam matches the expertise it is designed to validate and provides a secure testing platform.
Having target audience members rate the value of the expertise or knowledge you are validating is a key step in course design. Talking to top performers on what it takes to succeed in a role is the foundation to work your analysis from – you need to test what is relevant.
Analysis and adaptation
Before going to market, Manijak advises working with a mixed-ability test group and analyzing their performance: “Get a few top performers, a few novices, and a few midrange professionals. This will give you a very good idea whether the exam is on target.
“When a strong exam is in place, one which validates skills and knowledge, and the pass/fail mark is set accordingly, let the exam into the marketplace and plan on conducting an in-service analysis. Doing this after six months or after 100 published exams are taken would be a good guideline to follow. At the very least, in-service analysis should be conducted before any update to a test bank and/or test forms.
“The in-service data will show you how each item is performing. Are the top performers scoring well on the exam as expected? Are the novices getting the scores they should be getting? Are the borderline candidates getting borderline scores?”
There are many nuances involved, including whether the test takers have taken available training and the newness of a product or feature – but this is all valuable intelligence to feed back into the certification program, which can be adapted as required until the data shows it’s performing as planned.
Staying ahead of cheaters and test fatigue
Manijak’s view of what makes for a strong and secure certification exam has evolved in recent years. “I’m a big believer in scenario-based questions,” he says. “They’re very difficult to write. At the end of the day, if you create the type of questions where people have to think through the scenarios it validates their experience. When they pass, you feel comfortable they can do the job.
“From a security standpoint, people cannot memorize scenario-based questions with randomized answers to pass them on”
“From a security standpoint, people cannot memorize scenario-based questions with randomized answers to pass them on,” Manijak points out. “Years ago, I used to get very upset if someone got a hold of the exam questions. Now, I’m like, ‘If you get a hold of the exam questions and they’re all scenario-based, you’ve still got to figure it out…’ If they could actually learn all that stuff, that’s pretty good!
“Of course,” Manijak continues, “there are still guardrails to use, but exam creators’ energy should be spent on crafting and updating question banks.”
It’s a simple way of staying ahead of test cheaters and test cheat sites: never announce any updates to your exams. Simply refresh them. The test cheat sites will gradually fall behind and be ravaged by the people who purchased their illicit wares.
You should also take so-called “test fatigue” into account, warns Manijak, and be aware of when it’s time to update and/or refresh the test item bank. Test fatigue is not necessarily an indicator of test cheating, but over time people will have a better idea of what is on the exam and prepare better, or perhaps retake it. “A good rule of thumb is to refresh the item bank every 12-18 months, he suggests. “Always try to write ‘forever’ questions, which are not out of date in 30-90 days because of product updates.”
Data analysis should provide an extra level of security too: “Be sure that your test delivery includes automated flags for suspicious activity and set data parameters for your test vendor to flag, or for you to look for. Parameters could be an abnormally high or low score, or a time element, like spending next to no time on each question and then taking 50 minutes to respond to the last question.”
The bigger picture
Once a certification program is launched and established in the marketplace, the data it generates has a final crucial role to play: it’s a measure of success, tracking beyond financial ROI to paint a multi-dimensional picture of the contribution certification can make to a learning program and broader business performance.
Measuring the growth in number of exams taken and passed gives a clear picture of quarter to quarter and year on year growth, while monitoring associate-level exams taken will be an important indicator of future advanced participation from the audience pool. Having a sound marketing plan in place for certification initiatives will drive interaction with the learner community. Measuring community engagement can be a key indicator for the advantage of a quality program.
“When you’ve got a program to be proud of, make sure the market knows about it”
Finally, says Manijak, when you’ve got a program to be proud of, make sure the market knows about it . Certification Magazine has a yearly Salary Survey, which lists out the hottest certifications in the market and other valuable findings: it’s well worth getting involved. If your data stacks up – on course popularity, learner engagement, market perception, ROI and other key metrics – your certification program will tell its own powerful story.
