Mark Twain notoriously advised, “First get your facts; then you can distort them at your leisure.” Today, we have access to more facts to distort than ever before. In fact, more data has been created (and collected) in the last two years than in the entire history of the human race!
Being able to collect learning data holds significant promise for better learning experiences and ROI on training investments. Training managers and instructional designers also have more concrete information to work with than ever before. But there’s a catch – too often, data is poorly interpreted. And poorly interpreted data has no actionable value.
Learning insights are often referred to by other names, like learning analytics, or learning metrics. With the right goals and techniques in mind, you can gather all of the above and, most importantly, translate training insights into action that improves training for your learners and organization.
Why Gather Learning Insights?
The two big promises of learning insights are to help you make solid decisions, based on relevant data, and to demonstrate the value of your training. Improving this value using data-driven decisions then becomes a virtuous cycle.
Sounds great, right? Let’s take a closer look at these two core benefits.
1. Data-driven decision making
eLearning is a developing science, and this means there are both exciting opportunities for innovation, and some tricky, unknown challenges. For what we don’t yet know, it becomes somewhat necessary to rely on assumptions – even when they’re rooted in best practice – to design and deliver online training.
Hard facts can bridge the gap between assumption and informed decision, and help you design courses that are better suited to your audience, learning outcomes, and organizational needs.
The majority of decisions made using learning insights will directly affect learners and course design in the following ways.
Impact on learners: By using metrics that focus on individual performance and its relation to learning goals, learning resources, and the study habits of learners, you can improve:
● Learner retention
● Completion rates
● Learner engagement, and
● Facilitation quality
Impact on course design: Using metrics that focus on the social activity of learners, the development of knowledge and skill from easier to more difficult learning outcomes, the flow of the curriculum, and the feedback loops built into the course for assessment and communication, you can improve:
● Course design to better suit the audience’s preferences
● Personalization and adaptability
● Timing of course components in relation to learners’ habits
Learning insights can also be used to create models that predict the success or failure of learners on a course – but more on that later.
2. Improved ROI
Training is an investment, and learning insights are the proof of the success of that investment. While the changes in learner behavior that training aims to achieve can be pretty hard to measure, online learning insights can prove the effectiveness of your training in other ways.
For example, by comparing insights about course completion and assessment results, learning insights can demonstrate that a new course had an improved ROI over a previous iteration.
Data, Analytics Or Insights?
Now, before we get into the practical steps on how to gather effective learning insights, let’s make sure we’ve got the lingo down.
Data includes the raw figures, responses and statistics gathered from a Learning Management System’s (LMS) reporting and the metrics we set. Metrics are the questions or standards of measurement we use to gather data, like assessment completion rates. Unfortunately, data is an overwhelming jumble of nonsense by itself.
That’s why analytics are so important – they make sense of your data by finding trends and patterns. Analytics are a huge step towards getting value from your data, but they’re not actionable by themselves.
Insights are where you access the value of analytics. The insights gained through analytics should relate more directly to decision-making and improvement.
Here’s a simple learning insights example.
Let’s say that you have a list of the number of times learners signed into a course, how they signed in, and what they did when they signed in. This is your data. By analyzing this data, you find that in the past month, there were 400 sign-ins, half of which were using a mobile phone, and 40 of which were to access the course assessments. The key insight from this analysis is that only 10% of learners sign on using a mobile device to complete assessments.
You can see how the decision to make assessments more mobile-friendly could flow from this data.
Golden Rules for Gathering and Using Learning Insights
Many organizations struggle to make truly data-driven decisions. This is because they’re not using actionable learning metrics. Here are our four golden rules for making that sure you get all the value you can from your learning data.
1. Know why you’re collecting data
How do you get the insights you want? Start with knowing why you want those insights.
We’ve previously recommended that you try the SMART goals system to plan your analytics; so here’s a quick refresher. In most training contexts, learning insights can serve one or many of the following.
● Prediction: Identifying learners at risk of dropout or course failure, as well as predicting the overall success of a type of design or delivery.
● Personalization: Giving learners customized learning pathways, and/or assessments.
● Intervention: Providing instructors and facilitators with information about learner performance for improved support.
● Information visualization and communication: Dashboards that visually display an overview of learning data (per learner or course).
Not all of these will be equally important to every organization. So understand which insights are most useful to different stakeholders, and then tailor your metrics and training insights to those needs.
2. Evaluate learning success using tried and tested models
The standard post-training evaluation model is the Kirkpatrick model. This model is organized around four levels that translate into data collection.
According to this model, there are four categories of data that should be collected as a minimum:
1. Learner reaction: What learners thought about the training. Use surveys, course and facilitator ratings, and course completion statistics.
2. Learning: Changes in knowledge or skill. Measure learner performance through assessments results and course pass rates.
3. Behavior: The extent of behavior and skill improvement in the application. Measure and compare performance in assessments, and use manager surveys pre- and post-training.
4. Results: The effects on the business or environment resulting from the learner’s performance. Measure changes in efficiency through reduced work hours, faster turnaround, less rework, etc.
Start with these metrics, and then make them more specific to your needs with custom LMS metrics and reporting tweaks. For example, if you’re using a new quiz assessment, and your metrics show that many learners are leaving it incomplete, a secondary metric can be used to measure how long each learner spent on the assessment. Maybe the activity was too time-consuming?
3. Emphasize action upfront
If you can’t identify what action to take next using your learning insights, the chances are you didn’t have a clear goal in mind when you started measuring or analyzing your data.
Make sure your intended insights are actionable by phrasing them as questions. Where can the course content improve? How and when can we spot struggling learners? Is the course material appropriate to learners’ knowledge and skill levels? Is this training increasing organizational performance? Or, how is the online course reducing operating costs and increasing revenue?
4. Beware the vanity metric
Finally, be wary of only seeing what you want to see. To avoid this, steer clear of what is known as “vanity metrics”, i.e. metrics that show off what’s working, such as high course completion rates, and hide what isn’t working, like, low course pass rates.
Remember that data is only helpful if you’re asking the right questions, and you’re prepared to take a hard look at the results.
Using Your LMS to Support Learning Insights
If you’re using a quality LMS, it’s likely already set up to automatically track and record all the data you need to get powerful learning insights from your online training.
If your LMS offers custom metrics to get the exact information you need to gather actionable insights, and easy-to-use reporting features and tools like surveys and reporting dashboards, you can take even better steps toward improving the learning experience and ROI of your training.
