Measuring the effectiveness of security awareness training can be difficult. How do you know if you're making an impact? In this post we share some thoughts on how to test and measure how well your security awareness program is doing.
Sustained behavior change
This is what Anouk Vermeeren, our senior learning consultant, considers the main point of Infosequre’s cybersecurity culture scan.
The security awareness culture measuring tool is meant to be a sort of pulse-check of an organization. Unlike a standard test, it cannot be passed or failed, and instead relies on its results to give the company a better grasp of their standing and progress on their path to security awareness.
“We want to help organizations measure their current culture, how people think about information security, what they think is important to their colleagues, to their manager, to themselves, and whether or not they’re able to work securely. It gives organizations a lot of insight on how they’re doing and how they can improve.”
The culture scan measures 3 main parameters: people’s attitudes towards performing secure behavior, the norm in the organization and the control people have. The combination of these 3 factors is coupled with a set of open questions for each of them. “For example, if the attitude parameter is quite high compared to the control parameter, the employees think it’s important to behave securely, however they don’t have the ability to do so, or at least not as much as they would like to. From there we can reference the answers to the open questions in the control section, to get further insight as to why that may be.”
The process
The newest measuring tool has been in the works for a while now, a project helmed by Anouk herself with the aid of other Infosequre colleagues, built over many hours of research, trials and feedback runs.
“We started from the theory of planned behavior. Based on it, we made a set of open questions and had a target group of people answer them. This gave us insight on the important subject to touch on in the culture scan and based on this investigation we developed a set of statements to be rated from completely disagree to completely agree. This next set was also tested on a wider range of people from varied backgrounds and then taken, along with their answers, through a series of statistical and analytical measurements, making sure the information is trustworthy and valid. Based on this we made a few final adjustments and had the end product.”
A visually striking result
The end report is made in Microsoft Power Bi, which enables us to combine and measure results in countless ways, giving shape to a full dashboard. The results are visually represented in customizable graphs, sortable by department, age, position and other parameters related to the employee or to the measured behaviour.
On the dashboard there is also the option of viewing word clouds created from the open-ended questions, a feature Anouk is especially proud of. “You see all kinds of positive words! Even for us, and we’re not the company itself, is very encouraging to see that a lot of people mention the same things. It really gives an amazing visual perspective on the common ground the employees have.”
The missing piece in the puzzle of security awareness
A security awareness program is a combination of many elements, each with its role to play on the road to the desired result. The cybersecurity culture scan fits perfectly in that puzzle.
“Measurable behavior change. That’s the ultimate goal of a good security awareness campaign: to achieve good, secure behavior. But you can’t measure that only by a knowledge test. Of course, it’s part of it, but just like doing a phishing simulation by itself is not enough, measuring only knowledge doesn’t give you the complete picture. The combination of parameters is worth a lot.”
“A lot of organizations start a security awareness campaign with the idea “we have to do something, we have to inform and educate our people” which we always say is very good. But this tool helps organizations to stop, look around, and take account of everything they have already. So it helps them a lot to not just start, but start smartly.”
