Blog/Inside CyberWatch AI

How Our Security Awareness Approach Works: Simulate, Find the Gap, Train It, Measure

The method behind CyberWatch AI training: a safe simulation to find the real gaps, a short lesson aimed at each gap, and a measure of whether it moved.

CyberWatch AI8 November 2026 · 7 min read
Colleagues looking at a shared screen in a small meeting room

Most security awareness programs run in the wrong order. They begin with content, deliver it to everyone, record who finished, and only then wonder whether it changed anything. By the time a question about effectiveness comes up, the budget is spent and the only available answer is a completion percentage.

We run it the other way around: find out what is actually wrong before deciding what to teach. Four steps, repeating, with each step existing because the previous one produced a finding.

Step one: find out where people actually stand

The first thing that happens is a simulation, not a lesson.

A safe, authorized phishing email goes to a department or to everyone, and the result is a reading: who opened it, who clicked, who reported it, and how long each of those took. That reading is the thing training everyone on everything is designed to avoid producing, because it is uncomfortable and specific.

Three design decisions make the reading honest.

The email does not announce itself. A simulation with a "this is a test" footer or its own report button measures nothing, because the person is no longer in the situation you are trying to observe. Our templates carry a plain signature and nothing else. The lure reads like the email it imitates, and the only thing the recipient sees is the email.

Reporting happens through the real channel. If a simulation hands people a special button, you have measured whether they can use a button that will not exist during a real attack. Reports go through the organization's standing reporting route instead, and a report there is matched back to the open campaign. What gets measured is the behavior that will actually be available on the day.

Nothing collects a credential. A simulation measures whether somebody clicks. Nothing in the library asks for a password, a card number or a code, and the code that assembles an AI-written lure refuses any draft containing that kind of request. There is no justification for harvesting real passwords from your own staff to prove a point they already concede.

The library itself is organized by the lure rather than the technology: credential harvesting, payment and invoice fraud, internal and HR pretexts, collaboration and file sharing, consumer and delivery notices, and emails dressed as security warnings. Each carries a difficulty, where an easy one has visible tells and a hard one is internal, specific and plausible. That matters for interpreting results, which is the same reason NIST built the Phish Scale: a click rate without the difficulty of the lure attached is a number that cannot be compared with anything.

An administrator approves every campaign before it sends. We also recommend telling staff that testing happens, before the first one lands. A program that depends on employees not knowing it exists is a program that will not survive its first complaint.

Step two: train the gap, not the catalogue

The reading from step one points somewhere. The point of the method is to go there.

Someone who clicked a fake e-signature request has demonstrated a specific gap: they treated an e-signature notification about their own pay as routine. The useful response is a three-minute lesson on that pattern, delivered while it still means something, not a forty-slide compliance deck in March.

So the click leads to a page that names the tells in the email they just clicked, addressed to them by first name. For a lure written for that organization, the page lists that email's own tells rather than general advice: the sender domain that is not the real one, the bait chosen because nobody wants to miss it, the fact that signing a document should never require a work password. Then a short lesson on that category is assigned.

The library is built for this shape of delivery. Modules run three to five minutes and cover ten skill areas: phishing detection, payment and invoice fraud, social engineering, passwords and MFA, mobile and remote work, AI-enabled threats, data protection, workplace security, safe browsing and incident response. Someone who misses a payment-fraud lure gets payment-fraud modules at the level that fits them. Someone who has that area cold spends their attention elsewhere.

Behind this is a per-person model that tracks mastery across those ten skills, moving with every piece of evidence: what they did with each simulation, weighted by how hard the lure was, every assessment scenario they answered, their quiz results, and the real threats they reported. Older evidence counts for less, so the model reflects who someone is now rather than who they were a year ago. From it comes a plan: the next modules, weakest skills first, and the next simulation, aimed at the skill least proven and at a difficulty they are ready for.

One deliberate limit: the AI can draft a lesson for a specific person based on the lure they fell for, and an administrator decides whether to send it. Nothing reaches an employee automatically. A system that mails people on its own judgment will eventually mail somebody something indefensible.

Step three: test again, differently

A single simulation establishes a baseline. A program is what happens next.

The second round should not be the same lure. People who learned to distrust a parcel notice have learned about parcel notices, which is a narrow and perishable skill. Varying the pretext, the difficulty and the audience is what builds the general habit of checking, and it also prevents the quiet drift toward easy tests that makes a dashboard look good while nothing improves.

This is also where assessment scenarios earn their place alongside simulations. A simulation measures what somebody did with one email. A scenario can put them in situations an email cannot: a supplier whose bank details changed, a colleague's name on an approval request, a call claiming to be from IT. The scenarios describe a situation and ask what you do, because asking "what is phishing?" measures whether somebody read a glossary. Selection is adaptive and deliberately mild, weighted toward categories a person keeps missing but spread across categories, because an assessment that only ever asks about your worst area stops measuring the rest.

Pilot and practice rounds can be excluded from scoring. If an administrator was told a round would not count, it has to actually not count, or the score stops being something anyone trusts.

Step four: measure whether it moved

The last step is the one that distinguishes a program from an activity.

Completion is an input. The output worth reporting is a change in behavior, which means comparing click and report rates in a skill area before and after people trained in that area. That comparison is unflattering sometimes, and it is the only thing that tells you whether the previous three steps were worth doing.

The views that matter in practice:

  • Click rate by department, because an organization-wide average hides the concentration you need to act on.
  • Report rate and time to first report, because reporting protects everyone and speed decides whether a warning arrives before or after the damage.
  • A department-by-skill view, which shows that operations is weak on delivery lures while finance is weak on supplier changes, and those need different lessons.
  • Repeat patterns, meaning people who clicked more than one simulation or missed the same category twice. A handful of those is a specific, fixable finding.
  • One score over months, assembled from the measured signals and weighted, with any dimension that has no data behind it reported as unmeasured rather than filled in.

That score is the thing a management team can follow, and the detail underneath it is what survives being questioned.

Want the employee side of this? Anyone can paste a suspicious link, message or screenshot into CyberWatch AI Scan for a free check before acting on it.

The part that is about trust, not software

Everything above fails in an organization where being tested feels like being hunted.

If a click produces an email to someone's manager, people will start hiding clicks. If reports go unanswered, people stop reporting. If a name goes on a company-wide list, the next real attack will be quietly deleted instead of reported, and your most important detection channel will have been closed by your own program.

Which is why leaderboards are anonymous by default, why nobody is named to the whole company for one click, why everyone who reports something hears the outcome, and why an employee's private check is never visible to an administrator. The NCSC's board guidance makes the same point from the other direction: a culture where people fear admitting a mistake produces silence, and silence is exactly what the attacker needs.

For the arguments underneath this method, see what makes awareness training work, how simulations should be built and how to run them without damaging trust. For a department where this pays off fastest, training staff on payment fraud is usually the place to start, and the phishing guide covers what you are preparing them for.

Frequently asked questions

What are the four steps?

Baseline with a safe simulation to see where people actually stand, train the specific gaps it exposes, test again on different lures, and measure whether the numbers moved. The loop runs all year rather than once.

Why start with a test instead of training?

Because training everyone on everything wastes the attention you need later. A first simulation tells you which teams and which lures are the real problem, so the training that follows is aimed somewhere.

Do your simulation emails say they are a test?

No. A lure that announces itself teaches nothing. The email reads like the attack it imitates, and the person only learns it was a simulation after they click or report it.

How do employees report a simulation?

Through the same reporting channel they would use for a real suspicious email. A report there is matched back to the open campaign, so reporting behavior is measured on the route that actually exists.

What happens when someone clicks?

They see a page naming the specific tells in that email, addressed to them by first name, and a short lesson on that kind of attack is assigned. No credentials are collected and nothing they typed is kept.

How long are the lessons?

Three to five minutes each. The library covers ten skill areas, and someone who clicks a payment-fraud lure gets the payment-fraud lesson rather than the full course.

Does the AI send training by itself?

No. It can draft a lesson for a specific person based on the lure they fell for, and an administrator decides whether to send it. Nothing reaches an employee without a human approving it.

How do you know whether the training worked?

By comparing behavior in a skill area before and after people trained in it. Completion is an input, not a result, and the result worth reporting is a change in click and report rates.

How often should simulations run?

Regularly and varied, not constantly. The aim is practice that keeps skills current without becoming background noise people stop taking seriously.

Sources

  1. SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program, NIST Computer Security Resource Center
  2. Phishing attacks: defending your organisation, UK National Cyber Security Centre
  3. The Phish Scale: NIST-Developed Method Helps IT Staff See Why Users Click on Fraudulent Emails, NIST
  4. Recognize and Report Phishing, US Cybersecurity and Infrastructure Security Agency
  5. Cyber Security Toolkit for Boards: Developing a positive cyber security culture, UK National Cyber Security Centre
  6. Phishing Guidance: Stopping the Attack Cycle at Phase One, US Cybersecurity and Infrastructure Security Agency
  7. Phishing, Technique T1566, MITRE ATT&CK
  8. Awareness and Cyber Hygiene, European Union Agency for Cybersecurity
Share this article
LinkedIn WhatsApp X
For organizations

Your people get these messages at work too.

CyberWatch AI sends your staff realistic practice attacks, trains the gaps it finds, and shows management exactly where the organization stands.