Your Phishing Simulation Is Measuring the Wrong Thing 

Every quarter, a security team somewhere sends a fake phishing email to the entire company, waits a week, and pulls a report. The click rate has usually gone down since last quarter. Someone builds a slide for the board: 18% down to 4% over the year. The number is treated as proof that the organization is more secure than it was twelve months ago.  

Almost nobody in the room asks what the number actually measured.  

The Click Rate Illusion: What the Metric Actually Measures  

The implicit assumption behind every phishing simulation program is that click rate tracks organizational resilience: fewer clicks this quarter means the workforce got better at recognizing threats. Researchers at NIST have spent several years complicating that assumption. Steves, Greene, and Theofanos (2020), in the paper that introduced what is now called the Phish Scale, argue that click rates should be expected to vary primarily with how well a phishing email's premise aligns with the recipient's actual job context, not with how "aware" the recipient is in some general sense. An invoice-approval lure sent to finance staff will outperform the same lure sent to engineers, regardless of how much training either group received. A declining quarterly click rate, in other words, is at least as likely to reflect a vendor quietly reusing easier templates, or a security team calibrating difficulty downward to protect the number, as it is to reflect a workforce that has become harder to fool.  

The more direct challenge comes from a 2026 reproduction study by Rozema and Davis, conducted at a financial technology firm with 12,511 employees, one of the largest of its kind. Comparing training modalities in a live operational setting and grounding the analysis in the NIST Phish Scale, the authors found no statistically significant effect of training interventions on either click rates (p = .450) or reporting rates (p = .417). The Phish Scale itself predicted behavior far more reliably than any training intervention did. If the exercise a company runs to justify its awareness budget has no measurable effect on the behavior it claims to change, the audit is not testing the organization. It is testing the difficulty setting the vendor chose that quarter.  

Learning the Test, Not the Threat  

Kirlappos and Sasse argued over a decade ago that phishing education tends to fail for a specific reason: it corrects the wrong misconceptions, teaching people to check indicators they don't actually understand rather than challenging the mental shortcuts that make them vulnerable in the first place. Applied to recurring simulations, the same dynamic produces a familiar pattern. Employees don't necessarily get better at spotting phishing. They get better at spotting the simulation: the sender domain the vendor always uses, the branded "gotcha" landing page, the fact that these things tend to arrive on a Tuesday morning. That is pattern recognition aimed at the test, not the threat.  

There is a neurological version of this same problem. Anderson, Vance, Kirwan, Jenkins, and Eargle (2016) used fMRI and longitudinal field data to show that the brain's attentional response to a repeated warning stimulus habituates measurably over time; only warnings that change their appearance resist this decline. A quarterly simulation that reuses a narrow set of formats is, from a neurobiological standpoint, training employees to stop paying attention to it specifically, which is a different achievement than training them to stay alert to novel lures they haven't seen before.  

The Shame Tax  

Most programs still layer a punitive structure on top of this shaky metric: public leaderboards by department, "you failed the test" emails cc'd to a manager, mandatory remedial training assigned like a penalty. The design logic is intuitive. It is also very likely counterproductive, for reasons that have nothing to do with phishing specifically.  

Edmondson's (1999) research on psychological safety in work teams found that environments perceived as blame-oriented suppress the disclosure of errors, because admitting a mistake carries a social cost that outweighs the benefit of surfacing it early. A phishing audit that publicly names people who clicked is, structurally, exactly this kind of environment. The behavior most worth protecting is not "never click" - a sufficiently well-crafted lure will get a percentage of any workforce regardless of training, a fact both the Phish Scale and the Rozema and Davis reproduction support - it is reporting quickly once something looks wrong. Punitive programs optimize for the metric that is hardest to move and, in the process, tax the one behavior that is genuinely trainable and valuable at any click rate.  

What the Data Says Actually Works  

The literature is not entirely bleak about interventions; it is specific about which ones work. Kumaraguru et al. (2007), in the study that introduced embedded training through the PhishGuru system, found that just-in-time feedback delivered at the moment someone clicks a simulated link outperforms generic training scheduled independently of any mistake. A longer-term study by Lain, Kostiainen, and Capkun (2021), tracking phishing behavior across organizations over an extended period, similarly points toward organizational and contextual factors carrying more explanatory weight than an individual's training history. The common thread across the research that does show an effect is timing and specificity, not the existence of a recurring test.  

Toward an Audit Worth Running  

None of this argues for abandoning phishing simulations. It argues for redesigning what they optimize for.  

Report rate, not click rate, should be the headline metric, paired with the Phish Scale difficulty rating of the campaign so numbers are comparable quarter to quarter rather than quietly inflated by easier templates. Feedback belongs at the moment of the click, not in a module assigned days later, matching what the embedded-training research actually supports. Individual results should stay out of public view; aggregate data can inform where training investment goes, but naming names reintroduces the exact blame dynamic that suppresses reporting. And the simulation should be treated as one input into a broader resilience posture rather than the control itself, because it is testing a single human decision made under time pressure, and no training program gets that decision right one hundred percent of the time. The layer that matters most is the one that assumes some fraction of employees will click regardless of how the program is run, and is built to catch what happens next inside the network rather than to prevent the click in the first place.  

The number in next quarter's board deck is not, on its own, evidence of anything. The more useful question is what happens in the six seconds after someone clicks, and whether the organization has built a culture where saying so costs nothing.  

References:  

Anderson, B. B., Vance, A., Kirwan, C. B., Jenkins, J. L., & Eargle, D. (2016). From warning to wallpaper: Why the brain habituates to security warnings and what can be done about it. Journal of Management Information Systems, 33(3), 713–743.  

Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350–383.  

Kirlappos, I., & Sasse, M. A. (2012). Security education against phishing: A modest proposal for a major rethink. IEEE Security & Privacy, 10(2), 24–32.  

Kumaraguru, P., Rhee, Y., Acquisti, A., Cranor, L. F., Hong, J., & Nunge, E. (2007). Protecting people from phishing: The design and evaluation of an embedded training email system. Proceedings of the 2007 CHI Conference on Human Factors in Computing Systems, 905–914.  

Lain, D., Kostiainen, K., & Capkun, S. (2021). Phishing in organizations: Findings from a large-scale and long-term study. arXiv:2112.07498.  

Rozema, A. T., & Davis, J. C. (2026). Anti-phishing training (still) does not work: A large-scale reproduction of phishing training inefficacy grounded in the NIST Phish Scale. Proceedings of the ACM Web Conference 2026.  

Steves, M., Greene, K., & Theofanos, M. (2020). Categorizing human phishing difficulty: A Phish Scale. Journal of Cybersecurity, 6(1), tyaa009.