What AI Can and Cannot Tell You About a Suspicious Message
AI reads a suspicious message better than a blocklist ever could, and it still cannot know whether your supplier changed banks. An honest account of both sides.

Two messages. The first says your parcel is held pending a small customs charge and will be returned to sender in five days, with a link. The second says your supplier's bank account has changed following a treasury review, quotes the right invoice number, and asks that this month's payment go to the new details.
An AI reading the first one will do well. It knows this pattern: a small fee, a short deadline, a link that does not belong to the courier it claims to be. It can tell you what the tactic is and why it works.
An AI reading the second one is in trouble, and it is important to understand exactly why. The problem is not that the message is cleverly written. It is that whether it is a scam depends entirely on a fact outside the text: did your supplier change banks? Nothing in the words answers that. Not the grammar, not the tone, not the structure of the address. An honest tool has to say the thing it is least rewarded for saying, which is that it cannot tell.
Both of these are worth examining, because the gap between them is where most of the real argument about AI and scam detection lives.
What AI genuinely does better
The thing being replaced here is the blocklist, and the comparison is not close.
A blocklist answers "has anyone reported this before?" That is a useful question with a brutal limitation: every attack is new once. A domain registered two days ago, a scam running in a language the lists do not cover well, a message sent to four hundred people in one country and nowhere else. The blocklist says nothing, and saying nothing gets read as safe.
Reading the content changes the question to "what is this message trying to do?" That question has an answer even for something launched an hour ago.
In practice, a language model is good at a specific set of judgments:
- Naming the tactic. Manufactured urgency, impersonated authority, an advance fee, a fake payment confirmation, a request to move the conversation to another app, secrecy framing that discourages checking.
- Spotting structural deception. Link text that says one destination while the address goes elsewhere, a display name that does not match the sending address, a domain one character off a real brand.
- Working across languages and places. Because the tactics are universal even when the wording is not, the same analysis works for a mobile money scam in Accra and a tax refund scam in Manchester.
- Explaining itself. This is underrated. A verdict with the specific warning signs attached teaches the person something, which is worth more over time than the verdict itself.
That last point is why a plain-language explanation is the output, not a number alone. Someone who learns that a surprise salary change is bait chosen because nobody wants to miss it has gained something they keep.
What it cannot do, in order of how much it matters
1. It cannot verify the world
This is the ceiling, and no amount of model capability raises it. An AI reading a message has access to the message. It does not know your suppliers, your payroll schedule, whether your chief executive is in a meeting, or whether the courier really has a parcel for you. Any attack whose entire deception is a claim about reality sits outside what text analysis can settle.
That category includes most of the expensive ones: changed bank details, executive requests, payroll redirection, invoices that look exactly like your invoices because they were copied from your invoices. For those, the only reliable check is out of band. A phone number you already held. A walk to someone's desk. A login to the supplier portal rather than a reply to the email.
2. It can be wrong in both directions
False negatives happen with targeted attacks that contain no tactics to detect. A two-line email from a colleague's name asking whether you are at your desk is not suspicious as text. It is suspicious as context, and the context is not in the message.
False positives happen more often than people expect, and they matter because they erode trust. Real messages from banks and government services use genuine urgency, deadlines and alarming language, because the underlying situation is genuinely urgent. A system tuned to flag urgency will flag them. Our own FAQ states plainly that the tool can occasionally miss a sophisticated scam or flag something legitimate, and anyone claiming otherwise about any detector is describing marketing rather than behavior.
3. The content being analyzed is written by the adversary
This is the part most discussions of AI scam detection skip, and it is a real security property rather than a theoretical one.
When a tool reads a suspicious message, it is processing text authored by an attacker who may know the tool exists. That text can contain instructions aimed at the analyzer rather than the reader: assertions that the message has already been verified, attempts to redirect the task, phrasing designed to produce a reassuring verdict. OWASP lists prompt injection first among risks to applications built on language models, and it applies directly here.
The responses are architectural. Input is stripped of common injection phrasing before analysis. The analysis instructions live on the server rather than being sent by the page, so the endpoint cannot be repurposed into a general-purpose assistant. On the organization side, where reported emails are investigated in more depth, content under analysis is marked as untrusted throughout, and anything the system wants to change is queued as a proposal for a human to approve. An instruction hidden in a phishing email can, at most, produce a suggestion that a person reads and declines.
4. Confidence is not accuracy
Language models produce fluent output whether or not they have grounds for it. Fluency reads as authority, which is a problem when the subject is whether to send money.
The design response is to keep the model away from questions that can be answered exactly. In the organization console, email forensics runs in code with no model involved: authentication results, whether the visible link matches its real destination, attachment types and hashes, the delivery route. Those findings are exact and repeatable and state what they are based on. Threat intelligence lookups are scored in code with a stated reason for every point, and the result says which sources answered and which could not, so a clean result is never mistaken for a checked one. The model's job is narrower: read the language as a social engineer would, say what the lure is, quote the words that do the work, and say when the evidence does not support a conclusion.
That division is the whole trick. Use the model for judgment about meaning, which is what it is good at. Use code for facts, which is what code is good at.
5. Degraded input produces degraded answers
A screenshot has to become text before it can be read. When that extraction returns little or nothing, the honest behavior is to say so and ask for the text to be pasted instead, which is what our screenshot path does rather than producing a verdict on almost no information.
Checking something now? Paste the link, message or screenshot into CyberWatch AI Scan for a free read, and treat the result as one input rather than the decision.
The uncomfortable symmetry
The same technology is improving the attacks.
The NCSC's assessment of AI's near-term effect on the cyber threat expects more convincing social engineering, produced faster and at larger scale. The practical consequence is that the tells people were taught to look for are the first casualties. Bad spelling was never a security control; it was a side effect of attackers not bothering. AI-written phishing removes it, and spotting AI-generated scams by their style gets harder every quarter.
This cuts against a comforting conclusion. If detection and generation improve together, the long-term answer cannot be that people become better spotters. It has to be that the processes attackers target stop depending on anyone spotting anything: a verification step before payment details change, a reporting route that is one click, an expectation that nobody will be criticized for checking.
How to use a tool like this properly
Three rules, in order.
- Use it to understand, not to decide. The explanation of what the message is doing is the valuable part. It makes you better at the next one.
- Treat high risk as a stop and low risk as a maybe. The asymmetry is deliberate. A high score should end the interaction. A low score means nothing obviously wrong was found in the text, which is not the same as verified.
- Verify out of band before anything irreversible. Money leaving, credentials shared, access granted. Use a channel you established before the message arrived, never one the message supplies.
For the mechanics of using the scanner, see how to check a link, message or screenshot and how to check a link without clicking it. For what is currently arriving, the online scams guide is the broader picture.
Frequently asked questions
Can AI reliably tell whether a message is a scam?
It can reliably recognize the tactics of a scam: manufactured urgency, impersonation, lookalike web addresses, advance-fee structures. It cannot verify facts about the world, so it cannot know whether a specific request is genuine.
What is AI genuinely better at than a blocklist?
Anything new. Reading the content itself means a scam launched this morning, in any language, with a domain registered yesterday, is judged on what it says rather than on whether anyone has reported it yet.
What kinds of message does AI get wrong?
Two kinds. Highly targeted attacks with no obvious tactics, which read like ordinary business. And blunt but genuine messages, such as a real fraud alert from your bank, which use the same urgency a scam does.
Can a scam message manipulate the AI checking it?
It can try. Text being analyzed is written by the attacker, so it may contain instructions aimed at the model. That is why input is stripped of common injection phrasing and why the analysis is treated as untrusted content throughout.
Does the scanner open the link to see where it goes?
The free scanner reasons about the content and the structure of the address rather than browsing to it. Deeper checks, including threat intelligence lookups and email forensics, run on the organization side when a reported email is investigated.
Why does a screenshot sometimes give a weaker answer?
A screenshot is converted to text first. If that extraction fails or returns little, there is less to analyze, and the right response is to say so and ask for the text to be pasted instead of guessing.
Should a low-risk result be treated as permission?
No. Treat it as one input. Before money moves or credentials are shared, verify through a channel you already trust, such as a phone number you held beforehand.
Is AI making attacks better too?
Yes. National agencies expect more convincing social engineering at greater volume, which removes the clumsy-writing tell people were taught to rely on. That is an argument for verification habits rather than for better spotting.
Why does an honest tool need to say it is unsure?
Because a confidently wrong verdict teaches someone to stop checking. A verdict that states what it could not determine keeps the person in the decision, which is where they belong.
Sources
- AI Risk Management Framework, NIST
- AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST Computer Security Resource Center
- LLM01:2025 Prompt Injection, OWASP Gen AI Security Project
- OWASP Top 10 for Large Language Model Applications, OWASP Foundation
- The near-term impact of AI on the cyber threat, UK National Cyber Security Centre
- Thinking about the security of AI systems, UK National Cyber Security Centre
- Phishing scams: how to spot and report them, UK National Cyber Security Centre
- Recognize and Report Phishing, US Cybersecurity and Infrastructure Security Agency


