Blog/AI & the future

Detecting AI-Generated Text and Images: An Honest Assessment

AI content detectors are far less reliable than their scores suggest. What the evidence shows, why provenance beats detection, and what to do instead.

CyberWatch AILast reviewed 21 November 2026 · 7 min read
A person holding a smartphone showing a photograph, with a laptop open on the table behind

Someone pastes a supplier email into an AI detector. The tool returns "92% AI-generated". What should they do next?

That question has no good answer, and noticing why is more useful than any detector. The number is not a probability that the text was machine-written in any calibrated sense. It is a statistical judgment about how predictable the writing is, offered without a confidence interval, by a tool whose error rate on your kind of text nobody has measured. Act on it and you will sometimes be right. You will also sometimes accuse a colleague who writes plainly, or clear a message that was generated and lightly edited.

Detection is the wrong question. This article explains why, with as much evidence as exists, and then describes what to do instead.

How text detectors work, and why that limits them

Almost all text detectors measure statistical regularity. Language models generate text by repeatedly choosing likely next tokens, so their output tends to sit closer to the statistical center of a language than human writing does. Detectors estimate that closeness, usually under names like perplexity and burstiness, and convert it into a score.

Three consequences follow directly from the method, and no amount of engineering removes them.

Plain human writing looks machine-like. Formulaic business correspondence, technical documentation, and prose written by a competent non-native speaker all tend to be more predictable than literary writing. The false positives are therefore not randomly distributed. They concentrate on second-language writers and on anyone whose style is unadorned, which is an equity problem as much as an accuracy one.

Light editing defeats them. Rewriting a few sentences, varying the structure, or passing the text through a paraphraser moves it away from the statistical signature. The effort required is about a minute.

Short text is undecidable. Statistical measures need volume. A three-sentence email, which is exactly what matters in fraud, carries almost no signal.

NIST's generative AI evaluation program has been measuring this directly. Its pilot study on text-to-text evaluation tested both generators and discriminators and found that detection performance varied significantly depending on the systems involved, with some generators producing text that defeated most detectors. The framing in that work is informative: detection is treated as an open research problem with an ongoing evaluation, not as a solved capability with a product category.

Images, video and audio

The folklore here ages badly. Counting fingers, looking for garbled text in signs, checking whether earrings match: each of these worked for a particular generation of models and then stopped. Advice built on visible artifacts has a shelf life measured in months, and repeating it in training is worse than saying nothing, because it teaches people that careful looking is a sufficient control.

Technical approaches are more durable and still limited. Analysis of sensor noise patterns, compression history and frequency-domain inconsistencies can identify some generated or manipulated images. All of it degrades when a file is screenshotted, re-encoded, resized or passed through a social platform's processing pipeline, which is what happens to essentially every image that reaches you. By the time a suspicious photograph arrives in a messaging app, most of the forensic signal is gone.

For audio, the FTC ran a public challenge on voice cloning detection and protection, which is a reasonable indicator of where the field sits: worth organizing a competition about, not worth betting a payment process on. Live video is currently harder to fake convincingly in real time than audio, and the gap is narrowing, so treat "ask them to turn their camera on" as a delaying tactic rather than a verification method.

Provenance is a better idea than detection

There is a structurally different approach: instead of inspecting content for signs of synthesis, attach a verifiable record of how it was made.

The C2PA standard does this. A cooperating camera, editing tool or generator signs a manifest describing the asset's origin and the edits applied, and that manifest travels with the file. A viewer can check the signature and see the claim. The Content Authenticity Initiative is the industry effort pushing adoption into cameras and creative software.

This is genuinely better than detection because it is cryptographic rather than statistical. It also has a hard limitation that is worth stating plainly, because marketing tends to skip it: provenance tells you about files that have credentials, and says nothing about files that do not. Most content has none. Stripping a manifest is trivial, and happens accidentally whenever a file is screenshotted or re-saved. So a valid credential is positive evidence of origin, and the absence of one is no evidence at all.

That asymmetry makes provenance useful in a narrow, valuable way. It helps a news organization prove its own footage is authentic. It helps a brand sign its own imagery. It does not help you decide whether the voice note in your WhatsApp is really your finance director, because that file will never have a credential either way.

Watermarking sits in the same category. It can mark the output of a cooperating generator, it can be removed with modest effort, and it is silent on output from models that do not participate. Useful for platform-scale moderation and attribution. Not a control you can place in front of a payment.

What this means in practice

Do not build a decision on a detector score

If you are tempted to use detection output, write down in advance what you will do at each score and what the consequence of being wrong is. Most of the time this exercise kills the idea, which is the right outcome. A tool that produces an unverifiable probability in cases where you need a decision has added a source of false confidence to your process.

In HR, academic and disciplinary contexts, the stakes make this sharper. A score cannot distinguish a false positive from a true one in the individual case, which means it cannot support an accusation. Process evidence can: version history, drafts over time, the ability to discuss the work. Use that.

Change the question from content to request

The questions that still work do not involve inspecting the artifact at all.

  • Does this request fit a known process? Bank detail changes, urgent payments outside the normal run, credential resets and access grants have defined routes. A request arriving outside its route is the signal, regardless of how it was written.
  • Did it come through the channel that process uses? A finance instruction in a personal messaging app is anomalous whether or not the voice was real.
  • Has it been confirmed through a channel you chose? Call back on the number in your own directory, not one supplied in the message. This is the control that works against a perfect fake, because it never asks anyone to judge authenticity.

Our walkthrough on how to verify a caller is the operational version of that last point, and a written payment verification procedure is what stops it depending on whoever happens to be on duty.

Retire artifact-spotting from training

Awareness training that teaches people to look for odd grammar, strange hands or unnatural blinking is now actively harmful. It builds confidence in a skill that does not transfer, and it implies that a message which passes inspection is safe. Replace it with the shape of requests that always require verification. Our guide to security awareness training covers how to structure that, and the practical signals worth keeping are in spotting AI-generated scams.

Not sure about a message or link? Paste it into CyberWatch AI Scan for a free check.

A prediction, and what would prove it wrong

This is speculation, labeled as such. Prediction: general-purpose detection of AI-generated text will not become reliable enough to base consequential decisions on within the next three years. The reasoning is that detection and generation improve together, that light human editing is cheap, and that short text carries too little signal.

What would falsify it: an independent evaluation, run by a body with no product to sell, showing a detector sustaining a false positive rate below one percent on short, real-world business text across multiple generator families including ones published after the detector, with results reproduced by a second evaluator. If that appears, the argument above is wrong and should be dropped.

A narrower prediction with better odds: provenance credentials will become common in professional camera and creative workflows and remain rare in the content that reaches ordinary inboxes. That would be falsified by messaging and email platforms displaying and preserving content credentials by default for the majority of images users receive.

Where this fits

Detection failing is not a crisis, because detection was never the control. The defenses that matter against convincing fakes are the ones that never relied on recognition: authentication that cannot be replayed, money and access processes with a second channel built in, and enough logging to reconstruct what happened. That argument runs through our guide to AI-powered cyberattacks, and the organizational version for impersonation is in deepfake and voice-cloning fraud against organizations.

The honest position to hold internally is this. We cannot reliably tell whether a piece of content was generated. We are not going to try. We have built processes that do not need to know.

Frequently asked questions

Do AI text detectors work?

Not well enough to act on a single result. They produce a probability, not a finding, and they are wrong in both directions: genuine human writing gets flagged, and lightly edited AI text often passes. NIST's own evaluation of text discriminators found performance varied substantially between systems, with some generators defeating most detectors.

Why do detectors flag human writing as AI?

Most work by measuring how predictable the text is. Writing that is plain, formulaic or produced by someone writing in a second language is statistically predictable, so it scores as machine-like. That makes the errors land hardest on the people least able to contest them.

What about detecting AI images?

The obvious artifacts, hands, text in signs, inconsistent reflections, are closing fast and were never a reliable test. Technical analysis of compression and sensor noise still finds some generated images, and it is defeated by screenshotting, re-encoding or routine social media processing.

Is watermarking a solution?

Partially, and only for cooperating generators. Visible and invisible watermarks can be cropped, re-encoded or stripped, and a model that does not watermark produces unmarked output. Absence of a watermark proves nothing, which is the central weakness.

What are content credentials?

A signed record attached to a file describing how it was made and edited, standardized by the C2PA. It is provenance rather than detection: it tells you what a cooperating camera or tool asserted, and says nothing about a file with no credential attached.

Should we buy a deepfake detection product?

Only with a clear view of how you would act on an uncertain result, and never as the control protecting a payment or an access decision. A detector that is right most of the time is a poor substitute for a callback on a number you already hold.

Can we use detectors in HR or academic decisions?

Not as evidence. A probability score should never be the basis for an accusation against a person, because you cannot distinguish a false positive from a true one in the individual case. Use process evidence instead, such as drafts, version history and a conversation.

So what actually works?

Shift from asking whether content is generated to asking whether the request is authentic. Verify through a channel the sender did not choose, require a second approver for money and access, and build processes that do not depend on anyone judging a file's origin.

Sources

  1. 2024 NIST GenAI (Pilot Study): Text-to-Text Evaluation Overview and Results, NIST
  2. 2025 NIST GenAI (Pilot) Evaluation Plan for Image Discriminators, NIST
  3. C2PA Specifications, Coalition for Content Provenance and Authenticity
  4. C2PA: Verifying Media Content Sources, Coalition for Content Provenance and Authenticity
  5. Content Authenticity Initiative, Content Authenticity Initiative
  6. NSA, FBI, and CISA Release Cybersecurity Information Sheet on Deepfake Threats, CISA
  7. FTC Announces Winners of Voice Cloning Challenge, Federal Trade Commission
  8. The near-term impact of AI on the cyber threat, UK National Cyber Security Centre
  9. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1), NIST
  10. ENISA Threat Landscape 2025, ENISA
Share this article
LinkedIn WhatsApp X
For organizations

Your people get these messages at work too.

CyberWatch AI sends your staff realistic practice attacks, trains the gaps it finds, and shows management exactly where the organization stands.