How AI Content Detectors Actually Work (and Why They Sometimes Get It Wrong)

Every week someone asks us the same question: "How does an AI detector actually know?" Teachers ask it about essays, recruiters about cover letters, and developers about the content flowing through their apps.

The honest answer is that no detector knows. It estimates. In this post we'll walk through the main techniques behind AI content detection, show a small Python experiment you can run yourself, and explain why every detector, including ours, sometimes gets it wrong.

Large language models generate text by repeatedly picking likely next tokens. As a result, AI-written text tends to be very predictable to another language model.

Perplexity measures that predictability. Low perplexity means the model saw each next word coming. Human writing usually has more unexpected word choices, odd phrasing and personal detail, so its perplexity tends to be higher.

Humans are inconsistent. We mix a long, winding sentence with a short one. We get vague, then very specific. Burstiness captures how much predictability varies from sentence to sentence.

AI text is often uniformly smooth: every sentence has similar length, structure and perplexity. Human text tends to be "burstier".

Here's a minimal experiment using the open GPT-2 model from Hugging Face. It's not a production detector, but it shows the core idea.

# pip install torch transformers import re import torch from transformers import GPT2LMHeadModel, GPT2TokenizerFast tokenizer = GPT2TokenizerFast. from_pretrained ( " gpt2 " ) model = GPT2LMHeadModel. from_pretrained ( " gpt2 " ) model. eval () def perplexity ( text: str ) -> float: enc = tokenizer ( text, return_tensors = " pt " ) with torch. no_grad (): out = model ( ** enc, labels = enc [ " input_ids " ]) return torch. exp ( out. loss ). item () def burstiness ( text: str ) -> float: sentences = [ s for s in re. split ( r " (?<=[.!?])\s+ ", text ) if len ( s. split ()) > 4 ] if len ( sentences ) < 2: return 0.0 scores = [ perplexity ( s ) for s in sentences ] mean = sum ( scores ) / len ( scores ) std = ( sum (( x - mean ) ** 2 for x in scores ) / len ( scores )) ** 0.5 return std / mean # coefficient of variation sample = " Paste a few paragraphs here and compare a human-written text with an AI-written one. " print ( f " Perplexity: { perplexity ( sample ):. 1 f } " ) print ( f " Burstiness: { burstiness ( sample ):.

2 f } " ) Enter fullscreen mode Exit fullscreen mode Run it on a paragraph you wrote and on a paragraph from a chatbot. You'll often see the AI text score lower on both. Then try a formal, textbook-style human paragraph, and you'll see why these signals alone are not enough.

Modern detectors don't rely on two hand-made numbers. They train a classifier, often a fine-tuned transformer, on large sets of human-written and AI-written text from many models. The classifier learns subtler patterns: word-choice distributions, how ideas are connected, repetition, and stylistic habits of specific models.

The catch is that classifiers are only as good as their training data. New models, new prompting styles and paraphrasing tools shift the distribution, so detectors need constant retraining.

Instead of guessing after the fact, some AI providers can watermark output at generation time by subtly biasing token choices in a pattern only a key holder can detect. Google's SynthID is one example. Watermarks are promising, but they only work for models that implement them, and heavy editing or paraphrasing can weaken them.

Images are a different problem. Common approaches include:

This is the part we think every developer integrating detection should understand.

That's why we design ProofAI to return a confidence score, not a verdict, and why we recommend using it as one input alongside human judgment, drafts, version history or a conversation with the writer.

If you're building edtech, HR, publishing or moderation tools, detection usually lives behind a simple API call. Here's what a request to the ProofAI API looks like:

curl -X POST https://ai.proofai.in/api/v1/detect \ -H "x-api-key: YOUR_API_KEY" \ -H "Origin: YOUR_DOMAIN" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d 'content=Your text here' Enter fullscreen mode Exit fullscreen mode A few integration tips, whichever provider you use:

We'll go deeper on the integration side in a future post.

AI detection is a useful tool, but it's statistics, not magic. Understanding perplexity, burstiness, classifiers, watermarks and image artifacts helps you build features that are useful and fair.

If you want to experiment, you can try the free detector at proofai.in (no login needed) or ask us about API access.

What would you want to see next? A deep dive into image detection, a full API integration tutorial, or a benchmark of how detectors handle edited AI text? Let us know in the comments. 👇

Disclosure: we're the team behind ProofAI at 4Born Solutions. This article was drafted with AI assistance and reviewed and edited by our team.