How AI content moderation works — and where it fails

Short answer. Automated moderation does not filter content; it scores it. A classifier gives a number for how closely a text or image resembles examples of a category, and a language model gives an opinion against written instructions. Neither knows whether a statement is true, or whether a person is identifiable to your audience. The useful systems therefore sort the queue and leave the uncertain middle to a person.

Automated moderation is often described as if it were a filter: content goes in, bad content is caught, good content passes. Anyone who has run a queue knows it is nothing like that. The tools produce scores and opinions, they are wrong in predictable ways, and how useful they are depends almost entirely on what you do with their output.

We build and operate a moderation pipeline, so this guide is written from the inside. It explains what the tools are doing, why they fail, and how to use them without fooling yourself.

Two kinds of machine

Nearly every moderation system today is built from two types of model, often used together.

Classifiers are trained for one narrow job: given a piece of text or an image, produce a number between 0 and 1 for each of a fixed list of categories — sexual content, harassment, hate, violence, self-harm and so on. They are fast and cheap, and they give the same answer every time for the same input. What they cannot do is explain themselves or handle a category nobody trained them on.

Large language models — the same technology as chat assistants — can be given written instructions: "Does this text accuse an identifiable person of wrongdoing? Answer yes or no and say why." They handle nuance, context and instructions that no classifier was built for. They are slower and more expensive, they can be inconsistent between runs, and they sometimes state wrong conclusions with complete confidence.

A classifier tells you how much a text resembles the abusive examples it was trained on. A language model tells you what it thinks about the text. Neither one knows whether the text is true.

Scores are not verdicts

A classifier returning 0.83 for "harassment" has not found harassment. It has reported that the text looks, statistically, like things that were labelled harassment in its training data. Somebody has to decide what number counts as too high. That number is the threshold, and choosing it is a trade-off with no correct answer.

Set the threshold low and you catch more genuinely harmful content, but you also flag more innocent content — false positives. Set it high and fewer innocent posts are flagged, but more harmful ones slip through — false negatives.

Which mistake is worse depends on what happens next. If a flag only sends the post to a human for a closer look, a false positive costs thirty seconds of someone's time, and a low threshold is sensible. If a flag deletes the post automatically, a false positive silences someone who did nothing wrong, and you should be far more cautious.

This is why the most useful systems do not block; they sort. Clear cases at either end are dealt with quickly, and the uncertain middle is sent to a person with the scores attached.

Why text is harder than it looks

Language mixing. Most moderation models were trained mainly on English. Performance drops for other languages and drops further when languages are mixed within a sentence — which is how a great many Malaysians actually write. A sentence that moves between English, Malay, Mandarin and Hokkien particles, in romanised spelling, may be read correctly by a model or may be read as noise.

Slang moves faster than training data. Insults, in-jokes and coded terms change every few months. A model trained last year has never seen this year's vocabulary.

Context decides meaning. "I'm going to kill him" after a football match and the same sentence after a description of being stalked are different things. So are a slur used as an attack and the same word quoted by the person it was aimed at. Classifiers see the words; they see little of the situation.

Deliberate evasion. People who want to get past a filter will substitute characters, add spaces, use lookalike letters, or put the text inside an image. Simple systems are defeated by simple tricks.

Sarcasm and reported speech. "Oh sure, she's definitely the most honest person on campus" contains no flaggable word at all.

What AI cannot know

Some of the most important moderation questions are not about language at all, and no model can answer them from the text alone.

• Is it true? A calm, specific, well-written accusation scores as perfectly safe. It may also be defamatory. Truth is a fact about the world, not a property of the sentence.
• Is this person identifiable to your audience? "The guy who runs the photocopy shop near the east gate" contains no name. Everybody on your campus knows exactly who it means. A model has never been to your campus.
• Is this person a public figure? Software can sometimes detect that a name is present. It cannot reliably tell a government minister from a classmate with the same name.
• Did this person consent? A photo of two people at a party might have been shared with their blessing, or might be the cruellest thing in your queue this month.

These are questions for a human who knows the community. The most a tool can do is notice that the question needs asking — for example, by flagging that a submission appears to contain a personal name — and route it to someone who can decide. See When a submission names someone.

Images

Image moderation typically combines several checks:

• An explicit-content classifier, which scores how likely an image is to be sexual. These are reasonably good at photographs and noticeably weaker with drawings, unusual lighting, and images that are suggestive without being explicit. Swimwear, medical images and people who are simply shirtless produce regular false positives.
• A vision-capable language model, which can be asked to describe the image and judge it against written rules — useful for violence, weapons, or drug use.
• Text extraction, to read words inside screenshots and memes so they can be checked like any other text. Without this step, a rule-breaking paragraph pasted into an image passes untouched.

What image models are generally not doing is protecting privacy. A clear photo of a stranger's face, a student ID or a car plate is not "unsafe" in any category these models know. That check is still yours — see Moderating images and video.

Video

A video is a long sequence of images with a soundtrack, and it is expensive to examine completely. Most systems sample frames — one every second or every few seconds — and classify each. Two consequences follow.

First, something that appears for less time than the gap between samples can be missed entirely. Second, a system that only checks the thumbnail or the first frame is barely checking the video at all, and you should find out whether yours does more than that.

Audio is a separate matter. Unless a tool transcribes speech and moderates the transcript, whatever is said in a video is not being checked. Many tools do not.

Why more than one model helps

Different models make different mistakes. A classifier misses a novel insult that a language model understands; a language model waves through an image that a purpose-built classifier scores as explicit. Running several independent checks and letting any one of them raise a flag catches more than the best single model — at the cost of more flags to review. For a queue where a human reviews everything flagged, that is usually the right trade.

The important word is independent. Two products built on the same underlying model will share the same blind spots.

Questions to ask about any moderation tool

1. What exactly does it check? Text only? Images? Text inside images? Every frame of a video or only a thumbnail? Audio?
2. Which languages was it tested on? If the answer is vague, assume English.
3. What happens when it flags something? Does it block, hold for review, or merely label?
4. Can you see why? A tool that shows which check fired and what it scored lets you learn its habits. A bare "rejected" teaches you nothing.
5. What happens when it is unavailable? If the service is down, do submissions go through unchecked, or wait?
6. Where does the content go? Submissions sent to a third-party service for checking have left your hands. Find out whether that service stores what it receives or uses it for training.
7. Can you adjust the thresholds? A fixed setting designed for a children's game is wrong for a university page, and the reverse.

Using it well

Treat automated moderation as a tireless first reader who is good at spotting the obvious and has no judgement about anything else. Let it clear what is plainly fine, stop what is plainly not, and attach its reasoning to everything in between. Then read the middle yourself.

Once a month, look at what it flagged that you approved and what it passed that you declined. Those two lists show you its blind spots, and they are different for every community. A team that reads them will catch what the machine misses. A team that trusts the green tick will eventually publish something it should not have — and "the AI approved it" has never been a defence to anything.

Common questions

How does AI content moderation work?

With two kinds of model, often used together. A classifier produces a score between 0 and 1 for each of a fixed list of categories, such as harassment or sexual content. A large language model is given written instructions and asked to judge the content against them. Someone then chooses a threshold above which a score counts as a flag.

Why does AI moderation flag innocent posts and miss harmful ones?

Because a score is not a verdict. A low threshold catches more harmful content and flags more innocent content; a high one does the reverse. Classifiers also see the words and little of the situation, so slang, sarcasm, quoted speech and deliberate evasion all fool them.

What can AI moderation not detect?

Whether a statement is true, whether a person is identifiable to your particular audience, whether someone is a public figure or a classmate with the same name, and whether the people in a photo consented. These are questions for a human who knows the community; the most a tool can do is flag that the question needs asking.

Does AI moderation work for Malay and mixed-language posts?

Less well. Most moderation models were trained mainly on English, performance drops for other languages, and it drops further when languages are mixed within a sentence — which is how a great many Malaysians write. Ask which languages a tool was tested on, and assume English if the answer is vague.

How is video moderated automatically?

Most systems sample frames — one every second or every few seconds — and classify each. Something shown for less time than the gap between samples can be missed, and a system that checks only the thumbnail is barely checking the video. Unless a tool transcribes speech, whatever is said in the video is not checked at all.

Is it better to use more than one moderation model?

Usually, where a person reviews everything flagged. Different models make different mistakes, so several independent checks, any one of which can raise a flag, catch more than the best single model, at the cost of more flags to review. They must be independent: two products built on the same underlying model share the same blind spots.