Who answers for an automated moderation decision
A classifier removes a post. Or mutes an account, or hides a message, or takes down a video. Nobody chose it; a threshold was crossed. Multiply that by the volume any large community produces in an evening and you have thousands of small decisions a day that nobody made and everybody owns.
The interesting question is not whether the model is accurate. It is what happens on the day someone asks why.
The 30-second version
- Automated moderation is not a loophole in European law — it is a disclosure obligation. If a decision was taken by automated means, that has to be said.
- An appeal cannot be decided solely by automation. Someone qualified has to supervise it.
- The cost of a false positive is not the removed post. It is the unanswerable question afterwards.
- Most of what makes this survivable is not model work. It is logging, thresholds and routing — ordinary operations.
The two errors are not symmetrical
Every moderation system trades one error against the other. Let the threshold up and harmful content stays visible longer. Pull it down and you take out people who did nothing wrong.
Both are bad, but they are not bad in the same way, and the difference matters when you set the dial. Missed harmful content damages the people exposed to it, and the platform usually finds out through reports — there is a channel, and the harm is visible. A false positive damages one person quietly, and the platform often never finds out at all, because most people who get wrongly muted do not appeal. They conclude the place is arbitrary and they go elsewhere.
Which means the metric that reassures everybody — precision on a labelled test set — says almost nothing about the second kind of damage. The number that matters is the one nobody collects by default: of the decisions nobody appealed, how many were wrong?
What European law actually asks
The Digital Services Act, Regulation (EU) 2022/2065, is the text that governs this in the EU, and it is less about what you moderate than about how you account for it. Four articles carry most of the weight:
| Article | What it requires |
|---|---|
| Art. 14 | Terms and conditions must set out your moderation policies and procedures — including any algorithmic decision-making and human review |
| Art. 16 | A notice-and-action mechanism, so anyone can flag content and get a decision |
| Art. 17 | A statement of reasons to the affected user for each restriction — stating whether automated means were used, both in detection and in the decision |
| Art. 20 | A free internal complaint-handling system, where decisions are not taken solely by automated means but supervised by appropriately qualified staff |
Read those together and the architecture the regulation assumes becomes obvious. It does not forbid automation. It requires that automation be declared, that it be reversible, and that a person stand behind the reversal. Those are three separate engineering requirements, and the third one is an operations requirement, not a model one.
Notice how closely that mirrors the logic of the AI Act's transparency rules, which I wrote about in the piece on Article 50: the legislator is relaxed about a machine producing the output, and strict about whether a human answers for it.
The sentence that does the damage
"Your content was removed for violating our Community Guidelines."
That sentence is where most trust is lost, and it fails on its own terms. It does not say which rule. It does not say which part of the content. It does not say whether a human looked. And it leaves the person with nothing to act on but a grievance.
A statement of reasons that works says four things: what was restricted, which specific rule it was matched against, whether the decision was automated, and what the person can do next. It is longer by two lines. It converts an arbitrary event into a procedure, and people argue with procedures far more calmly than they argue with walls.
There is also a quieter benefit, and it is the one I would put in a business case: every well-formed statement of reasons is a labelled data point about your own system. A vague one teaches you nothing.
Appeals are an operation, not a form
This is where a moderation programme usually fails, and it fails for reasons that have nothing to do with AI. The appeal queue is the lowest-prestige queue in the building. It is staffed last, measured least, and it is the only place where the system's own errors are visible.
What it needs is what any service queue needs, which is my whole professional déformation:
- An SLA that is real, and different by severity. An account suspension and a hidden message should not sit in the same queue at the same priority.
- A routing rule that does not send the appeal back to whoever — or whatever — made the original decision.
- Qualified reviewers, with the context the automated system did not have: the thread, the history, the language, the joke.
- An overturn rate that someone reads on purpose. A category with a high overturn rate is not an appeals problem; it is a threshold that is wrong, and it is cheaper to fix at the threshold.
That last point is the one I would defend hardest in a steering committee. The appeals queue is the only honest feedback loop a moderation system has. Treating it as a cost centre to be minimised is how an organisation arranges never to learn that its classifier is drifting.
What has to be reconstructable
Six months after a decision, someone will ask about it: a regulator, a journalist, a client, a court, or just a very persistent user. What you need at that moment is a record, not a recollection. For each decision, that means being able to say:
- What was decided and when, to the second.
- By what — which model, which version, which threshold, or which human.
- Against which rule, in the version of the rules that was in force that day. Policies change; a decision has to be judged against the policy it was taken under.
- What the person was told, in the exact words they received.
- What happened next — appealed or not, upheld or overturned, and by whom.
None of this is exotic. It is the same evidence discipline that any regulated industry runs on, and I learned it somewhere with no algorithms in it at all: a hospital under Joint Commission accreditation, where no system went live until you could show who decided what, when, and on what information.
Before a model goes anywhere near production
The evaluation that matters is not the one that produces a headline accuracy figure. It is the one that answers awkward questions in writing:
- Does performance hold per language? A classifier trained mostly on English will quietly be a worse system for your Portuguese-speaking community, and the aggregate number will hide it.
- Does it hold per community? Reclaimed slurs, in-group banter and competitive trash talk — in a game, a forum or a comments section — are the classic false-positive factory.
- What is the error profile, not the error rate — who gets hit wrongly, and is it always the same kind of person?
- What is the fallback when the model is unavailable or degraded? Fail-open and fail-closed are both defensible; not having decided is not.
And the one I keep coming back to: what would make us turn this off? If there is no answer, it is not a controlled deployment. It is a hope.
Why this is a management problem
Everything above can be bought as technology and will still fail, because the hard parts are a threshold nobody owns, a queue nobody staffs, a policy version nobody logged, and a metric nobody reads. Those are not AI problems. They are the ordinary problems of running a service operation at volume, in several languages, around the clock — with the extra property that the mistakes land on people rather than on tickets.
Which is, in the end, the reason I find this subject worth writing about. The interesting part of AI governance was never the model. It is whether the organisation around it can still explain itself six months later.
This article is general information, not legal advice. The reference text is Regulation (EU) 2022/2065 (Digital Services Act), in particular Articles 14, 16, 17 and 20. For a specific decision with legal consequences, consult a qualified lawyer.
About this article
iaetica.cat is a space for writing in the open: what I am learning about artificial intelligence, its ethics and its law.
— Laura Graells