Trust and Safety has a Governance issue

My post contentTrust & Safety Has a Governance Problem

Trust & Safety has become very good at answering one question:

What do we do when things go wrong?

Remove the content.

Restrict the account.

Escalate the case.

Update the policy.

Retrain the model.

Add another control.

The machinery is sophisticated.

But there is a more important question that many organisations struggle to answer:

Who decides what level of risk is acceptable before something goes wrong?

That's a governance question.

And this is where Trust & Safety has a problem.

We built enforcement before we built accountability.

The evolution of T&S makes the problem understandable.

Platforms scaled fast, harm emerged faster than the organisations could anticipate it, T&S functions were therefore built around the immediate operational need: identify harmful behaviour, apply policy, manage escalations and respond to incidents.

This produced an impressive operational discipline.

Policies were written.

Reviewers were trained.

Quality programmes were introduced.

Detection systems were deployed.

Escalation pathways were formalised.

But operational maturity isn't governance maturity.

A team can have excellent detection, highly calibrated reviewers and sophisticated automation but still lack an answer to a basic question: when the evidence is ambiguous, who decides what risk the organisation is willing to accept?

That's a question not solved by another moderation queue.

T&S is often made responsible for risks it doesn't control

This creates a structural problem.

A product introduces a new feature.

The feature creates an abuse pathway.

T&S identifies the risk.

A policy is written.

Operations are instructed to enforce it.

An incident occurs.

T&S is asked why it wasn't prevented.

Notice what happened.

The organisation created the risk upstream but the responsibility for managing its consequences migrated downstream.

T&S becomes the risk absorber without necessarily becoming the risk owner.

Those are fundamentally different roles.

T&S shouldn't automatically own every risk it identifies.

Product should own product decisions, engineering technical controls, legal legal interpretation, security security risks, leadership enterprise level risk acceptance, and T&S should provide the specialised safety analysis, controls, evidence and challenge needed to make those decisions defensible.

The boundaries need to be explicit, otherwise accountability flows downhill until it reaches the person reviewing the case.

By then, the governance failure has already happened.

Policy isn't governance.

This distinction is frequently missed.

A policy tells people what should happen.

Governance determines:

* who has authority to decide;

* what evidence informs the decision;

* what risk is acceptable;

* who owns the residual risk;

* what controls are required;

when the decision must be revisited.

A policy can therefore be perfectly written while governance is weak.

The document exists, the workflow exists, the training exists, the audit exists, but nobody can answer who consciously accepted the remaining risk.

That's not governance.

That's risk by default.

A better model: govern the risk, not just the violation

A mature T&S function should connect 5 things.

1. Risk

What could happen?

Not simply: does this violate policy?

What harm could occur, to whom, through which mechanism, at what scale, and with what severity?

2. Decision rights

Who can decide what happens next?

T&S should have clearly defined authority to challenge and escalate material safety risks but should not become the dumping ground for unresolved organisational decisions.

3. Controls

What actually reduces the risk?

Controls should exist before, during and after deployment:

Before: threat modelling, abuse-case analysis and safety requirements.

During: detection, intervention, human review and escalation.

After: incident analysis, control redesign and verification.

4. Residual risk

What remains after the controls are applied?

Zero risk isn't a realistic operating condition for most platforms.

The real governance question is: who is consciously accepting the residual risk?

If the answer is nobody, the organisation hasn't made a decision, it's simply allowed the risk to persist.

5. Review triggers

What evidence would cause us to change our mind?

A spike in severe incidents.

A new abuse pattern.

A significant false-negative rate.

Disproportionate impact on a vulnerable population.

A material product change.

A new regulatory requirement.

These shouldn't merely become dashboard metrics.

They should become decision triggers.

That's the difference between monitoring risk and governing it.

The frontline is an intelligence layer.

There's another piece of this architecture that organisations routinely underuse.

The people closest to enforcement often see systemic problems before leadership does.

A reviewer notices a recurring ambiguity.

An investigator sees a new abuse pattern.

A quality analyst identifies a consistent failure mode.

An escalation specialist notices that several apparently unrelated incidents share the same underlying mechanism.

Individually these observations can look like edge cases, but collectively can be evidence of a governance problem.

The organisation therefore needs a mechanism that converts frontline signals into organisational decisions.

A recurring escalation should be capable of triggering policy review.

A new abuse pattern should be capable of triggering product review.

A persistent enforcement inconsistency should be capable of triggering governance review.

The frontline shouldn't just feed the enforcement machine.

It should help the organisation understand whether its assumptions about risk are still true.

AI makes this more urgent.

AI will automate more T&S decisions.

That doesn't make governance less important.

It makes decision ownership harder to ignore.

If an automated system makes millions of decisions, organisations need to know: what is it authorised to decide? What uncertainty can it tolerate? When must it defer? Who can override it? How is performance monitored? What happens when the policy changes? Who owns the consequences when the system systematically fails?

The future T&S architecture can't just be:

Human moderation → AI moderation

It needs to become:

Policy → risk assessment → control design → automated enforcement → human oversight → monitoring → governance decision

Automation changes the scale of enforcement.

It doesn't remove the need for accountability.

Stop asking for a seat at the table.

I think the T&S industry should retire one of its favourite ambitions: "T&S needs a seat at the table."

A seat is not the objective.

Decision accountability is.

A seat means little if T&S can raise a concern but cannot trigger escalation.

A dashboard means little if nobody has defined what its metrics should trigger.

A policy committee means little if product decisions can simply bypass it.

The better question is: which safety decisions exist, who has the authority to make them, what evidence must inform them, and who remains accountable after the decision?

That is governance.

And it's where T&S should be heading.

From enforcement function to governance discipline

The next evolution of Trust & Safety should not be measured solely by better detection, faster enforcement or larger moderation operations.

Those capabilities will remain necessary.

But the strategic opportunity is bigger.

T&S can become the organisational discipline that connects policy, product decisions, operational reality and measurable risk.

That means moving: from enforcement to control design, from policy ownership to risk accountability, from incident response to continuous risk monitoring, from metrics as reporting to metrics as decision triggers.

Ultimately, the question is not whether T&S gets invited into strategic conversations.

The question is whether the organisation has built a system in which safety risks are identified early, decisions are explicit, controls are proportionate, residual risk has an owner, and operational evidence can change the decision.

That's what governance looks like when safety is treated as an operational outcome.

And if T&S can build that system, it won't need to ask for a seat at the table.

It will be helping design the table.