Surveillance Considered Harmful

There is an interesting detail emerging from the attempted terrorist attack on Flydubai Flight 1073.

According to current reports, the co-pilot had already been barred from flying by Oman Air because of concerns about extremist views. Yet somehow he ended up flying for Flydubai. And somehow an Omani citizen, from a country whose pilots are not allowed to fly into Israel, ended up in the cockpit of a flight to Tel Aviv (Source: apnews.com/article/cf64a228799).

This happened in a region containing some of the most heavily surveilled societies on the planet. Saudi Arabia, where the aircraft eventually landed, is a particularly extreme example: internet activity, social media and private communications are subject to extensive state surveillance (Source: freedomhouse.org/country/saudi). Other Gulf states are not far behind in the scale and reach of their surveillance.

And yet here we are. This points to a more fundamental problem with surveillance as a weapon against terrorism.

What does a terrorist actually want?

Physical destruction is only the means. The real target is psychological: fear, distrust, disruption and ultimately the fragmentation of society. A terrorist cannot destroy a society through physical violence alone. But it may be sufficient to make that society afraid enough to start damaging itself.

And this is where mass surveillance becomes dangerously counterproductive. A functioning society depends on trust: trust between citizens, trust in institutions, and the assumption that most people around us are not our enemies. Terrorism attacks exactly that trust.

Mass surveillance does, too.

It tells everyone that everyone is potentially suspicious. It normalizes observation, suspicion and self-censorship. In trying to make society more resilient against individual terrorists, we risk weakening the very social cohesion that makes a society resilient in the first place.

The irony of Flight 1073 is therefore hard to miss.

We have built extraordinarily powerful machinery for watching millions of ordinary people, while apparently failing at the much more mundane task of asking why a pilot previously considered an extremism risk was sitting in the cockpit of an airliner bound for Israel.

Surveillance considered harmful. Not merely because it invades privacy. But because it can fail even at the trivial things while quietly weakening the society it was supposed to protect.

In this case, there is also a simple mathematical reason why mass surveillance works like shit.

You have three ingredients:

a very large population,
a tiny number of actual terrorists,
and a detection system with a measurable false-positive rate.

That combination is mathematically nasty.

Assume you monitor one million people and ten of them are actually terrorists. Now assume you have an extraordinarily good detection system: it identifies 90% of the terrorists and has a false-positive rate of only 0.1%.

It will correctly identify nine terrorists.

But among the 999,990 innocent people, a 0.1% false-positive rate produces roughly 1,000 false alarms.

So your supposedly excellent surveillance system gives you about 1,009 alerts, of which only nine are real. More than 99% of your alerts are false positives.

And this example is deliberately generous. A false-positive rate of just 0.1% is unrealistically low for a real-world system trying to infer something as vague and complicated as terrorist intent from communications, behavior, associations, travel patterns, social media activity and other surveillance data. At 1%, the same example would produce roughly 10,000 false positives for nine true positives. At 5%, it would produce nearly 50,000.

This is the base-rate problem. When the thing you are looking for is extremely rare, even a remarkably accurate detection system can produce vastly more false positives than true positives. In the real world, where detection is messy and the criteria themselves are ambiguous, the problem gets much worse.

And now you have created a second problem: alarm fatigue.

Someone has to investigate all those innocent people. Analysts learn that almost every alarm they see is bullshit. Attention becomes the scarce resource. The system designed to find the needle in the haystack responds to the difficulty of finding the needle by continuously adding more hay.

Mass surveillance therefore has a fundamental scaling problem: collecting more data is easy. Producing more alerts is easy. Producing more human attention is not.

Follow

@masek AI systems are clearly the way forwards here: with sufficient budget and training time we could easily flip those problematic rates to 0.1% detection and 90% false positive, you just watch...

Sign in to participate in the conversation
mastod1.ddns.net

Mastod1 be nice. (sorry, closed for new registrations after a bunch of 'commercial/spamming' accounts jumped in - rule 3 on site)