« On V's Most Thought P… | Home | Why They Leak (Or Bet… »

Assumptions in a Haystack: Milton Friedman, J.M. Berger and the NSA

(To read the rest of "On Violence’s Most Thought Provoking Foreign Affairs Event of 2013", please click here.)

"A physicist, a chemist and an economist are stranded on an island, with nothing to eat. A can of soup washes ashore. The physicist says, "Lets smash the can open with a rock." The chemist says, "Let’s build a fire and heat the can first." The economist says, "Lets assume that we have a can-opener..."

Old Economist Joke

A long time back, on a topic completely unrelated to the NSA scandal, I found this link to a post on the Crooked Timber blog which described, “The Correct Way to Argue with Milton Friedman”.

In short, if you find yourself engaged in an argument with Milton Friedman, or a disciple, you usually find yourself accepting some initial, key assumption. If you accept this assumption, you will find yourself, several logical conclusions later, trapped in a losing position like a player losing a queen in chess. In the Crooked Timber post, they demolish the initial, key assumption that renters and landlords have equal power in a negotiation. On paper, they can both be profit-maximizing individuals. In reality, no one doubts that landlords have much, much more power than renters.     

This brings me to a widely-cited and referenced article published in Foreign Policy, called “Evil in a Haystack” by intelligence analyst J.M. Berger (of IntelWire.com), where Berger explains to the layperson how the NSA uses meta-data to stop terrorism.

While I love J.M. Berger’s work on the whole, I couldn’t help but think of the “The Correct Way to Argue with Milton Friedman” post when I read his article.

I’ll concede this: J.M. Berger accurately describes how the NSA goes about using meta-data. But let me make this shocking accusation: His description shows the single key flaw which undermines most intelligence agencies. Mainly, Berger presents an authoritative and unwavering belief in the accuracy of intelligence. Along the way, he presents a case study for “over-confidence bias” in action. And he does this all without ever thinking about the consequences to the people (Americans) who turn up in his searches. 

Berger starts by setting the scene:

“We start with a classic scenario. U.S. intelligence officials have captured an al Qaeda operative and obtained the phone number of an al Qaeda fundraiser in Yemen.”

When I read, “We start with a classic scenario...”, I see, “Let’s assume that...”.

Instead of saying, “intelligence officials believe” or, more accurately, “intelligence officials assume”, Berger has set the stage to show the efficacy of meta-data by giving the reader certainty: “We have captured an al Qaeda operative,” not “we have captured someone we believe is an al Qaeda operative.” Berger presents no doubt or hesitancy as to the identity of the suspects.

In real life, determining the facts is incredibly difficult.  Even determining an operative’s level of involvement is incredibly difficult. For instance, a CIA source could have fingered the suspect as an operative, but only did so in exchange for cash. Or under threat of blackmail. Or the person is an al Qaeda operative, but incredibly low on the totem pole. (Believe it or not, CIA bribes--er payments for information--usually escalate for information about more valuable people. This could incentivize the people giving the information to lie. I know, a liberal is bringing up incentives based on profit maximization but go with me here.)

You shouldn’t trust any intelligence analyst--or detective or district attorney or federal prosecutor or federal agent or military intelligence officer--who comes to you with absolute certainty. In psychology terms, it’s called the “over-confidence effect”. Studies show that whenever people have a “99% certainty”, they are often very wrong. In some extreme examples, people who rate their confidence as “99%” certain are right only 40% of the time. (Think political or sports forecasters if you want a daily repeating example.)

As a final point, this scenario hardly ever happens. I know Berger calls it a “classic”, but really intelligence analyst hardly ever come across a smoking gun to begin their investigation.

Nevertheless, the analyst proceeds to investigate the phone number, which leads to finding more suspects:

In our example data, the result is a list of 79 phone numbers that were involved in an incoming or outgoing call with the fundraiser's phone within the last 30 days. The fundraiser is a covert operator and this phone is dedicated to covert activities, so almost anyone who calls the number is a high-value target right out of the gate.

This is how bad intelligence happens. Berger’s analyst doesn’t just start with absolute certainty, he proceeds down an investigation and triples his assumptions:

    1. Assumes the target must be a covert, al Qaeda fundraiser.

    2. Assumes anyone who calls phone must also be a high-value target.

If the fictional analyst is correct, then he has indeed identified 79 new targets. If he is wrong, than 79 largely innocent people could now be investigated and added to intelligence databases. For instance, say the blackmailed or bribed “al Qaeda” member from above fingered a perfectly harmless Hawala business-owner. With all these assumptions, when will the intelligence analyst find out his mistake? A Hawala business-owner certainly acts like an al Qaeda financier, but with completely different ends.

Berger then proceeds to show how the analyst keeps plumbing at meta-data including expanding the size of the list of possible contacts, cross-referencing data in other databases, creating new reports on possible contacts, expanding the search to many American phone numbers (even with possible inaccuracies) and even initiating wiretaps on Americans, all without needing anything more than the word of a detainee in some foreign country (who may be under “enhanced interrogation”).

This expansive scope of government powers, based only on hunches and tips, really frightens strong civil liberties proponents like myself. It should frighten small government types as well. To assuage your fears, though, Berger trots out a favorite trope...

For one thing, U.S. policies are still informed by the idea that all terrorist attacks should be interdicted. A frequently expressed corollary to that premise states that, while tradeoffs against civil liberties might be bad in the abstract, those issues are meaningless when faced with a ticking time bomb…

I believe the NSA/intelligence community must have a guidebook which says, “When in doubt, bring up a ticking time bomb”. It also has the corollary, “If anyone questions your funding, bring up 9/11 (with personal example of what you were doing when you watched the planes hit for added emotional pull).” The issues with Berger’s analyst aren’t ticking time bombs, it’s about bad intelligence. And that bad intelligence violates the civil rights and liberties of Americans enshrined in the Constitution.

eleven comments

I doubt that landlords have much more power than renters. If they didn’t, there would never be specials, there wouldn’t be workout rooms, nothing new would ever get built and most important of all, there wouldn’t be free popcorn from 5 to 7 on Friday nights in the office.

Mr. Berger’s initial supposition undermines the whole case for indiscriminate data collection. He starts out his argument by saying, in effect, ‘Ok, we got this lead.’ He is saying that everything starts from a lead. My understanding of the collecting everything there is argument is that you can use some wonderous algorithm to sift through it all to generate leads so you don’t need a lead to start with. But even when trying to defend that circular reasoning Mr. Berger starts out by saying ‘Ok, we got this lead see, and…’.

There isn’t much objection by many people if all that tech capacity is used to investigate a lead generated by something that is able to be articulated. You tell that to the judge and away you go. It is very objectionable if you tell the judge let me go through everything and I’ll find something.

The other argument the police/surveillance state advocates like to use is that need to collect everything there is now inc case they have a need for it later. That is treating the entire mass of the Americans as active criminals whose every action needs to be recorded so the evidence can be piled on when they eventually come to the attention of the authorities. That is not the kind of thing I imagined when they first taught me the lyric “the land of the free and the home of the brave” those many years ago.

At least as far as police work goes, I think there is functional certainty. You got the forensic evidence. You got a confession. You got witnesses. Oh yeah, he did it all right. If you didn’t have that, nothing would ever come to trial.

Carl – Lots to unpack, but I will just handle your last paragraph. There are countless defendants who were convicted after “confessing”. They confessed after hours of coercion, sleep deprivation, threats, and the police basically wrote their confessions. Not everyone will confess, but more will confess than you would believe. We’ve written about that before.

What I mean more is that some police and attorneys make a judgement, then build the case, not the other way around. Not every detective, but again, much more then you think. Further, look at are series on Intelligence is Evidence, or any On V Updates where we show convictions being overturned. One of the repeatedly shocking/disturbing parts of human nature is that the attorneys refuse to overturn convictions once the evidence starts to pile up. That is the certainty that worries me. It is the lack of self-questioning that analysts of all stripes need to worry about.

Michael C:

I suppose I should have stipulated that I was speaking of good police and DA work done by honest men. Bad guys in uniforms or in the DA’s office are criminals themselves and produce bad cases.

It has been my personal experience that most all officers and DAs do the best they can and perform honorably. When the cases I was familiar with went forward, there was no doubt.

One of the situations that leads to really bad results is if you have politically charged case investigated by a dishonest officer, prosecuted by an ambitious DA and a defense attorney who is incompetent. No good that. That goes back to an article done by Chicago magazine back in the 90s about innocent men behind bars that was one of the factors that kicked the whole movement to use DNA to look at old cases.

Michael C:

Oh I forgot. About the sickening sight of a DA’s office or a PD refusing to admit they have made a mistake. I think that comes from the old fashioned sin of pride exacerbated by bureaucratic social dynamics. The two tend reinforce each other to the point where they will stand there in an open field at high noon on a cloudless day with their arms folded, their faces set hard and insist that, no the sun isn’t shining and it really is dark out. (Come to think of it, that sounds an awful lot like the US gov talking about our ‘ally’ the Pak Army/ISI.)

“(…)some police and attorneys make a judgement, then build the case, not the other way around.”

That’s how the Neonazi NSU murders were missed by German authorities for years. Somehow they had convinced themselves that the bomb was related to imaginary organized crime links of the victims.
Xenophobic extremists were an obvious alternative, but not considered to be worthy of investigation.

The biggest assumption in economics is that people behave rationally, meaning they seek to maximise personal benefit. That’s likely an equally problematic assumption in intelligence work.

It’s good to see people understanding that the only way to debate people like Friedman is to start with the axiom that they’re always wrong, always lying, and you just have to find the lie. Because once you have that axiom (proven by the example you give), you can dispense with actually finding the lie. It’s proven to always be there, and to always be sufficient to utterly demolish anything you disagree with. That’s an axiom.

It’s pretty cool to be the guys who are axiomatically infallible.

@ Steve Brompton – We didn’t write any of that. We didn’t write anything close to “they’re always wrong, always lying, and you just have to find the lie.” We used the word “lie” once, in relation to CIA operatives in a parenthetical. We were discussing arguing and assumptions.

I think the opening of the post was a lot more nuanced than you think.

I think that the human process of forensic computer science investigation is just as inherently screwed as normal criminal investigations. In law enforcement, how many investigative sources, “leads”, are incorrectly pursued and unfortunately criminally indicted before an alternative “person of interest” is eyeballed? Wrong time, wrong place discipline. Coincidence or not, no one with a phone (or potential motive to commit a crime against a person) is impervious to link analysis. The ethical issue with meta-data is exactly that — it’s a shit-ton amount of data. Not bad data. Not good data. Just data. The fact that J.M. Berger has to create an assumption of illegal activity to begin a formal investigative process before analyzing/interpreting and disseminating data is very, very frightening.

Hey everybody. I’ve been reading this blog now for sometime and I have been enjoying it very much. I’m a former infantry grunt and, though I don’t always agree, I have found everything here very much worthy of discussion. Thanks for the hard work fellows.

This post is very thought provoking on more than one level.I agree with your assessment of Berger’s arguments and deconstruction of his assumption(s). One further note, all arguments (I think) start from some axiomatical or premise assumption. The reason many people get turned around when somebody offers a valid argument is because most do not realize the difference between logical validity and having a sound premise. One can offer a argument that is perfectly valid yet that fails to have a sound premise. In fact, in today’s world of highly analytic philosophy (think Saul Kripke, Alvin Plantinga, Thomas Nagel…) most of the philosophy debates come down to debating the soundness of the premises rather than the validity of the logic.

As a PhD student in economics, I am getting the opportunity to see first hand how rigorous the arguments are that touch economics. Of the social sciences, they really have done a good job of bringing in sharp, analytical logic and rigorous math underpinnings in the theory. This doesn’t make the axioms of completeness and transitivity necessarily sound but it does provide a useful framework to conduct modeling. Also, a great deal of work is going on in behavioral economics to tweak the basic theory in cases empirical results are showing people do not always make rational decisions as the models predict. Lots of interesting work going on there to be sure. But, some of the assumptions that are made (such as by Friedman) deserve being examined and pointed out. The weaknesses in arguments are always at the assumptions or axioms. How strong are they? How weak? In the end, we will still have to come back to some assumption, however, even if its an assumption that you can understand what I am writing or indeed exist to read it!

Have a good one folks and please forgive any errors in the above.