« "I'm allergic to microwaves. They release space hamsters into my bloodstream." | Main | "Stranger things have happened!" "Only two spring to mind, Lister: the spontaneous combustion of the Mayor of Warsaw in 1546 and that incident in 12th century Burgundy when it rained herring." »
July 12, 2006
"Realize these things are being done by primates—by apes dressed up in funny 'customes' [sic], like chimps who drive motorcycles in circuses—[...] it's quite astounding that the apes can handle the machinery and walk upright and so on."
md5 reverse lookup is like the Rainbow Table but for MD5—that is, it's a big hash table.
MD5 sums (see RFC 1321 - The MD5 Message-Digest Algorithm) are used as a one-way hash of data. Due to the nature of the formula used, it is impossible to reverse it. To find out what the source data was, one would traditionally attempt every single possible input value until they found the solution.
Even on fast machines this is difficult. To find the input value for a single MD5 will take some two weeks.
Using the disk space / speed trade-off, this database was started.
A helpful professor has provided definitive mathematical proof of the inefficacy of the NSA's domestic spying programs. via
What is the probability that people are terrorists given that NSA's mass surveillance identifies them as terrorists? If the probability is zero (p=0.00), then they certainly are not terrorists, and NSA was wasting resources and damaging the lives of innocent citizens. If the probability is one (p=1.00), then they definitely are terrorists, and NSA has saved the day. If the probability is fifty-fifty (p=0.50), that is the same as guessing the flip of a coin. The conditional probability that people are terrorists given that the NSA surveillance system says they are, that had better be very near to one (p=1.00) and very far from zero (p=0.00).
The mathematics of conditional probability were figured out by the Scottish logician Thomas Bayes. If you Google "Bayes' Theorem", you will get more than a million hits. Bayes' Theorem is taught in all elementary statistics classes. Everyone at NSA certainly knows Bayes' Theorem.
To know if mass surveillance will work, Bayes' theorem requires three estimations:
1. The base-rate for terrorists, i.e. what proportion of the population are terrorists;
2. The accuracy rate, i.e., the probability that real terrorists will be identified by NSA;
3. The misidentification rate, i.e., the probability that innocent citizens will be misidentified by NSA as terrorists.
No matter how sophisticated and super-duper are NSA's methods for identifying terrorists, no matter how big and fast are NSA's computers, NSA's accuracy rate will never be 100% and their misidentification rate will never be 0%. That fact, plus the extremely low base-rate for terrorists, means it is logically impossible for mass surveillance to be an effective way to find terrorists.
The US Census shows that there are about 300 million people living in the USA.
Suppose that there are 1,000 terrorists there as well, which is probably a high estimate. The base-rate would be 1 terrorist per 300,000 people. In percentages, that is .00033%, which is way less than 1%. Suppose that NSA surveillance has an accuracy rate of .40, which means that 40% of real terrorists in the USA will be identified by NSA's monitoring of everyone's email and phone calls. This is probably a high estimate, considering that terrorists are doing their best to avoid detection. There is no evidence thus far that NSA has been so successful at finding terrorists. And suppose NSA's misidentification rate is .0001, which means that .01% of innocent people will be misidentified as terrorists, at least until they are investigated, detained and interrogated. Note that .01% of the US population is 30,000 people. With these suppositions, then the probability that people are terrorists given that NSA's system of surveillance identifies them as terrorists is only p=0.0132, which is near zero, very far from one. Ergo, NSA's surveillance system is useless for finding terrorists.
Suppose that NSA's system is more accurate than .40, let's say, .70, which means that 70% of terrorists in the USA will be found by mass monitoring of phone calls and email messages. Then, by Bayes' Theorem, the probability that a person is a terrorist if targeted by NSA is still only p=0.0228, which is near zero, far from one, and useless.
Suppose that NSA's system is really, really, really good, really, really good, with an accuracy rate of .90, and a misidentification rate of .00001, which means that only 3,000 innocent people are misidentified as terrorists. With these suppositions, then the probability that people are terrorists given that NSA's system of surveillance identifies them as terrorists is only p=0.2308, which is far from one and well below flipping a coin. NSA's domestic monitoring of everyone's email and phone calls is useless for finding terrorists.
NSA knows this. Bayes' Theorem is elementary common knowledge. So, why does NSA spy on Americans knowing it's not possible to find terrorists that way? Mass surveillance of the entire population is logically sensible only if there is a higher base-rate.
Instead of presuming there are 1,000 terrorists in the USA, presume there are 1 million terrorists. Americans have gone paranoid before, for example, during the McCarthyism era of the 1950s. Imagining a million terrorists in America puts the base-rate at .00333, and now the probability that a person is a terrorist given that NSA's system identifies them is p=.99, which is near certainty. But only if you are paranoid. If NSA's surveillance requires a presumption of a million terrorists, and if in fact there are only 100 or only 10, then a lot of innocent people are going to be misidentified and confidently mislabeled as terrorists.
Also, mass surveillance of the entire population is logically plausible if NSA's domestic spying is not looking for terrorists, but looking for something else, something that is not so rare as terrorists. For example, the May 19 Fox News opinion poll of 900 registered voters found that 30% dislike the Bush administration so much they want him impeached. If NSA were monitoring email and phone calls to identify pro-impeachment people, and if the accuracy rate were .90 and the error rate were .01, then the probability that people are pro-impeachment given that NSA surveillance system identified them as such, would be p=.98, which is coming close to certainty (p=1.00). Mass surveillance by NSA of all Americans' phone calls and emails would be very effective for domestic political intelligence.
Monkeys are statistically indistinguishable from most stock-market investors. via
The capuchin is a New World monkey, brown and cute, the size of a scrawny year-old human baby plus a long tail. ''The capuchin has a small brain, and it's pretty much focused on food and sex,'' says Keith Chen, a Yale economist who, along with Laurie Santos, a psychologist, is exploiting these natural desires -- well, the desire for food at least -- to teach the capuchins to buy grapes, apples and Jell-O. ''You should really think of a capuchin as a bottomless stomach of want,'' Chen says. ''You can feed them marshmallows all day, they'll throw up and then come back for more.''
The Harvard monkeys were cotton-top tamarins, and the experiments with them concerned altruism. Two monkeys faced each other in adjoining cages, each equipped with a lever that would release a marshmallow into the other monkey's cage. The only way for one monkey to get a marshmallow was for the other monkey to pull its lever. So pulling the lever was to some degree an act of altruism, or at least of strategic cooperation.
The tamarins were fairly cooperative but still showed a healthy amount of self-interest: over repeated encounters with fellow monkeys, the typical tamarin pulled the lever about 40 percent of the time. Then Hauser and Chen heightened the drama. They conditioned one tamarin to always pull the lever (thus creating an altruistic stooge) and another to never pull the lever (thus creating a selfish jerk). The stooge and the jerk were then sent to play the game with the other tamarins. The stooge blithely pulled her lever over and over, never failing to dump a marshmallow into the other monkey's cage. Initially, the other monkeys responded in kind, pulling their own levers 50 percent of the time. But once they figured out that their partner was a pushover (like a parent who buys her kid a toy on every outing whether the kid is a saint or a devil), their rate of reciprocation dropped to 30 percent -- lower than the original average rate. The selfish jerk, meanwhile, was punished even worse. Once her reputation was established, whenever she was led into the experimenting chamber, the other tamarins ''would just go nuts,'' Chen recalls. ''They'd throw their feces at the wall, walk into the corner and sit on their hands, kind of sulk.''
The essential idea was to give a monkey a dollar and see what it did with it. The currency Chen settled on was a silver disc, one inch in diameter, with a hole in the middle -- ''kind of like Chinese money,'' he says. It took several months of rudimentary repetition to teach the monkeys that these tokens were valuable as a means of exchange for a treat and would be similarly valuable the next day. Having gained that understanding, a capuchin would then be presented with 12 tokens on a tray and have to decide how many to surrender for, say, Jell-O cubes versus grapes. This first step allowed each capuchin to reveal its preferences and to grasp the concept of budgeting.
Then Chen introduced price shocks and wealth shocks. If, for instance, the price of Jell-O fell (two cubes instead of one per token), would the capuchin buy more Jell-O and fewer grapes? The capuchins responded rationally to tests like this -- that is, they responded the way most readers of The Times would respond. In economist-speak, the capuchins adhered to the rules of utility maximization and price theory: when the price of something falls, people tend to buy more of it.
Chen next introduced a pair of gambling games and set out to determine which one the monkeys preferred. In the first game, the capuchin was given one grape and, dependent on a coin flip, either retained the original grape or won a bonus grape. In the second game, the capuchin started out owning the bonus grape and, once again dependent on a coin flip, either kept the two grapes or lost one. These two games are in fact the same gamble, with identical odds, but one is framed as a potential win and the other as a potential loss.
How did the capuchins react? They far preferred to take a gamble on the potential gain than the potential loss. This is not what an economics textbook would predict. The laws of economics state that these two gambles, because they represent such small stakes, should be treated equally.
So, does Chen's gambling experiment simply reveal the cognitive limitations of his small-brained subjects? Perhaps not. In similar experiments, it turns out that humans tend to make the same type of irrational decision at a nearly identical rate. Documenting this phenomenon, known as loss aversion, is what helped the psychologist Daniel Kahneman win a Nobel Prize in economics. The data generated by the capuchin monkeys, Chen says, ''make them statistically indistinguishable from most stock-market investors.''
During a recent capuchin experiment that used cucumbers as treats, a research assistant happened to slice the cucumber into discs instead of cubes, as was typical. One capuchin picked up a slice, started to eat it and then ran over to a researcher to see if he could ''buy'' something sweeter with it. To the capuchin, a round slice of cucumber bore enough resemblance to Chen's silver tokens to seem like another piece of currency.
Then there is the stealing. Santos has observed that the monkeys never deliberately save any money, but they do sometimes purloin a token or two during an experiment. All seven monkeys live in a communal main chamber of about 750 cubic feet. For experiments, one capuchin at a time is let into a smaller testing chamber next door. Once, a capuchin in the testing chamber picked up an entire tray of tokens, flung them into the main chamber and then scurried in after them -- a combination jailbreak and bank heist -- which led to a chaotic scene in which the human researchers had to rush into the main chamber and offer food bribes for the tokens, a reinforcement that in effect encouraged more stealing.
Something else happened during that chaotic scene, something that convinced Chen of the monkeys' true grasp of money. Perhaps the most distinguishing characteristic of money, after all, is its fungibility, the fact that it can be used to buy not just food but anything. During the chaos in the monkey cage, Chen saw something out of the corner of his eye that he would later try to play down but in his heart of hearts he knew to be true. What he witnessed was probably the first observed exchange of money for sex in the history of monkeykind. (Further proof that the monkeys truly understood money: the monkey who was paid for sex immediately traded the token in for a grape.)
Posted by Jon Rubin at July 12, 2006 10:37 PM
Trackback Pings
TrackBack URL for this entry:
http://www.ubiquit.us/movabletype/mt-tb.cgi/207