How the U.S. Uses Technology to Mine More Data More Quickly

(…snip…) While once the flow of data across the Internet appeared too overwhelming for N.S.A. to keep up with, the recent revelations suggest that the agency’s capabilities are now far greater than most outsiders believed. “Five years ago, I would have said they don’t have the capability to monitor a significant amount of Internet traffic,” said Herbert S. Lin, an expert in computer science and telecommunications at the National Research Council. Now, he said, it appears “that they are getting close to that goal.”

Thanks to Tyler Cowen for the link to this NYT piece by James Risen and Eric Lichtblau. I've been casually following Palantir Technologies for some time. They are arguably one of the “new disruptors” and by chance happen to be headquartered next to the Phillz Coffee that we favor in Palo Alto (Phillz is of course full of Palantir and other denizens of the brain-powered companies based around Palo Alto). From listening to a couple of podcast interviews with CEO Dr. Alex Karp, I understood Palantir's specialty was to help clients such as drug developers or intelligence agencies discern patterns in mountains of data. Palantir is not an automated data-mining algorithms company. Their secret sauce is enabling human brainpower to analyze enormous and often separate data sets. So visualization of relationships is an important part of their solutions.

Palantir has a large number of YouTube videos - conference lectures, demos, and “Palantir 101″ type overviews.

I'll close with an apt Alex Karp quote from the Palantir website:

“There is no point in having a war on terrorism if civil liberties are being undermined to the extent that we aren’t willing to fight that war.”

Further to the cognitive computing topic, I'll also note that IBM Research Director John Kelly is speaking tonight at the Computer History Museum in nearby Mountain View. We immediately tried to reserve seats, but Kelly's talk is already sold out. His book Smart Machines: IBM’s Watson and the Era of Cognitive Computing , will be published in the fall by Columbia University Press. You can read a free chapter here.