Google Wants To Listen In

TechCrunch, among others, is reacting to this post at Google’s research blog on a paper [PDF] about “Real-Time Ambient-Audio Identification.” The proposed technology would enable your computer to listen to the ambient sounds emitted from your TV, automatically determine what is being watched, and deliver “relevant content” (read: advertising) to your web browser.

To assuage privacy concerns, the paper notes that the system won’t listen in on user conversations, although the only explanation I can find in the paper on how it does this is to stop analyzing sound when a conversation temporarily drowns out the ambient sound from the TV. But would it record an in-room conversation if it occurs during a moment in which there is no ambient sound from the TV (ie, a silent moment in a dramatic scene? a love scene with no dialoge, etc)? If the volumes are similar, how will it differentiate between a conversation on TV and one between me and my wife?

A second privacy consideration is that all the actual sound processing occurs on the user’s computer, and only “summary statistics” are sent to the remote servers to deliver up the related content. The software records 5-second snippets, and processes them into 415 frames of 32-bit descriptors, which are then passed to the audio servers. This helps ensure that no one will be able to intercept or eavesdrop on the ambient sound being processed, although it might still be possible to identify what someone was watching based on the particular descriptor’s time and date stamp. It will depend on how many other programs have the same descriptor at that particular moment in time. A little data-mining might be able to determine whether I was watching The West Wing or Star Trek last night.

This is an interesting idea, and some of the services Google envisions will be attractive to many, such as identifying ad hoc social peer communities watching the same show at the same time. In many ways, it is similar to Arbitron’s Portable People Meter, where the goal is to monitor all the radio transmissions a person is exposed to (while riding in a taxi, eating in a restaurant, waking down the street). It will be interesting to watch this research progress.