The Internet knows what you'll do next

[Al-khiyal]

A FEW years back, a technology writer named John Battelle began talking about how the Internet had made it possible to predict the future. When people went to the home page of Google or Yahoo and entered a few words into a search engine, what they were really doing, he realized, was announcing their intentions.

They typed in "Alaskan cruise" because they were thinking about taking one or "baby names" because they were planning on needing one. If somebody were to add up all this information, it would produce a pretty good notion of where the world was headed, of what was about to get hot and what was going out of style.

Mr. Battelle, a founder of Wired magazine and the Industry Standard, wasn't the first person to figure this out. But he did find a way to describe the digital crystal ball better than anyone else had. He called it "the database of intentions."

The collective history of Web searches, he wrote on his blog in late 2003, was "a place holder for the intentions of humankind — a massive database of desires, needs, wants, and likes that can be discovered, subpoenaed, archived, tracked, and exploited to all sorts of ends."

"Such a beast has never before existed in the history of culture, but is almost guaranteed to grow exponentially from this day forward," he wrote. It was a nice idea, but for most of us it was just an abstraction. The search companies did offer glimpses into the data with bare-bones (and sanitized) rankings of the most popular search terms, and Yahoo sold more detailed information to advertisers who wanted to do a better job of selling their products online. But there was no way for most people to dig into the data themselves.

A few weeks ago, Google took a big step toward changing this — toward making the database of intentions visible to the world — by creating a product called Google Trends. It allows you to check the relative popularity of any search term, to look at how it has changed over the last couple years and to see the cities where the term is most popular. And it's totally addictive.

You can see, for example, that the volume of Google searches would have done an excellent job predicting this year's "American Idol," with Taylor Hicks (the champion) being searched more often than Katharine McPhee (second place), who in turn was searched more often than Elliot Yamin (third place). Then you can compare Hillary Clinton and Al Gore and discover that she was more popular than he for almost all of the last two years, until he surged past her in April and stayed there.

Thanks to Google Trends, the mayor of Elmhurst, Illinois, a Chicago suburb, has had to explain why his city devotes more of its Web searches to "sex" than any other in the United States (because it doesn't have strip clubs or pornography shops, he gamely told The Chicago Sun-Times). On Mr. Battelle's blog, somebody claiming to own an apparel store posted a message saying that it was stocking less Von Dutch clothing and more Ed Hardy because of recent search trends.(A disclosure: The New York Times Company owns a stake in Mr. Battelle's latest Internet company, Federated Media Publishing.)

It's the connection to marketing that turns the database of intentions from a curiosity into a real economic phenomenon. For now, Google Trends is still a blunt tool. It shows only graphs, not actual numbers, and its data is always about a month out of date. The company will never fully pull back the curtain, I'm sure, because the data is a valuable competitive tool that helps Google decide which online ads should appear at the top of your computer screen, among other things. .

But Google does plan to keep adding to Trends, and other companies will probably come up with their own versions as well. Already, more than a million analyses are being done some days on Google Trends, said Marissa Mayer, the vice president for search at Google.

When these tools get good enough, you can see how the business of marketing may start to change. As soon as a company begins an advertising campaign, it will be able to get feedback from an enormous online focus group and then tweak its message accordingly.

I've found Pepsi's recent Super Bowl commercials — the ones centered around P. Diddy — to be nearly devoid of wit, but that just shows you how good my marketing instincts are. As it turns out, the only recent times that Pepsi has been a more popular search term in this country than Coke have been right after a Super Bowl. This year's well-reviewed Burger King paean to Busby Berkeley, on the other hand, barely moved the needle inside the database of intentions.

Hal R. Varian, an economist at the University of California, Berkeley, who advises Google, predicts that online metrics like this one have put Madison Avenue on the verge of a quantitative revolution, similar to the one Wall Street went through in the 1970's when it began parsing market data much more finely. "People have hunches, people have prejudices, people have ideas," said Mr. Varian, who also writes for this newspaper about once a month. "Once you have data, you can test them out and make informed decisions going forward."

There are certainly limitations to this kind of analysis. It's most telling for products that are bought, or at least researched, online, a category that does not include Coke, Pepsi or Whoppers. And even with clothing or cars, interest doesn't always translate into sales. But there is no such thing as a perfect yardstick in marketing, and the database of intentions clearly offers something new.

In the 19th century, a government engineer whose work became the seed of I.B.M. designed a punched-card machine that allowed for a mechanically run Census, which eventually told companies who their customers were. The 20th century brought public opinion polls that showed what those customers were thinking. This century's great technology can give companies, and anyone else, a window into what people are actually doing, in real time or even ahead of time.

You might find that a little creepy, but I bet that you'll also check it out sometime.

The Internet knows what you'll do next

"On its home page, Google Trends offers a number of sample searches. Some of them can be interesting — like the recent one showing that "blogs" is now searched more often than "magazines" — but I found most of them to be too general. It's much more fun to do your own....."

Continue reading.... How to use Google Trends


(Tony Starks has previously posted on Google Trends here)

[Al-khiyal]

From Amis to Zeppelin, what your web searches reveal

Analysing what we look for on the web can offer a remarkable insight into our anxieties and enthusiasms

December 29, 2007 -- Four years ago, the writer and internet entrepreneur John Battelle had a sudden epiphany - the kind of moment that leaves you giddy, teetering on a conceptual cliff, as you contemplate its full ramifications. Battelle had already been preaching the transformative power of the internet for some time. But now his thinking turned to the millions of web searches that people were conducting around the world each day, using Google and a handful of other sites.

As people searched, he realised, they were inadvertently leaving a trail - a gargantuan historical archive of whatever was on the world's mind at a particular time, which remained stored on the central computers of firms such as Google, Yahoo and Microsoft.

Battelle called it "the database of intentions". "This information represents, in aggregate form, a placeholder for the intentions of humankind," he wrote breathlessly on his blog. What had been created was "a massive database of desires, needs, wants, and likes that can [be] archived, tracked, and exploited to all sorts of ends. Such a beast has never before existed in the history of culture ... this artefact can tell us extraordinary things about who we are and what we want as a culture. And it has the potential to be abused in equally extraordinary fashion."

Since then, the database of intentions has grown dizzyingly: in one month alone during 2007, the number of searches conducted using the five leading sites reached 9.4bn. But until recently we could only glimpse at the secrets it contained - for example, when the internet company AOL mistakenly released information on what 658,000 of its members had been searching for. (No names were released, but individual users' search histories included eyebrow-raising anomalies: "replica louis vuitton bag ... how to secretly poison your ex ... how to colour hair with clairol professional ...") And in 2005 an obscure internet forum about video technology became deluged with messages after it became the top result for Google searches on the phrase "I am lonely", which thousands of people, it turned out, were typing every day.

In contrast to those brief glimpses, the graphs on these pages provide a radically broader and deeper view of what people are searching for online, and what that might mean. They were generated using Google Trends, an experimental service that uses aggregated data from Google search results to compare the numbers of people searching for different words and phrases over time, from 2004 to the present. This enables you to track, for example, how interest in Tony Blair was gradually superseded by interest in Gordon Brown over the course of 2007, or how Amy Winehouse overtook Lily Allen in the notoriety stakes, or how the awareness of the term global warming has grown down the years.

Searches are also broken down geographically. So you can discover, for example, that of all British towns and cities Luton has the highest proportion of searches for "sex", followed by Milton Keynes - although whether this means their inhabitants are unusually liberated, or desperate, or just bored, is a matter for speculation. (The phrase "I am bored", by the way, forms a larger proportion of Google searches in Sheffield than anywhere else.) Google Trends is free to use, at google.com/trends, and it is easy to waste far too much time playing with it.

Indications of scale

This form of measurement is far from perfect. The results are only approximate, and Google will not reveal the actual numbers of searches - presumably because that information is gold-dust for internet advertisers, and it intends to make them pay for it.

As a result, the graphs come with no indication of scale. They merely show the volume of searches for a particular term as a proportion of all searches on Google, which makes it impossible to tell whether a sudden surge in searches for, say, Paris Hilton represents a leap of several thousand or several million. So the graphs here are only impressionistic.

Even with those limitations, though, they point to the extraordinary amount of information that is waiting to be mined from internet search data: as Battelle rightly suspected, these charts help show the shifting concerns of an entire culture.

Sometimes, people's interests are driven fairly obviously by the news agenda: when the Spice Girls announce a reunion, there's an immediate rush to find out more about them. Other results are strikingly seasonal: not too surprisingly, people seem to go shopping online for coats in winter, and for sandals in summer.

But the most fascinating possibility is that search data might help to predict behaviour. After all, when we search online for a certain brand of stereo system rather than another, we are surely indicating that it's more likely we will buy that brand.

Perhaps we search for a political candidate's name when we are thinking about voting for him or her, and maybe we even search for "stock market crash" or "recession" just before we start pulling out of our investments. This information could clearly be useful to a savvy marketer - it's already how Google decides which ads to show on its search results pages - or to a political campaign manager.

Marissa Mayer, a Google vice-president, argues that Google Trends correctly "predicted" George Bush's victory over John Kerry in the 2004 election: our graph shows that Bush maintained his lead over his rival, in terms of search volumes, even when polls suggested the race was on a razor's edge.

The same approach leads to the prediction that Hillary Clinton will beat Barack Obama in the race for the Democratic presidential nomination in 2008; before too long, we'll be able to verify that. But then again, as the graph here shows, the conservative Republican Ron Paul ranks above them both. This is the result of an internet cult around Paul that has not been reflected in regular opinion polls, which may demonstrate the limits of treating web searches as if they were representative of an entire population's opinions - unless, of course, he ends up winning.

Unsettling

There is something very unsettling about all this. We do not like to think that other people can see inside our brains, even on a collective level, and many of us will have conducted hundreds of thousands of web searches in recent years without ever giving a thought to where all that data was going. But though so much seems ephemeral in the age of the web, nothing really is. It is all stored somewhere. The internet never forgets.