<Sometimes, all it takes is a little spark to set off a major forest fire. That is what seems to have happened with my New Year’s Day post on Why We Desperately Need a New (and Better) Google. Over the last two months, there has been an avalanche of articles echoing my post (and a few before it from notable people like Jeff Atwood), including New York Magazine, Business Insider, GigaOm, TechCrunch, CNN, and The Wall Street Journal.

I had a feeling that this would get Google’s attention. And I had the same concern as when I challenged the Russian government, once, in a Bloomberg BusinessWeek article about Skolkovo (a new tech park). I feared that Google would either blacklist me or do its equivalent of putting me in a Gulag—deliver even more spam when I search websites.

But I was delighted to get an e-mail from Amit Singhal, the head of Google’s SEO team. His message was exemplary for those wanting to learn how to handle a PR crisis. Here is part of what he wrote:

I read your post on TechCrunch yesterday and was quite disappointed by the fact that Google search failed your students at their task. My team and I treat every such failure as an inspiration to improve Google. Would it at all be possible for me to get a few queries from your students for which our algorithm failed? We will debug every aspect of our system for those queries.

He went on to invite me to visit Google to show me how they run the search system and listen to any other criticism I had to offer.

I took Amit up on this and spent hours with him; with Matt Cutts, who heads Google’s webspam team; and with their lead developers. They were incredibly open and honest. They acknowledged the deficiencies of Google search, shared ideas on how they plan to fix them, and asked for feedback.

I raised concerns that “content farms” are turning the web into a massive garbage dump, that many sites are simply replicating the content of others like TechCrunch, and that Google has no incentive to stop this because it gains advertising revenue from the spammers.

The Google developers assured me that there is a very high Chinese Wall between them and the business side of the company; that they have been instructed by Google’s executives to do only what is in the interest of users—to keep improving quality of search results and the user experience. They said they understood the issues and had many solutions to the technical problems. I questioned whether the spam problems could even be solved algorithmically; whether the only solution was a curated web-search model like that of Blekko and DuckDuckGo. They convinced me that they could, and would, win the battle.

Matt said he would post a blog, which he did, on Jan 21. In it, he explained that Google had already made improvements to make it harder for “spammy on-page content to rank highly”; had radically improved its ability to detect hacked sites, which were a major source of spam in 2010; and was about to implement a change that would directly address the issue of sites copying others’ content. Most importantly, he acknowledged that something had to be done about the “content farms,” and said that Google would.

Not surprisingly, Matt’s blog led to another avalanche of media coverage. As it turns out, the biggest content farm of them all, Demand Media, was set for an IPO this week (on Jan 26).   Savvy bloggers and journalists began to question whether it could sustain its profits without Google’s support. The Wall Street Journal asked Did Google Just Make Demand Media Less in Demand?, and GigaOm wondered Did Google Just Declare War on Demand Media?

Nonetheless, Demand Media had a spectacular IPO. Its investors reaped huge bounties, with the company achieving a market cap of $1.7 billion—valuing it higher than the New York Times. So the public markets rewarded junk over quality. And they called Google’s bluff.

Where does that put us? Do we have to watch the web become one big toxic waste dump—as the spammers rake in billions of dollars? Or will Google indeed save the day?

There is an event on Tuesday, Feb 1, called Farsight 2011: Beyond the Search Box, to discuss these questions. It will be live-streamed on TechCrunch (watch for a post by Jon Orlin on that day) and is being organized by BigThink, a public online forum for intellectuals (people like Gary Kasparov, Jimmy Carter, Malcolm Gladwell, Salman Rushdie, Nouriel Roubini, and Paul Krugman). BigThink has thousands of videos on its site, which 1.5 million people watch every month.

I am emceeing the BigThink event and moderating a panel with three big players: Matt Cutts from Google; Harry Shum, Microsoft Corporate Vice President who heads Bing development, and Rich Skrenta, founder and CEO of Blekko.

Here are some questions that I plan to ask the panelists. Please share your comments below and suggest additional questions. I can’t promise I’ll cover all the topics you raise, but I will bring up as many as I can.

1. How will they save the web? Is it possible for search engines to separate the wheat from the chaff—tell the difference between content produced by regular people and large-scale junk produced by the spammers?

2. How are the engines really different? Most people can’t tell the difference between Google and Bing. Where is the magic?

3. What lies ahead? What is the future of search?

I have no doubt that this will be a very lively and informative event.  There are also other great presentations such as:

  • Jaron Lanier, named by Time Magazine in 2010 as one of the 100 most influential people in the world, speaking about the need for a new sustainable revenue model for search.
  • Esther Dyson, in an address entitled “The Future of Search is a Verb,” speaking about how we want search to help us do something—a set of many verbs.
  • Demos from Blaise Agüera y Arcas, Architect of BING maps at Microsoft; and Luc Barthelet, Executive Director of Wolfram|Alpha.

I hope you’ll tune in.

  • Guest

     I used to be more than happy to seek out this site.
    low sneakers
    I needed to thanks for this nice learn I
    undoubtedly having fun with every little little bit of it and I have you
    bookmarked to check out new stuff you post.
    low sneakers
    and many writers do not get the credit they
    deserve. I am sure Ia€?ll be back and will send some of my friends. Your
    site is very useful for me .I bookmarked your site!
    low sneakers

  • Pingback: Wyspy zaufania w „morzu śmieci”()

  • Pingback: TV is Back: Why internet companies are taking out TV ads (announcing a panel at the 2011 IBC) | German Media Pool()

  • Pingback: Search Wars « John Akerson's Thoughts()

  • Pingback: blog-thing : Search Wars()

  • So – the Farsight is here. Was it productive? nnDanny Sullivan broke some information.http://selnd.com/g4L6PmnGoogle trapped Microsoft/Bing copying Google Results. http://bit.ly/dImGje Microsoft admitted to using opt-in data. http://bit.ly/dZl3IH Google obviously has been in the browser tool bar business for quite some time, and they also use the data. Microsoft and Google both make billions from advertising, from operating systems, from browsers, from devices, and ultimately from advertisers, based on hexa-giga-peta-bytes of “opt-in” customer data.nhttp://bit.ly/gSxwsB nnThere is endless value in the data that comes from browsers. Microsoft knows it. Google knows it. Everybody knows it. Microsoft is using the data to improve the tool that its customers are using. Google uses that value to sell AdWords. Everybody makes money counted in the Billions. nnThis argument between Google and Microsoft is Risky for either or both.nnHere are two possible outcomes that would benefit Microsoft.n1) If people conclude from the resulting press that Google and Bing are not that different… nhttp://selnd.com/hrOe9z and if businesses conclude that neither has a competitive advantage, that is a huge win for Microsoft’s Bing.nn2)If people are interested in having tools that learn from their input and misspelllllings, they might use Bing more often.nnConsidering that Bing (as of 1/14/11) has 12% market share vs 66% for Google, any win for Microsoft’s Bing could be extremely significant. http://selnd.com/fFZOfDn nHOWEVER, If people believe that Microsoft is stealing, that might benefit Google. If people believe that Google is emphasizing this issue to reduce the problem of junk-filled content farms which you address above as being able to differentiate “content produced by regular people and large-scale junk produced by the spammers” that is a problem for Google. nnIt is most risky, however, for both. I think this spat has the potential to further help people understand the value of the data they provide, the button-bars they install, the search engines they use, the sites they visit, the information that THEY willingly provide. If that is the case, it is risky to both Microsoft AND Google. nnGiven the risk on both sides, why does either company have interest in this fight?

    • John, thanks very much for your insights on this. I am not sure which side to pick! What I do know is that Google very skillfully changed the topic: from spam–which they profit from to cheating–that others profit from.

      • Thank you Vivek – it might take a book to flesh this out. So many things are happening between these two companies. Microsoft’s timing for the release of its Chrome H264 plugin, for instance, (http://bit.ly/i9WvgC) probably wasn’t coincidence. Ultimately I think there is substantial risk for both Google and Microsoft that whether you look at this as spam wars or search wars. In Microsoft vs Google war, very very smart people are working on both sides. From my perspective, this war may produce no victory – only casualties. (Wrote about it a bit on my blog too http://bit.ly/homMz4 )nn

      • Steve

        I have a different problem with Google. They deindexed my entire site after a 5 year run-despite the fact that we can find nothing that violates ANY of their guidelines.As a former president of an intellectual property information company and an MBA from Columbia University, I know my site is not spam and has considerable information value. And while i dont expect a high ranking unless
        I earn it-I dont expect to be deindexed for no reason either. Can Google searches really be trustedm i fhey do this to a highly useful site like mine? Take a look for yourself at my trademark/brand site. http://www.findownersearch.com.


        Steve Wolfson

  • Pingback: Tweets that mention The Future of Search: Who Will Win The Spam Wars? -- Topsy.com()