Analyst: Bing’s Nice, but Google Still Works Better–Unless You’re Booking a Trip or Have a Rash
by Peter Kafka
Posted on July 7, 2009 at 9:30 AM PT
An endless ad barrage may be enough to get you to sample Bing. But it can’t ensure you’ll like the results once you try it.
That’s the conclusion Citigroup’s (C) Mark Mahaney reached after taking Microsoft’s (MSFT) new search engine for a spin and comparing it to Google’s (GOOG) and Yahoo’s (YHOO). The result: Google still delivers better results most of the time. In 71 percent of searches, Google either supplied the most relevant answer or tied with other engines. Bing did that 46 percent of the time.
But it’s worth checking out the results of Mahaney’s test to see exactly how the search engines stack up. For one thing, as he notes, Bing’s results are getting better as it matures–it’s only been live for about a month–and as it adds bells and whistles (for instance, it is catering to vanities of certain bloggers by including their Twitter stream in search results).
And Bing also performs better for at least two kinds of searches: Health and travel. So if you need to book a trip, or if you pick up a rash on that trip, you might want to think about Binging before you Google.
First, Mahaney’s methodology:
Over the past two weeks, we conducted 200 queries across the three major Search engines–Google, Yahoo! and Bing. Most of the queries we picked came from Google Zeitgeist, Microsoft’s xRank and Yahoo! Buzz, along with some personal queries. These queries spanned multiple categories, including Entertainment, Health, Local, News, Retail, Sports, Travel and Other. After conducting the same query across all three Search sites, we picked a winner based on: 1) relevancy of the organic search results; and 2) robustness of the search experience, which included factors such as image and video inclusion, Search Assist, and Site Breakout.
And his results. Overall relevancy (click to enlarge):
Relevancy by topic (click to enlarge):
Comments
Do you have access to the full report? Does it go deeper into the methodology, in particular the source of the 200 queries? As Mahaney notes, most of them were selected from the three search engines’ own listings of the most popular searches during the test period.
For a single test query:
(1) Was it based on identical queries that appeared high on the list of all three search engines?
(2) Was it based on similar queries? If so, was it paraphrased or based on the exact wording of one of the search engine’s query?
(3) Was it selected from one of the search engine’s listings independent of where or whether it appeared on the others? Ex., “Forever beta ranked high on Google; we’ll use that. Foul mouthed CEOs ranked high on Yahoo; we’ll use that. And two girls one cup was tops on Bing; we’ll use that.”
What was the breakdown of sources for queries, by search engine?
How did that correlate with the relevance of the results?
Given that the algorithms each engine employs refines the results based on results users click on, one would expect the results for popular searches to be highly relevant and to get better with time. If there is a bias in selection of the queries, it may show up in the reported results.
Because of these limitations, the “personal” queries may seem a better choice for the test. But interpreting the results would be much more subjective, because my intent when Googling “leather harnesses” might be different than someone else’s.
A study like this is not an easy task, but a fuller understanding of the methodology would make it more valuable.
Thanks,
Posted by Rob Campbell at July 7th, 2009 at 11:41 amRob
Hi Rob. I’ve got the full report, but there’s no more detail about the methodology.
Posted by Peter Kafka at July 8th, 2009 at 5:39 amI wonder if this sort of specialized search won’t become a pretty big part of the race. I can see where the big companies would start to aim to please by specifically focusing on top-notch results for frequently searched topics. There could be a competition to see who can provide the best results in travel, health, even sports or music.
Posted by Will Cox at July 8th, 2009 at 6:24 amI know there are niche searches out there that are growing in popularity and hopefully they can stay afloat before the major entities try to disrupt that.
Overall though, you are right. Google still probably churns out the better results as a whole but Bing is showing it’s willing to work towards improvement and has really went after it with these new partnerships with twitter and facebook.
I’d also like to invite anyone to check out meta search engine eZanga.com, just for those interested in trying out different ways to search.