For a single test query:

(1) Was it based on identical queries that appeared high on the list of all three search engines?

(2) Was it based on similar queries? If so, was it paraphrased or based on the exact wording of one of the search engine’s query?

(3) Was it selected from one of the search engine’s listings independent of where or whether it appeared on the others? Ex., “Forever beta ranked high on Google; we’ll use that. Foul mouthed CEOs ranked high on Yahoo; we’ll use that. And two girls one cup was tops on Bing; we’ll use that.”

What was the breakdown of sources for queries, by search engine?

How did that correlate with the relevance of the results?

Given that the algorithms each engine employs refines the results based on results users click on, one would expect the results for popular searches to be highly relevant and to get better with time. If there is a bias in selection of the queries, it may show up in the reported results.

Because of these limitations, the “personal” queries may seem a better choice for the test. But interpreting the results would be much more subjective, because my intent when Googling “leather harnesses” might be different than someone else’s.

A study like this is not an easy task, but a fuller understanding of the methodology would make it more valuable.

Thanks,
Rob