Study performs Coke vs. Pepsi Test on Engines

According to a new study out of France, today’s search engines are more interested in banging out fancy new features than improving search relevance and customer satisfaction.

French professor Jean Veronis and his students used three American search engines: Google, Yahoo! and MSN and three French search engines: Exalead, Voila and Dir.com to compare the results from fourteen different queries. The search topics were selected to reflect a broad range of subject areas, including animals, cinema and current events. Participants were then able to construct their own search queries, using their own format (with or without quotation marks, querying one or multiple terms, etc.) to perform a ‘blind test’ on the search engines. To keep the identity of the engines a mystery, the students never performed an actual search. Instead, they wrote down their queries and Veronis performed all the searches himself. The search-URL result pairs where then given to the students to evaluate. After receiving the results, the students were asked to rate the engines in terms of relevance, scoring them on a scale of zero to five, with five being the highest. A passing grade was considered to be 2.5.

Result: All of the engines failed.

Google and Yahoo tied for ‘first place’ in terms of relevancy (MSN came in third), with a rating of 2.3, but when it came to user satisfaction, the engines failed universally with flying colors. Many engines, including Google with 28.6 percent and Yahoo with 27.7 percent, had links that scored a pathetic 0, classifying them as ‘totally useless’. Nice.

The results hint at some interesting conclusions. First, there is no difference between Google and Yahoo! in terms of search relevancy. If the study is accurate, users deem them 100 percent identical. So why is Google the hands-down favorite? Is it merely a result of their branding? A user is loyal to their engine because they are also using their email system, desktop, toolbar and exploring Google Earth? Second, are the search engines placing too much energy in their extras department instead of refining their search engines? Would a true, search-only engine truly satisfy user needs without the extra features? We’ll have to wait for someone to design one to find out.

The full study is available in PDF format for those interested.

(Hat-tip to Andy Beal)

Lisa Barone is a writer, content marketer & VP of strategy at Overit Media. She's also a very active Twitterer, much to the dismay of the rest of the world.

See Lisa's author page for links to connect on social media.