The study, which consisted of 5,000 individual queries, compared video search engines by measuring the breadth of coverage of websites with European video content.
The basic procedure is simple:
- Select a video to search for.
- Search for it in a video search engine.
- Record if the video was found.
- Repeat steps 1 to 3 for more videos and more video search engines.
Five industry-leading video search engines were evaluated using 1,000 distinct videos that are viewable online. These videos are selected from 100 websites that are in turn chosen from lists of high traffic websites published by Alexa, http://www.alexa.com. The list of top 100 websites is made up of the top 20 Alexa-ranked video content websites with the highest traffic rank (as recorded on January 22, 2009) in each of the 5 largest European markets (Great Britain, Germany, France, Italy and Spain). The top 20 websites in each country that had a reasonable number of videos and were appropriate for the study were selected for testing.
Then, the study selected 10 representative videos from each of the websites and queried each search engine for each video. If a search engine returned at least five of the 10 representative videos, it was counted as covering that website. Because the test was designed to measure coverage, the search engine needed to return the exact video from the original site in order to be counted. In addition, each query was run and evaluated on all search engines at the same time before moving on to the next query, to make sure no search engine had more time than the others to ingest each video. The queries were run between January 26, 2009 and February 16, 2009.
.
