Social Tagging Does Badly In Search Effectiveness Study

On her websearchguide.ca blog, Gwen Harris summarizes the results of a recent study that compared the effectiveness of search services that employ tagging (so-called folksonomies used by social bookmarking sites like del.icio.us) with that of the more traditional directories and search engines.

The study, entitled Tagging and Searching: Search Retrieval Effectiveness of Folksonomies on the Web, is part of a thesis at the College of Communication and Information of Kent State University.

The study looked at measures of precision, retrieval and recall in information retrieval for 5 types of information queries: factual, known site or document, selection of documents, news, and entertainment.

Overall, search engines have the highest precision (number of relevant search results divided by total number of results retrieved). The engines also do much better than folksonomies at recall (defined for the study as the number of items retrieved by a search system divided by the total number of relevant items in the collection):

"The paper is interesting for showing the strengths of search engines and directories. Folksonomies do not come out well, being weak even in news and entertainment where we would expect better performance. They are also not good for factual answers or known site".

"But this doesn't mean writing off folksonomies. A url that is found through a folksonomy and a search engine is likely to be of higher relevance (someone 'voted' for it) suggesting that search engines would be stronger if they included the content of folksonomies" (...)

"Searches submitted to the search engines for factual queries had the highest precision (44.1%), and search engines also performed well for specific item queries (35.4% precision, 26.8% recall) and queries for a range of documents (43.5% precision, 9.6% recall). The directories and folksonomies performed worst in specific item searches in precision, and worst in factual answer searches in recall. The folksonomies performed best when executing queries that required a selection of relevant documents – in that case, the precision was 16%".