摘要：Currently existing web search engines return the same results for the same query issued to them. But, such systems do not satisfy the needs of different users having different information need underlying the same queries. In this study, we use explicit relevance judgment to show the variation in search results users find to be relevant. To get multiple judgments for the same query, we provide users with list of previously generated queries from our search engine and asked them to choose queries which are of interest to them and evaluate the search results quality for the query. Users are also asked to choose the queries they generated and evaluate the search results quality in the same fashion. The result we get shows that there is a great variation in users explicitly rating the same result for the same query and we use discounted cumulative gain to quantify this variation in relevance judgment. The result we get shows that with an increase in the number of people evaluating the same result for the same query, the gap between user satisfaction with an individual ranking and group ranking grows. Our experiments show that the best group ranking for a group of five people on average gives rise to a 26% improvement in discounted cumulative gain over the web ranking, while the best individual ranking leads to a 61% improvement over the web ranking.
