Institutions | About Us | Help | Gaeilge
rian logo


Mark
Go Back
PRES: A score metric for evaluating recall-oriented information retrieval applications
Magdy, Walid ; Jones, Gareth J.F.
Information retrieval (IR) evaluation scores are generally designed to measure the effectiveness with which relevant documents are identified and retrieved. Many scores have been proposed for this purpose over the years. These have primarily focused on aspects of precision and recall, and while these are often discussed with equal importance, in practice most attention has been given to precision focused metrics. Even for recalloriented IR tasks of growing importance, such as patent retrieval, these precision based scores remain the primary evaluation measures. Our study examines different evaluation measures for a recall-oriented patent retrieval task and demonstrates the limitations of the current scores in comparing different IR systems for this task. We introduce PRES, a novel evaluation metric for this type of application taking account of recall and the user’s search effort. The behaviour of PRES is demonstrated on 48 runs from the CLEF-IP 2009 patent retrieval track. A full analysis of the performance of PRES shows its suitability for measuring the retrieval effectiveness of systems from a recall focused perspective taking into account the user’s expected search effort.
Keyword(s): Information retrieval; PRES; Recall-Oriented Information Retrieval; Patent Retrieval; Evaluation Metric
Publication Date:
2010
Type: Conference item
Peer-Reviewed: Yes
Language(s): English
Institution: Dublin City University
Citation(s): Magdy, Walid and Jones, Gareth J.F. (2010) PRES: A score metric for evaluating recall-oriented information retrieval applications. In: The 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2010), 19-23 Jul 2010, Geneva, Switzerland. ISBN 978-1-4503-0153-4
Publisher(s): Association for Computing Machinery
File Format(s): application/pdf
Related Link(s): http://doras.dcu.ie/16180/2/PRES_A_Score_Metric_for_Evaluating_Recall-Oriented_Information_Retrieval_Applications.pdf
First Indexed: 2011-07-23 05:15:00 Last Updated: 2012-08-25 05:19:03