Low-cost evaluation techniques for information retrieval systems

A review

Shiva Imani Moghadasi, Sri Devi Ravana, Sudharshan Naidu Raman

Research output: Contribution to journalArticle

10 Citations (Scopus)

Abstract

For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of 'must read' papers in the area of low-cost evaluation of IR systems.

Original languageEnglish
Pages (from-to)301-312
Number of pages12
JournalJournal of Informetrics
Volume7
Issue number2
DOIs
Publication statusPublished - Apr 2013

Fingerprint

Information retrieval systems
information retrieval
Information Retrieval
Evaluation
costs
evaluation
Costs
Pooling
Reusability
Information retrieval
Retrieval
test evaluation
Review
Substitute
Statistical Inference
Judgment
Query
Partial
Metric
Alternatives

Keywords

  • Effectiveness metrics
  • Pooling
  • Relevance judgment
  • Retrieval evaluation
  • Test collection

ASJC Scopus subject areas

  • Applied Mathematics
  • Modelling and Simulation
  • Statistics and Probability
  • Management Science and Operations Research
  • Computer Science Applications

Cite this

Low-cost evaluation techniques for information retrieval systems : A review. / Moghadasi, Shiva Imani; Ravana, Sri Devi; Raman, Sudharshan Naidu.

In: Journal of Informetrics, Vol. 7, No. 2, 04.2013, p. 301-312.

Research output: Contribution to journalArticle

Moghadasi, Shiva Imani ; Ravana, Sri Devi ; Raman, Sudharshan Naidu. / Low-cost evaluation techniques for information retrieval systems : A review. In: Journal of Informetrics. 2013 ; Vol. 7, No. 2. pp. 301-312.
@article{b6e698cbe14e4489a51a2e49ab379985,
title = "Low-cost evaluation techniques for information retrieval systems: A review",
abstract = "For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of 'must read' papers in the area of low-cost evaluation of IR systems.",
keywords = "Effectiveness metrics, Pooling, Relevance judgment, Retrieval evaluation, Test collection",
author = "Moghadasi, {Shiva Imani} and Ravana, {Sri Devi} and Raman, {Sudharshan Naidu}",
year = "2013",
month = "4",
doi = "10.1016/j.joi.2012.12.001",
language = "English",
volume = "7",
pages = "301--312",
journal = "Journal of Informetrics",
issn = "1751-1577",
publisher = "Elsevier BV",
number = "2",

}

TY - JOUR

T1 - Low-cost evaluation techniques for information retrieval systems

T2 - A review

AU - Moghadasi, Shiva Imani

AU - Ravana, Sri Devi

AU - Raman, Sudharshan Naidu

PY - 2013/4

Y1 - 2013/4

N2 - For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of 'must read' papers in the area of low-cost evaluation of IR systems.

AB - For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of 'must read' papers in the area of low-cost evaluation of IR systems.

KW - Effectiveness metrics

KW - Pooling

KW - Relevance judgment

KW - Retrieval evaluation

KW - Test collection

UR - http://www.scopus.com/inward/record.url?scp=84872417208&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84872417208&partnerID=8YFLogxK

U2 - 10.1016/j.joi.2012.12.001

DO - 10.1016/j.joi.2012.12.001

M3 - Article

VL - 7

SP - 301

EP - 312

JO - Journal of Informetrics

JF - Journal of Informetrics

SN - 1751-1577

IS - 2

ER -