User-centered evaluation for IR: Ranking annotated document algorithms

Syarifah Bahiyah Rahayu, Shahrul Azman Mohd Noah, Andrianto Arfan Wardhana

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document.

Original languageEnglish
Title of host publicationCommunications in Computer and Information Science
Pages306-312
Number of pages7
Volume181 CCIS
EditionPART 3
DOIs
Publication statusPublished - 2011
Event2nd International Conference on Software Engineering and Computer Systems, ICSECS 2011 - Kuantan
Duration: 27 Jun 201129 Jun 2011

Publication series

NameCommunications in Computer and Information Science
NumberPART 3
Volume181 CCIS
ISSN (Print)18650929

Other

Other2nd International Conference on Software Engineering and Computer Systems, ICSECS 2011
CityKuantan
Period27/6/1129/6/11

Fingerprint

Semantic Web

Keywords

  • annotation document
  • human ranking
  • semantic web

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Rahayu, S. B., Mohd Noah, S. A., & Wardhana, A. A. (2011). User-centered evaluation for IR: Ranking annotated document algorithms. In Communications in Computer and Information Science (PART 3 ed., Vol. 181 CCIS, pp. 306-312). (Communications in Computer and Information Science; Vol. 181 CCIS, No. PART 3). https://doi.org/10.1007/978-3-642-22203-0_27

User-centered evaluation for IR : Ranking annotated document algorithms. / Rahayu, Syarifah Bahiyah; Mohd Noah, Shahrul Azman; Wardhana, Andrianto Arfan.

Communications in Computer and Information Science. Vol. 181 CCIS PART 3. ed. 2011. p. 306-312 (Communications in Computer and Information Science; Vol. 181 CCIS, No. PART 3).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Rahayu, SB, Mohd Noah, SA & Wardhana, AA 2011, User-centered evaluation for IR: Ranking annotated document algorithms. in Communications in Computer and Information Science. PART 3 edn, vol. 181 CCIS, Communications in Computer and Information Science, no. PART 3, vol. 181 CCIS, pp. 306-312, 2nd International Conference on Software Engineering and Computer Systems, ICSECS 2011, Kuantan, 27/6/11. https://doi.org/10.1007/978-3-642-22203-0_27
Rahayu SB, Mohd Noah SA, Wardhana AA. User-centered evaluation for IR: Ranking annotated document algorithms. In Communications in Computer and Information Science. PART 3 ed. Vol. 181 CCIS. 2011. p. 306-312. (Communications in Computer and Information Science; PART 3). https://doi.org/10.1007/978-3-642-22203-0_27
Rahayu, Syarifah Bahiyah ; Mohd Noah, Shahrul Azman ; Wardhana, Andrianto Arfan. / User-centered evaluation for IR : Ranking annotated document algorithms. Communications in Computer and Information Science. Vol. 181 CCIS PART 3. ed. 2011. pp. 306-312 (Communications in Computer and Information Science; PART 3).
@inproceedings{d5cb7a08eb5843f1a51ad4a18749a6b5,
title = "User-centered evaluation for IR: Ranking annotated document algorithms",
abstract = "Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document.",
keywords = "annotation document, human ranking, semantic web",
author = "Rahayu, {Syarifah Bahiyah} and {Mohd Noah}, {Shahrul Azman} and Wardhana, {Andrianto Arfan}",
year = "2011",
doi = "10.1007/978-3-642-22203-0_27",
language = "English",
isbn = "9783642222023",
volume = "181 CCIS",
series = "Communications in Computer and Information Science",
number = "PART 3",
pages = "306--312",
booktitle = "Communications in Computer and Information Science",
edition = "PART 3",

}

TY - GEN

T1 - User-centered evaluation for IR

T2 - Ranking annotated document algorithms

AU - Rahayu, Syarifah Bahiyah

AU - Mohd Noah, Shahrul Azman

AU - Wardhana, Andrianto Arfan

PY - 2011

Y1 - 2011

N2 - Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document.

AB - Since the introduction of Semantic Web, the practice of seeking and retrieving documents had been evolved. In this paper, the retrieved documents are ranked based on their annotated documents. We adopt two IR algorithms; Lucene Luke and ComFFICF. In order to verify the generated rankings, we run a user-centered evaluation, where it involved 10 human judges. Then, we assess the performance of ranking using NDCG metric. The assessment shows a ranking by ComFFICF algorithm outperforms a ranking by Lucene Luke. This method is proven to be one of preferable IR algorithms for searching and ranking annotated document.

KW - annotation document

KW - human ranking

KW - semantic web

UR - http://www.scopus.com/inward/record.url?scp=79960361955&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960361955&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-22203-0_27

DO - 10.1007/978-3-642-22203-0_27

M3 - Conference contribution

AN - SCOPUS:79960361955

SN - 9783642222023

VL - 181 CCIS

T3 - Communications in Computer and Information Science

SP - 306

EP - 312

BT - Communications in Computer and Information Science

ER -