A Comparative Study of Ontology Matching Systems via Inferential Statistics

Journal article (2018)

Authors

DOI: https://doi.org/10.1109/TKDE.2018.2842019

Geoscience Robustness Statistical analysis Holm Friedman Bergmann Task analysis Benchmark testing McNemar Nemenyi Ontologies Ontology alignment evaluation Paired t-test Post-hoc Quade Shaffer Wilcoxon signed-rank

To reference this document use:

http://resolver.tudelft.nl/uuid:18b4db2a-4c9c-45f7-97cd-1fadc5409bdc

More Info

expand_more

Published Date

2018

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Abstract

Comparing ontology matching systems are typically performed by comparing their average performances over multiple datasets. However, this paper examines the alignment systems using statistical inference since averaging is statistically unsafe and inappropriate. The statistical tests for comparison of two or multiple alignment systems are theoretically and empirically reviewed. For comparison of two systems, the Wilcoxon signed-rank and McNemar's mid-p and asymptotic tests are recommended due to their robustness and statistical safety in different circumstances. The Friedman and Quade tests with their corresponding post-hoc procedures are studied for comparison of multiple systems, and their [dis]advantages are discussed. The statistical methods are then applied to benchmark and multifarm tracks from the ontology matching evaluation initiative (OAEI) 2015 and their results are reported and visualized by critical difference diagrams.

Files

08369114.pdf

(pdf | 2.06 Mb)

- Embargo expired in 30-11-2018

Unknown license