he past 20 years have seen a great improvement in the rigor of information retrieval experimentation, due primarily to two factors: high-quality, public, portable test collections such as those produced by TREC (the Text REtrieval Conference), and the increased practice of statistical hypothesis testing to determine whether measured improvements can be ascribed to something other than random chance. Together these create a very useful standard for reviewers, program committees, and journal editors; work in information retrieval (IR) increasingly cannot be published unless it has been evaluated using a well-constructed test collection and shown to produce a statistically signiﬁcant improvement over a good baseline.
But, as the saying goes, any tool sharp enough to be useful is also sharp enough to be dangerous. Statistical tests of signiﬁcance are widely misunderstood. Most researchers and developers treat them as a “black box”: evaluation results go in and a p-value comes out. But because signiﬁcance is such an important factor in determining what research directions to explore and what is published, using p-values obtained without thought can have consequences for everyone doing research in IR. Ioannidis has argued that the main consequence in the biomedical sciences is that most published research ﬁndings are false; could that be the case in IR as well?
Our goal with this tutorial is to help researchers and developers gain a better understanding of how tests work and how they should be interpreted so that they can both use them more effectively in their day-to-day work as well as better understand how to interpret them when reading the work of others. It will be appropriate for researchers and practitioners who are new to IR and wish to learn the standards of the community for signiﬁcance testing and reporting, but also for experienced IR researchers and practitioners who already perform tests but desire a deeper understanding of what they are and how to interpret the information they provide.