NOTE: Please visit the Errata web page of this book here for errors found and corrections made.

The third edition of this book was very well received by researchers working in many different fields of research. The use of that text also gave these researchers the opportunity to raise questions, and express additional needs for materials on techniques poorly covered in the literature. For example, when designing an inter-rater reliability study, many researchers wanted to know how to determine the optimal number of raters and the optimal number of subjects that should participate in the experiment. Also, very little space in the literature has been devoted to the notion of intra-rater reliability, particularly for quantitative measurements. The fourth edition of this text addresses those needs, in addition to further refining the presentation of the material already covered in the third edition.

The methods and techniques developed in this edition of the handbook of inter-rater reliability can handle missing ratings, which are common in most experiments. This is an improvement over the second edition, which describes the methods for complete data sets only. Part II and Part III contain new chapters aimed at providing researchers with a broader coverage of inter-rater reliability techniques. Even chance-corrected agreement coefficients, already covered in the second edition, are presented in the current edition with more depth and more clarity.

Features of the Fourth Edition include:

1) New material on sample size calculations for chance-corrected agreement coefficients, as well as for intraclass correlation coefficients. The researcher will be able to determine the optimal number raters, subjects, and trials per subject.

2)The chapter entitled “Benchmarking Inter-Rater Reliability Coefficients” has been entirely rewritten.

3)The introductory chapter has been substantially expanded to explore possible definitions of the notion of inter-rater reliability.

4)All chapters have been revised to a large extent to improve their readability.

I expect the Handbook of Inter-Rater Reliability to be an essential reference on inter-rater reliability assessment to all researchers, students, and practitioners in all fields of research.