The overall probability of random match is the probability that they agreed on yes or no, i.e. Kappa assumes its theoretical maximum value of 1 only if the two observers distribute the codes equally, i.e. if the corresponding row and column sums are identical. Everything else is not a perfect match. Nevertheless, the maximum value that kappa could achieve with uneven distributions helps to interpret the value actually obtained from kappa. The equation for the κ-maximum is:[16] Note that Cohen`s kappa only measures the agreement between two evaluators. For a similar level of agreement (Fleiss kappa) used when there are more than two evaluators, see Fleiss (1971). The Fleiss Kappa, however, is a multi-evaluator generalization of Scott`s Pi statistics, not Cohen`s Kappa. Kappa is also used to compare performance in machine learning, but the directed version known as Informedness or Youden`s J Statistics is considered more suitable for supervised learning.

[20] Suppose you are analyzing data from a group of 50 people applying for a grant. Each grant application was read by two readers and each reader said “yes” or “no” to the proposal. Suppose the data on the number of disagreements are as follows, where A and B are readers, the data on the main diagonal of the matrix (a and d) count the number of matches and the data outside the diagonal (b and c) count the number of disagreements: To calculate pe (the probability of a random match), we find that: Kappa is an index, which takes into account the match observed in relation to a reference match. However, researchers should carefully consider whether Kappa`s basic agreement is relevant to the particular research question. Kappa`s baseline is often described as the agreement based on chance, which is only partially correct. The Basic Kappa Agreement is the agreement that would be expected due to random allocation given the quantities indicated by the limit sums of the square contingency table. Thus, kappa = 0 if the observed allocation is apparently random, regardless of the opinion defined limited by the limit sums. In many applications, however, researchers should be more interested in the quantitative consistency of limit sums than in the allocation notice as described by the additional diagonal information in the square contingency table. Therefore, kappa`s baseline is more distracting than insightful for many applications. .