Skip to main content
Calcimator

Inter-Rater Reliability Calculator

Calculate Cohen's kappa coefficient to measure agreement between two raters beyond what would be expected by chance. Get an interpretation of your reliability score.

Inputs

%
%

Results

Cohen's Kappa

0.6

InterpretationModerate
Observed Agreement80%
Expected Agreement50%
Agreements80
Disagreements20
How to Use This Calculator
  1. Enter the number of raters and the number of items rated.
  2. Input the rating scale type (nominal, ordinal, interval).
  3. Enter the percentage of agreements observed across all rater pairs.
  4. Review the Cohen's Kappa or ICC coefficient along with its interpretation.
  5. If Kappa is below 0.60, schedule a rater calibration session and re-rate a subset of items.

How the result changes with Total Ratings

Total RatingsCohen's Kappa
10,001-0.98
35,001-1
65,000-1
90,000-1

What each input means

Number of Agreements
The number of items where both raters assigned the same category or score.
Total Ratings
The total number of items rated by both raters.
Rater 1 Positive Rate (%)
Percentage of items that rater 1 classified as positive/yes. Used to calculate expected chance agreement.
Rater 2 Positive Rate (%)
Percentage of items that rater 2 classified as positive/yes. Used to calculate expected chance agreement.

How this is calculated

Worked example, using the default values

  1. Identify Input Parameters
    4 parameters
    Number of Agreements = 80, Total Ratings = 100, Rater 1 Positive Rate (%) = 55, Rater 2 Positive Rate (%) = 50 = 4 input(s) provided
  2. Calculate Cohen's Kappa
    Cohen's Kappa
    0.6 = 0.6
  3. Calculate Interpretation
    Interpretation
    Moderate = Moderate
  4. Calculate Observed Agreement
    Observed Agreement
    80 = 80

Engine last updated . Checked against 2 independently-derived tests — how we verify calculators.

Related Calculators

The questions that sit next to this one — chosen by subject, including calculators filed under a different category.

Survey Research

Net Promoter Score Calculator

Calculate your Net Promoter Score (NPS) from survey responses. Enter the count of promoters, passives, and detractors to see your score and classification.

Survey Research

Likert Scale Analyzer

Analyze 5-point Likert scale survey responses to get mean, median, standard deviation, and top/bottom-box percentages with a distribution chart.

Power Grid

Reliability Indices Calculator

Calculate SAIDI, SAIFI, and CAIDI from outage data.

Survey Research

Conjoint Analysis Calculator

Calculate the relative importance of product or service attributes from conjoint analysis utility ranges. Determine which features matter most to your respondents.

More in Math & Statistics.