Worked example Fleiss' kappa



in following example, fourteen raters (



n


{\displaystyle n}

) assign ten subjects (



n


{\displaystyle n}

) total of 5 categories (



k


{\displaystyle k}

). categories presented in columns, while subjects presented in rows. each cell lists number of raters assigned indicated (row) subject indicated (column) category.


data

see table right.






n


{\displaystyle n}

= 10,



n


{\displaystyle n}

= 14,



k


{\displaystyle k}

= 5


sum of cells = 140

sum of




p

i





{\displaystyle p_{i}\,}

= 3.780


calculations

the value




p

j




{\displaystyle p_{j}}

proportion of assignments (



n
×
n


{\displaystyle n\times n}

, here



10
×
14
=
140


{\displaystyle 10\times 14=140}

) made



j


{\displaystyle j}

th category. example, taking first column,








p

1


=



0
+
0
+
0
+
0
+
2
+
7
+
3
+
2
+
6
+
0

140


=
0.143


{\displaystyle p_{1}={\frac {0+0+0+0+2+7+3+2+6+0}{140}}=0.143}



and taking second row,








p

2


=


1

14
(
14

1
)




(

0

2


+

2

2


+

6

2


+

4

2


+

2

2



14
)

=
0.253


{\displaystyle p_{2}={\frac {1}{14(14-1)}}\left(0^{2}+2^{2}+6^{2}+4^{2}+2^{2}-14\right)=0.253}



in order calculate






p
¯





{\displaystyle {\bar {p}}}

, need know sum of




p

i




{\displaystyle p_{i}}

,










i
=
1


n



p

i


=
1.000
+
0.253
+

+
0.286
+
0.286
=
3.780


{\displaystyle \sum _{i=1}^{n}p_{i}=1.000+0.253+\cdots +0.286+0.286=3.780}



over whole sheet,










p
¯



=


1

(
10
)



(
3.780
)
=
0.378


{\displaystyle {\bar {p}}={\frac {1}{(10)}}(3.780)=0.378}












p
¯




e


=

0.143

2


+

0.200

2


+

0.279

2


+

0.150

2


+

0.229

2


=
0.213


{\displaystyle {\bar {p}}_{e}=0.143^{2}+0.200^{2}+0.279^{2}+0.150^{2}+0.229^{2}=0.213}








κ
=



0.378

0.213


1

0.213



=
0.210


{\displaystyle \kappa ={\frac {0.378-0.213}{1-0.213}}=0.210}








Comments