Rating an essay is considered a complicated task for raters to remain consistent in their decision making process (DMP) when rating. It is because different raters have different qualities which affect raters’ behavior when assessing writing. Several researchers describe these qualities as raters’ proficiency level, raters’ experiences and tasks as the cause of raters’ inconsistencies in raters’ performance in their studies.Other researchers in recent studies suggest that rating process and rater cognitive process are the main cause of these variabilities.Since the level of variabilities determines both the score validity and the rater’s reliability, it is necessary to identify other causes of rater inconsistencies in the DMP.
The purpose of this paper is to critically review two articles that describe the inconsistencies in rater behaviors related to the factors studied. Khaled Barkaoui (2010), in his article “Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience”, identifies effects of rating scales and experiences on raters behavior through think aloud protocol. Beverly Anne Baker (2012), in her article ”Individual Differences in Rater Decision-Making Style: An Exploratory Mixed-Methods Study”,states that decision making style (DMS) is related to the DMP in writing assessment rating.The following critical evaluation will be presented first with a summary of the two articles and its evaluation will be presented later with each articles strength and weaknesses ,prior to the conclusion of this article.
Summary of text one
Barkaoui’s (2010) quantitative study of variability in raters assessments discusses the impact of rating scales and rater experiences in essay rating context (p. 54).In this study, 25 raters assessed 12 essays silently and 12 while thinking aloud. Each raters were identified as either novice or experience raters with at least five years experience (Barkaoui, 2010, p. 56). After rating holistically and analytically, the rating scores were collected. Results show that rating scales have greater impact on rating processes, especially in analytical ratings( Barkaoui, 2010, p. 56). Although novice raters rely more on rating scales than experience raters, the author contends that the effects of raters experiences are not evident ( Barkaoui, 2010, p. 61). The author concluded that essay raters and researchers should consider rating scale a greater effect than rater experience in a rater DMP ( Barkaoui,2010, p. 66).
Summary of text two
Baker (2012) focuses on rater decision making style (DMS) and rater profiles frequencies that affect the DMP ( p. 228). Rating scores and comments of six different raters were collected during the English Exam for Teacher Certification (EETC) by ‘Write aloud’ protocol and general decision making style inventory (GDMSI) questionnaires ( Baker, 2012, p. 227). Baker assumes that the study will provide an insight to individual rater cognition ,enabling...