Differences in rater behaviors are among the factors responsible for variability in the decision making process(DMP) during ratings. The interference of either the rater rating style or rater experience determines the validity and reliability of the rating score and the rater themselves. Factors related to rater inconsistencies identification and measurement in DMP is necessary to avoid factors underlying variability in decision making process . Several studies have identified rater proficiency level, rater experiences and tasks as its factors. The purpose of this paper is to critically review two articles that contribute to describe insights of rater behavior related to the factors studied. Barkaoui (2010) ‘Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience’ identifies effects of rating scales and experiences on raters behavior through think aloud protocol. While Baker (2012)’Individual Differences in Rater Decision-Making Style: An Exploratory Mixed-Methods Study” defines and addresses decision making style (DMS) as a factor related to decision making process in writing assessments rating. The following summary and critical evaluation of these articles explicitly provide what involves in raters decisions as well as the strength and weaknesses of these two articles.
Summary of text 1
Barkaoui (2010) quantitative study of variability in raters assessments discusses the impact of rating scales and rater experiences in writing rating (p.54).12 ESL essays were holistically and analytically assessed by 25 raters, novice and experienced( Barkaoui,2010,p.56). Results shows that rating scales has greater impact on rating processes, especially in analytical ratings( Barkaoui,2010,p.56). Although novice raters rely more on rating scales than experience raters, the author contends that the effects from raters experiences are not evident ( Barkaoui,2010,p.61). The implications for this study were mainly drawn from the score results of the holistic and analytical scale collected. The author concluded that essay raters and researchers should consider rating scale a factor with greater effect than rater experience in rater decision making process ( Barkaoui,2010,p.66).
Summary of text 2
Baker (2012) focuses on rater decision making style (DMS) and rater profiles frequencies that effect decision making process( p.228). Rating scores and comments of 6 different rater are collected during the English Exam for Teacher Certification(EETC) by ‘Write aloud’ protocol and DMSI questionnaires ( Baker,2012,p.230). The author assumes that the study will provide an insight to individual rater cognition ,enabling the establishment of individualized rater DMS profiles( Baker,2012,p.228). Baker findings shows that variability in scoring is a result of sociocognitive differences in DMS and DMS profiles suggested is valid( Baker,2012,p.237-239). Baker further suggests that the DMS profiles implemented is only an approach to...