Research domains
This is applicable to any domain where assessors are asked to assess and compare qualitative information, notably relevant in the case of narrative CV assessment (see associated idea), but it can be relevant to individual and project assessment alike.
Context and considerations
Experiments in building and providing training for assessors could look at effectiveness of the training in making more reliable qualitative assessment (as recommended in this review). This could investigate assessor confidence in assessing qualitative information, repeated scoring consistency, or any other elements that may impact evaluation reliability after the training has been followed.
While training can be provided by the organisation conducting the assessment, funders also report that allowing assessors to interact with one another (exchange) is useful and valuable in improving the assessment. Exchange discussions could be organised before and during assessment exercises. Assessor-to-assessor (peer-to-peer) training can also help. The peer-to-peer platform PEP-CV could be relevant to look into. https://pep-cv.mariecuriealumni.eu
Challenges and mitigations
Challenge: Uptake of training may be unequal across assessors.
Mitigation: Making training mandatory for assessors, or providing direct exchange and discussion can motivate assessors to undertake the required training.
Evaluating success
Relevant resources and literature
This section includes resources, literature, and reports relevant to this specific experimental idea.
Templates from funders and institutions
Case examples and literature
FNR Luxembourg provides resources to support assessors in the assessment of narrative CVs (see https://www.fnr.lu/new-video-resource-for-funders/ and associated video).
The Royal Society resources on unconscious bias may also help support assessors in understanding how they can minimise bias in qualitative assessment.