Research domains
Weightings in assessment models can be applied to any assessment with a comparative aspect. It is easier to implement where scoring is taking place, but it may also be relevant to highlight elements that are most important in a fully qualitative assessment.
Context and considerations
This process is especially relevant for funding applications. Many funding organisations attribute weightings to specific sections of grant applications, thereby identifying to applicants which sections have more impact on the final scoring of their application.
Weightings is also used in systems evaluations and scoring, for example the UK Research Excellence Framework, which uses weights to highlight how much each category of submissions (Strategy, people and research environment; Contribution to knowledge and understanding; Engagement and impact) is worth in the final scoring of the higher education institution. Similar calculations are used in rankings or country-level R&I comparisions.
While weightings can also be used at individual evaluation level, for example in hiring and promotion, care must then be taken to consider potential impacts of the weighting processes on the selection process (e.g., does it reduce the diversity of candidates, does it create a uniform view of ‘a good researcher’, does it influence practices, etc.).
Challenges and mitigations
Challenge: Weightings provide applicants an indication of what matters most to the evaluatiors. In this regard, they may influence practices, discourage diversity in the pool of applicants, and influence how much worth applicants and researchers place on specific aspects of their work.
Mitigation: Carefully thinking about the possible influence that weights may have in the long term is crucial for responsible assessment. This may include an Equality Impact Assessment and additional analyses to capture possible unintended consequences and gaming that may be generated by the weights imposed.
Challenge: If weights are only known by a subgroup of applicants (e.g., those with connections as assessors), inequalities and unfairness may ensue.
Mitigation: Making sure weighting and assessment criteria are transparent and accessible to all applicants may help promote fair assessment processes.
Evaluating success
Weighting and changing assessment criteria may have impacts on research practices in the long term. Assessments of success should consider longitudinal studies of impact.
Relevant resources and literature
This section includes resources, literature, and reports relevant to this specific experimental idea.
Templates from funders and institutions
Case examples and literature
Many different funding agencies use weighted scoring in their assessment criteria.
An example from the UK Space Agency NSIP programme in Annex C of the Call for proposals: National Space Innovation Programme - International shows how they distribute the weightings between Innovation, Relevance of innovation proposed, Benefit to the UK, and Sound management and planning, with each criteria explicitly detailed and described.
Similarly, Wellcome UK shows how they use weightings in assessing fellowship proposals, distributing weights between skills and experience (25%), project (50%), and proposed plan to foster a positive and inclusive Research Environment (25%) (see here) At Wellcome, the types of elements included in the weightings and the weighting levels differ among the grant programmes to take into consideration the ultimate objective of the grant programmes (see for example how different criteria are weighted in the Mental Health Award).
Other resources
The Multi Criteria Decision Analysis may also provide useful information on different approaches to weighting.