Objectives and potential outcome

Wildcards, or golden tickets, are the option given to reviewers, once per call, to unilaterally choose a proposal of their liking for direct funding. The idea is that a reviewer would spend their golden ticket on a proposal that they think is particularly promising, but that the reviewer suspects – or knows – may not be reviewed as positively by the rest of the panel, e.g., because of its risky, innovative, or niche nature. Therefore, wildcards can be considered a possible intervention to promote researchers, research topics and research ideas that are traditionally biased against by reviewers, including applicants from disadvantaged demographic groups, interdisciplinary projects, and high-risk high-reward project ideas.

What happens if this is successful? What changes if this works? The adoption of a wildcard rule will be successful if it has a positive, observable impact on funding decisions. Success criteria may include: - Reviewers use their wildcards. Not meeting this goal means that no project is deemed worthy of a wildcard, and/or that reviewers do not want to take reputational risks in the review panel by unilaterally proposing a specific project. - Wildcard are spent on projects that would not be funded otherwise. Not meeting this goal means that wildcards have no impact whatsoever on funding decisions. - Proposals funded via wildcard have a distinct profile of desirable attributes – e.g., they are from traditionally disadvantaged groups of applicants, or are about traditionally disadvantaged topics. - Reviewers on the panel – both those that spent their wildcard and those who did not – see value in the wildcard option.

What improvements are we aiming for? The introduction of the wildcard rule aims to mitigate the impact of negative biases by individual reviewers and by the discussion dynamics of peer review panels.

*If there are multiple experiments fitting, put them here** There are two different flavors of wildcard rules that may have distinct advantages and disadvantages. These were never discussed in the literature – as far as I know – so I’m taking the liberty of giving them new labels: - “Postal” wildcards. This is when reviewers may use their wildcard in the postal review stage, i.e., prior to knowing what other reviewers think of the proposal. Supposed advantage: reviewers would be even less biased in their choice of project by not being exposed to the opinions of others. Disadvantage: wildcards may be more likely to be spent on projects that would be funded regardless, meaning that the wildcard would be ‘wasted’. - “Panel” wildcards. This is when reviewers may use their wildcard during or after the panel discussion, i.e., after having learned what the other panel members think of the proposal. Supposed advantage: the reviewer can use their wildcard more strategically, and not waste it on proposals that would be funded anyway. Disadvantage: reviewer’s choice of using the wildcard would be skewed by the information they’d have collected during the panel meeting. For example, a reviewer may shy away from using a wildcard on an excellent, high-risk high-gain project, after learning that all of the other panel members strongly dislike that project.

Research domains

Wildcards can be introduced in any funding call where it is the funder’s priority to mitigate bias against some traditionally disadvantaged groups of applicants or proposal characteristics that are not blind to the reviewers.

Context and considerations

A wildcard or ‘golden ticket’ could be added to any type of review panels where reviewers assess multiple applications (>2) and where a final decision is based on the joint scoring or decisions of multiple review panel members scoring the same application. This idea is most applicable to research funding assessment where multiple applications may be funded, and would be difficult to implement in a setting where only one application is selected such as in hiring decisions.

Challenges and mitigations

Challenge: A strong challenge to the evaluation of the impact of wildcards is numerosity. Few projects will be funded via wildcards that would not be funded otherwise. This means that it will take many years to accumulate sufficient observations to quantitatively evaluate the impact of wildcards. It is however difficult to envision ways of mitigating this challenge.

Challenge: By providing review panels with a wildcard or golden ticket, there is a possibility that biases and personal preferences permeate in such decisions.
Mitigation: While biases may not be completely cancelled, they could also permeate the way scoring is attributes by the reviewers in a normal decision process. A possible mitigation could be to ask for a justification for the decision, including a justification for why the reviewer anticipated that the proposal may not be funded, and why they believe it should nonetheless be funded.

Challenge: While wildcards may be implemented to strenghten novelty and risk taking in funded applications, there are discussions suggesting that they may favouring novelty as they may be “more effective in supporting projects that develop entirely new theories, concepts, methods, or tools (new vintage)”.
Mitigation: Introducing a dedicated novelty panel, or relying on lotteries may be more appropriate if the ultimate objective is to promote high risk, novel research.

Evaluating success

Relevant resources and literature

This section includes resources, literature, and reports relevant to this specific experimental idea.

Thomas Feliciani, Junwen Luo, Kalpana Shankar, Funding lotteries for research grant allocation: An extended taxonomy and evaluation of their fairness, Research Evaluation, Volume 33, 2024, rvae025, https://doi.org/10.1093/reseval/rvae025 describes a wildcard model – also called a bypass model – where favoured proposals bypass a lottery system. The article describes wildcards as “the opportunity given by the funder to individual reviewers to arbitrarily choose one or a few proposals to be funded, even though the rest of the review panel would disagree.”

Templates from funders and institutions

Case examples and literature

The funders Villum Fonded and Volkswagen Stiftung currently implement wildcards in at least some of their calls. The U.S. NSF (see https://doi.org/10.1038/d41586-023-03933-3) has also been working towards their implementation, although to date (October 2025) there are no publicly available reports on how this implementation is proceeding.

Other resources

Paula Stephan & Chiara Franzoni (2023) Encouraging high-risk high-reward research at NIH, https://www.brookings.edu/wp-content/uploads/2023/05/StephanFranzoniFinal-3.pdf. Associated papers may be found at https://ifp.org/building-a-better-nih/

Case Studies or Implementation Examples