Meta AI is pleased to invite university faculty to respond to this call for research proposals about the Dynabench platform. Launched in 2020, Dynabench is a research platform for dynamic data collection and benchmarking.
Static benchmarks have well-known issues. They saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts and have unclear or imperfect evaluation metrics. The Dynabench platform is a scientific experiment to answer the question: Can we make faster progress if we collect data dynamically, with humans and models in the loop, rather than in the old-fashioned static way? The aim is to continuously challenge existing benchmarking dogma and try to embrace dynamic solutions at all times.
The Dynabench platform initially comprised four core tasks in natural language processing. It has now reached the level of maturity where we hope it will be useful to the broader research community. Meta aims to open up the platform for anyone interested in human-and-model-in-the-loop data collection to run their own task, and is soliciting proposals for interesting tasks and experiments that Meta can help realize.
To foster further innovation in this area and deepen our collaboration with academia, Meta AI is pleased to invite university faculty to submit research proposals pertaining to Dynabench. A total of 4 to 8 awards are available, in the range of $15,000 to $55,000 each, including overhead in an amount up to 40% of project costs. Funding will be provided to RFP winners pursuant to a Sponsored Research Agreement (SRA) containing open science terms. Please note that the terms of the SRA will not be subject to negotiation. We strongly encourage researchers from diverse backgrounds and of diverse abilities to apply.
The deadline for submissions is March 23, 2022.
Meta AI encourages creative submissions along the theme of “rethinking benchmarking” that leverage the Dynabench platform, with areas of interest that include, but are not limited to, the following:
New tasks or benchmarks that would benefit from having a dynamic (adversarial) component.
Experiments that examine concrete questions about the human-and-model-in-the-loop paradigm for data collection or evaluation.
Proposals for moving Dynabench beyond natural language processing and/or beyond the English language.
Proposals must comply with applicable U.S. and international laws, regulations, and policies;
Applicants must be current full-time faculty at an accredited academic institution that awards research degrees to PhD students;
Applicants must be the Principal Investigator on any resulting award;
Meta cannot consider proposals submitted, prepared or to be carried out by individuals residing in, or affiliated with an academic institution located in, a country or territory subject to comprehensive U.S. trade sanctions; and
Government officials (excluding faculty and staff of public universities, to the extent they may be considered government officials), political figures, and politically affiliated businesses (all as determined by Meta in its sole discretion) are not eligible.
