Large Language Model (LLM) Evaluation Research Grant

Overview

Meta is pleased to invite university faculty to respond to this call for research proposals for LLM evaluations.

As AI models become increasingly complex and capable of responding to queries, and also acting on the world and reacting to the results of their actions, new benchmarks will be required to assess these advanced behaviors. These standards are critical to expand the scope of who the models can reach, how they interact with humans, and the types of tasks they can perform, while ensuring that these models can be trusted.

To foster further innovation in this area and deepen collaboration with academia, Meta is pleased to invite university teams to submit research proposals that contribute to building a new library of evaluations within three main areas: complex reasoning, emotional & social intelligence, and agentic behaviors.

A total of up to 4 awards of $200,000 are available each - and each award is inclusive of overhead. Funding will be provided to RFP winners pursuant to a Sponsored Research Agreement (SRA) containing open science terms. Please note that the terms of the SRA will not be subject to negotiation. We strongly encourage researchers from diverse backgrounds and of diverse abilities to apply.

We encourage submissions that utilize evaluations in the following areas:

1. Complex Reasoning: As AI advances and becomes increasingly capable of interacting with humans and taking actions on their behalf, we need methods for understanding their reasoning and planning capabilities. How would we measure different types of intelligence and reasoning effectively as these models approach and/or surpass human performance? Proposals with a special focus on complex reasoning are encouraged, including a focus on complex mathematical reasoning, tool integration, hierarchical planning, memory and infinite context window, System 1 vs System 2 reasoning, multi-step deduction and induction, analogical thinking, and narrative analysis.
2. Emotional and Social Intelligence: If AI models and agent systems of the future are expected to interact with humans effectively, they will need not only cognitive intelligence but also emotional and social intelligence—the ability to understand human emotions and sentiments, respond emphatically, "read the room", and generally adapt their communication as a capable human assistant would. How do we measure these capabilities in a model or system? Proposals with a special focus on evaluating different facets of emotional and social intelligence, such as emotion recognition and comprehension, empathy, intent understanding, identity and personality, persuasion, inspiration, theory of mind, conversation management, and multimodal emotion recognition, are encouraged.
3. Agentic Behavior: Llama 3 has been integrated into Meta AI, Meta’s intelligent assistant. What datasets, environments, or other assets will the academic community need to assess progress agents’ abilities to successfully complete tasks of interest for users? Proposals with a focus on taking actions in web, desktop, and app environments, continuous training, steerability (especially for conversational tone and safety settings), personalization, and retrieval capabilities in the system, are encouraged.

Prior to submitting a proposal, please confirm that your institution will agree to a contract for sponsored research from Meta. The agreement will operate under “Open Science” terms meaning that all research results from this grant will be made available for publication or public release into the public domain. Application materials include:

• A summary of the project (2-4 pages), in English, explaining the area of focus, a description of techniques, any relevant prior work, and a timeline with milestones and expected outcomes;
• A draft budget description (1 page) including an approximate cost of the award and explanation of how funds would be spent (including a line for overhead and the required travel to the Open Innovation AI Research Community Workshop in London, from October 29th - October 30th, 2024*)
• Curriculum Vitae for all project participants; and
• Organization details; this will include tax information and administrative contact details.

• Proposals must comply with applicable U.S. and international laws, regulations, and policies;
• Applicants must be current full-time faculty at an accredited academic institution that awards research degrees to PhD students;
• Applicants must be the Principal Investigator on any resulting award; please note that only one grant recipient will be awarded per proposal.
• Meta cannot consider proposals submitted, prepared or to be carried out by individuals residing in, or affiliated with an academic institution located in, a country or territory subject to comprehensive U.S. trade sanctions; and
• Government officials (excluding faculty and staff of public universities, to the extent they may be considered government officials), political figures, and politically affiliated businesses (all as determined by Meta in its sole discretion) are not eligible.
• The research period for all proposals is one year, or 12 months from the date of receiving the grant.
• Grant recipients must be able to attend the Open Innovation AI Research Community Research Workshop in London, UK from October 29th - 30th, 2024.

Note: Proposals can utilize any permissively licensed LLM and are not limited to a requirement to use Meta’s Llama models in the work.

All eligible Proposals will be reviewed and rated by Meta during September 2024.

Important Dates:

• Launch date: Friday, August 9th, 2024 at 9am PT
• Deadline: Friday September 6, 2024 at 11:59pm PT
• Estimated notification date: The first week of October 2024

Every year, the Research Community comes together for an annual workshop - this year, the workshop will be held at Meta’s London offices from October 29th - 30th, 2024. Our goals are to share the research findings within the community, provide an opportunity for new collaborations, and network with other researchers. Grant recipients are required to attend to present their research and their proposals at the workshop. Please make sure to include travel for this workshop into your budget.

If you are not able to attend this workshop, please include a note in your proposal or identify an alternative member from your team who could represent your project at the workshop.

Yes, you may partner with researchers at other universities in your proposal. Please note however, that grants are only awarded to the organization employing the Principal Investigator of the proposal and cannot be split.

Large Language Model (LLM) Research Grant Finalists

In August 2024, we launched our first Research Grant specifically for academic researchers, looking for innovative proposals to build new benchmarks in the areas of general intelligence, emotional intelligence and agentic behaviors for large language models. We are now excited to announce the four finalists of this grant!

The INCLUDE project will reset standards for multilingual evaluation, developing benchmarks to assess LLMs in 100+ languages on region-specific knowledge and real-world reasoning that is rooted in the authentic settings where these languages are used

This project focuses on systematically evaluating and improving large language models' capabilities in complex, logical reasoning, with an additional emphasis on legal reasoning and argumentation.

This project introduces modular and adaptive frameworks powered by dynamic AI agents to advance the reasoning capabilities of vision and language models for fine-grained spatial understanding and 3D perception tasks

CodeArena is a next-generation benchmark for generative AI in software development, encompassing a diverse and realistic set of tasks, including bug fixing, responding to code reviews, test generation, multilingual code migration, and other essential workflows critical to modern software engineering.