I recently spent a bit of time hacking on a new benchmark, FundingBench. You can see a proof-of-concept here: https://fundingbench.org/
Basically: we give different language models a historical proposal from Manifund, and ask them to evaluate it from the perspective of a funder (like cG, LTFF, or SFF). They predict how much that project will raise, in the proposal year and every year after.
To be clear, this benchmark is currently terribad; don’t trust the numbers it puts out.
Why make a benchmark for grant review?
I wanted to:
Figure out how close we are already to AI-powered grantmaking.
Scott Alexander noted that the AI superforecasters are here. Are the AI supergrantmakers already here? How would we know?
See how different scaffolds and systems do at assisting grantmaking
cG claims that we’re bottlenecked on human grantmakers. But trying to scale AIS philanthropy 100x by hiring 100x more grantmakers is doomed to fail.
Analogy: ForecastBench quantified the value of good scaffolds for the domain of forecasting (+6mo of frontier progress) and time til AIs match superforecasters (now!)
Speculatively: help LLMs get better at reviewing grant applications.
I suspect that the existence of ForecastBench itself helped pave the way for better AI-powered forecasting. (I’m curious if anyone at a lab can confirm.)
The design of FundingBench
The core idea was:
Take 20 Manifund proposals, a mix of standout successes, well-funded, and random other proposals
Trawl the internet for data on how much these proposals were funded, the year the were proposed and every year after
Construct 4 different “funder profiles” (for LTFF, SFF, and cG’s NTAI and GCRO funds)
Ask different LLMs to guess how much a particular funder would fund this proposal, across each year.
Score the LLMs based on how accurate they were
If you like, you can see my original megaprompt for Claude Code here.
Open problems
Memorization. Latest models know that eg Apollo turned out to be successful, so their forecasts of grant funding received are skewed by this.
Data quality. In particular, we know for a fact that the grant data that we have for most of these proposals is incomplete or wrong. I haven’t tried at all to make this better, mostly because I’m still just experimenting with the concept.
Scoring. The current scoring function is some weird thing Claude invented that I don’t trust. Carol left me some feedback on better scoring approaches here; I’m currently most excited for a “portfolio” approach.
Funder profile. I agree that currently, this is probably overcomplicated and we should just drop this piece of the benchmark. But longer term, trying to figure out whether a particular model can act under a specific donor’s preferences — is the hard important problem to solve with grantmaking.
If you have thoughts on any of these topics, hit me up at austin@manifund.org!
Related links
ForecastBench: https://www.forecastbench.org/about/
Conceptual Reasoning Index: https://conceptualreasoning.ai/
Ivan Bercovich on TerminalBench: https://ivanbercovich.com/2026/good-benchmarks
https://jaredheyman.medium.com/on-rebel-theorem-4-0-55d04b0732e3, h/t Derik Kauffman
Dynamic vs static benchmarks: here
I really resonate with Naufal’s point here, especially having just gone through a job change in Taiwan recently. It hits close to home: when evaluation systems rely strictly on historical benchmarks, they often struggle to recognize non-standard potential.
In Austin Chen’s FundingBench, using past funding decisions as the ground truth creates a tricky dilemma. If a benchmark only rewards models for matching past human choices, it inadvertently trains them to double down on status quo bias. Genuine breakthroughs—much like unconventional career pivots—rarely look like the past. They bring new terminology, frameworks, or paradigms, which causes historical models to default to low scores simply because there's no precedent.
We see this everywhere in everyday systems:
AI HR Screening: Resume tools built on classic employee profiles often penalize applicants with non-linear paths or domain switches (feels a bit too personal after my recent job search 😅), treating anything unfamiliar as high risk.
Credit Scoring for Creators: Risk engines flag indie developers or solopreneurs simply because their income streams don't look like standard monthly payslips.
To be honest, I've felt that exact frustration before. There were times when, just to gain validation or get a single chance to prove what I could do, I felt forced to lower my own value. Deep down, I knew I could deliver better outcomes than what traditional templates asked for. But as long as you sit outside the rigid rules of the "system"—whether you appear to have less traditional experience or fewer visible proof points on paper—the doors stay shut.
This is why Naufal’s concept is so powerful. It goes far beyond evaluating grantmakers. If we can teach systems to recognize uncertainty and look past rigid historical distributions, we won't just build better AI benchmarks—we'll start uncovering the true, hidden gems in every domain that the world currently overlooks.
such a cool experiment!