We’re aiming to raise $450k, and think we can effectively deploy up to $1.2m to run more great events, improve our space, and incubate new fellowships. To kick things off, one anonymous donor has offered Mox a 1:1 match up to $100k -- if we can raise $50k+.

Please support us through our Manifund page, or read on to learn more about our work!

Donate

Interested in donating a large amount? Reach out to Rachel Shu, Mox Director at rachel@moxsf.com.

Our first year in review

Mox launched on Feb 15, 2025. In a single year on a shoestring budget, we’ve become a primary nexus for the AI safety community and other EA-adjacent work in San Francisco. We’ve hosted hundreds of members and thousands of visitors, organized our own public events and fellowship, and supported dozens of other impactful orgs.

By the numbers:

• 183 active members

• 15 private offices

• 377 events hosted

• Partnerships with 19 AI Safety & EA orgs

• 4 floor (40k sq ft) buildout

Fellowships & programs

We supported five residencies in the last year:

• FLF Fellowship on AI for Human Reasoning: 30 fellows exploring, researching, and developing potential beneficial AI for Human Reasoning tools.

• PIBBSS Fellowship 2025 (now Principles of Intelligence): 17 fellows in residence for cross-disciplinary AI safety research.

• Seldon Lab Accelerator, Batch 1: 4 startups building AI safety infrastructure, including Andon Labs, Workshop Labs, and Lucid Computing.

• Seldon Lab Accelerator, Batch 2: 6 startups, currently in residence.

• The Frame Fellowship: An 8-week program for 8 video creators communicating about AI safety, developed in-house at Mox.

We also offer our space for retreats, workshops, and hackathons, serving orgs like Tarbell, 1DaySooner, Elicit, Asimov Press, FutureSearch, and AVA International.

Finally, we work with local conferences, hosting pre/post conference coworking and side events for EAG Bay Area, LessOnline, Manifest, and The Curve.

How does Mox contribute to these programs’ success?

• Provides a fully furnished office

• Situates them alongside other groups doing similar work

• Provides event venue space directly connected to their workspace

• Handles daily catering, janitorial and supplies

• Troubleshoots participant tech

Public Events

We hosted 377 events over the last year, including:

• Senator Scott Wiener on AI safety legislation — live Q&A with CA Senator and author of SB-53.

• Sentient Futures Summit — a 350+ attendee, 3-day conference in February 2026 focused on AI and Animal Welfare, as featured in the SF Standard.

• Models in Moral Mazes — Anthropic research scholars previewed an unpublished paper on misalignment at Mox before public release.

• Man vs Machine Hackathon (METR × Factory.AI) — a 300 person, live RCT on AI coding agent productivity

• Q&As and fireside chats with Joe Carlsmith, Eli Lifland, Nate Soares, Scott Aaronson, Joel Becker and Bryan Caplan.

Mox also hosts recurring community events, such as:

• Effective Altruism SF, biweekly events and meetups

• Astral Codex Ten SF, monthly meetups

• 90/30 Club, machine learning paper reading group

• Mathematics with Lean, the interactive theorem prover

Mox has been an invaluable resource for us when running EA SF [Effective Altruism San Francisco], since its large and well-equipped facility allowed us to cater food, run speaker events, workshops, and otherwise host much larger and more ambitious events than we otherwise would have been able to.

— Lead organizers of EA SF

Individuals & Coworking

We currently have 183 active members; on a typical coworking day, 50-80 people are at Mox. A sampling of individual members who are frequently at Mox:

• Justin Kuiper, AI safety video producer

• Ross Rheingans-Yoo, philanthropic investor

• Itsi Weinstock, Senterra Funders

• Kamile Lukosuite, GovAI

• Joshua Levy, Holloway

• Ronak Mehta, AI safety researcher

You can see a list of all members here: https://moxsf.com/people

Testimonials from our August 2025 feedback survey:

It feels like a second home, but more lively. I can always expect to run into a friend who is down to cowork or hang.

— Constance Li, founder of Sentient Futures

I can walk up to anyone and have an interesting conversation; every single person I’ve met here has welcomed questions about their work and been curious about mine.

— Gavriel Kleinwaks, Horizon Fellow

Mox has the best density of people with the values & capabilities I care about the most. In general, it’s more social & feels better organized for serendipity vs any coworking space I’ve been to before, comparable to perhaps like 0.3 Manifests per month.

— Venki Kumar

Private offices and partner organizations

In Year 1, Mox was home to 15 private offices, including:

• Sentient Futures: promoting animal welfare and sentience research

• Tampersec: building physical computing infrastructure security

• Andon Labs: building autonomous organizations such as Project Vend, via Seldon accelerator

• Pantograph: building a preschool for robots

• BlueDot Impact (pending visa): online courses for AI safety upskilling

We also maintain a Guest Program with 19 partner organizations, providing complimentary drop-in access for a variety of orgs we highly respect. Public program partners include: MIRI, FAR.AI, Redwood Research, Palisade, GovAI, Epoch, AI Impacts, Timaeus, Elicit, Evitable, FAI, and MATS.

Our teammates visit San Francisco a couple of times a month. Instead of renting a coworking spot, Mox gives us a familiar space with friendly faces that we reliably run into. It feels closer to going to the college library with friends than to an office. We hang out there for many hours after our work is done!

— Deger Turan, CEO of Metaculus

Past funding and budget updates

Grant updates

In our initial fundraising post a year ago, we proposed three budget tiers — minimal ($1.6M/year), mainline (~$2M), and ambitious ($3.6M).

What we spent annualized to roughly $1.2M, less than even our ‘minimal’ tier projection. What we delivered landed closer to ‘mainline’: 183 members, 144 Guest Program participants, 15 offices, a team of 5, and 2 tentpole events most months. And from the ‘ambitious’ tier, we succeeded at expanding Mox to all four floors of 1680 Mission.

Mox operates on a lean budget; we believe our per-member and total costs compare favorably to other AI safety hubs such as Constellation, Lighthaven, and LISA. We’ve done this by keeping our team small, finding good deals on rent and furnishings, and charging fair prices to our members and clients. We expect monthly revenue to continue growing by $10-15k/mo for the next 3-6 months, with offices and memberships both scaling steadily, and project steady state expenses to be ~$150k/month.

See our monthly revenue and expenses spreadsheet: May 2025 - Jan 2026

Budget basics

Grant funding (Craig Falls & EAIF): $300k
Grant funding (Manifund): $500k
Revenue to date: $600k
Projected revenue in 2026: $1.4m

• 35% — Memberships

• 35% — Offices

• 20% — Programs

• 10% — Events

Projected costs in 2026: $1.5m

• 40% — Labor

• 30% — Rent & utilities

• 25% — Office supplies

• 5% — Event costs

(all figures approximate)

Upcoming Plans

1. Grow and improve our main offerings

Events:

• More major conferences like Sentient Futures Summit

• More public talks with key speakers like Senator Weiner

• Improve our first floor and make it highly usable and more publicly accessible, building our ability to provide a good space, which mostly shows up in the impact we have, and somewhat in revenue.

Programs:

• Serve repeat cohorts of the fellowship programs that have used our space so far

• Additionally serve 3-7 new fellowships and workshops in this coming year

Coworking:

• Continue growing our community of individual members to 120-150 daily users, 300+ total members

• Maintain the ability to select private offices based on fit, rather than market rate

• Create additional meeting rooms and other communal areas in the coworking space

2. Attract international talent via Global Expert Fellowship

A key part of our second-year vision is the Global Expert Fellowship: hosting independent researchers, domain specialists, and builders through J-1 visa programs to create new frontier technology collaborations within the Mox community and internationally. Learn more here.

This may be the highest-impact thing we can achieve this year. It has immediate external impact by enabling independent researchers to quickly enter the US to do work, and it strengthens Mox by expanding our network of high-quality talent. Mox is in a rare position to pull this off, as we are able to meet State Department requirements for visa-qualifying cultural exchange which many other organizations cannot.

3. Incubate new workshops and programs

We have an advantage in creating our own programs, sourcing from the talent pool we’re developing.

Upcoming example: the Muybridge Fellowship for Visual Interpretability, which would bring together technical visual and interactive pioneers to improve the presentation of mechanistic interpretability research and broaden its accessibility. This builds on the experience gained running the existing Frame Fellowship.

Support Mox

Thank you for reading this! We’d appreciate your support, whether via:

• A direct donation (through Manifund, or contact Rachel Shu at rachel@moxsf.com)

• Sharing this fundraiser with other potential donors

• Leaving a comment about your own experience of Mox

And of course, we’re always on the lookout for excellent members, orgs, or events to work with us; please send them our way!

Donate

Intro

EA was founded on the principle of cost-effectiveness. We should fund projects that do more with less, and more generally, spend resources as efficiently as possible. And yet, while much interest, funding, and resources in EA have shifted towards AI safety, it’s rare to see any cost-effectiveness calculations. The focus on AI safety is based on vague philosophical arguments that the future could be very large and valuable, and thus whatever is done towards this end is worth orders of magnitude more than most short-term effects.

Even if AI safety is the most important problem, you should still strive to optimize how resources are spent to achieve maximum impact, since there are limited resources.

Global health organizations and animal welfare organizations work hard to measure cost-effectiveness, evaluate charities, make sure effects are counterfactual, run RCTs, estimate moral weights, scope out interventions, and more. Conversely, in AI safety/x-risk/longtermism, very few efforts are spent on measuring impact. It’s hard to find anything public that compares the results of interventions. Perhaps funders make these calculations in private, but one of the things that made Givewell so great was that everything was out in the open. From day 1, you could see Givewell’s thinking on their blog, examine their spreadsheets, and change parameters based on your own thinking.

I’m making a first attempt at a “GiveWell for AI safety”. The end goal is to establish units to measure the cost-effectiveness of AI safety interventions, similar to DALY/$ in global health. Scott Alexander’s survey from last year is a great example of different kinds of relevant units for AI safety work. Given units like these, a donor or funder can look at the outputs of different AI safety orgs and “buy” certain things for far cheaper than they can for others.

To begin, I’m looking into the cost-effectiveness of AI safety communications. I’m starting with communications because it’s easiest to get metrics, and the outputs of comms work are publicly viewable (compared to e.g., AI policy work) and more easily assessable (compared to e.g., technical AI safety work). For this post, I’m going to be specifically focused on YouTube videos, where metrics were easiest to gather; I’m introducing my framework for evaluating different YouTube channels, along with measurements and data I’ve collected.

Step 1: Gathering data

The end goal metric I propose for AI safety videos' cost effectiveness is “quality-adjusted viewer-minute, per dollar spent”. I started by collecting data for channel viewership and costs.

Viewer minutes

To calculate time spent watching videos, I first wrote a Python script to query the YouTube API for views and view lengths for every video in specific channels. I multiplied these, and adjusted by a factor for the average percentage of a video watched: 33%, based on conversations with creators. I also messaged creators asking them for their numbers, using those directly where possible. For any creators that responded, I included their metrics (which were usually screenshotted, for authenticity).

Costs and revenue

To measure cost-effectiveness, we also need to estimate the costs of making the videos. In addition to direct costs (such as equipment and editing), I consider the value of time by people producing videos to be a major cost. This is because they could be doing other things, such as earning to give. Because of this, I generally ask people to estimate their market rate salaries if they are doing the work unpaid. For example, for AI in Context, this included the salaries paid to 80000 Hours employees for the production of their video. For other creators who had lots of personal savings and weren’t getting paid, I asked them to include the value of their time.

Some channels and podcasts also produce revenue through ads and sponsorships. This is a very good thing and is a sign that people want to see the content. In fact, I expect the best content will be able to self-fund after some time, and even be profitable. That said, for now, most aren’t profitable and subsist on donations; thus, I count revenues as offsetting costs for cost-effectiveness calculations, because this funding was produced organically, though I’m open to treating this differently.

Results

Before we get into quality adjustments, here’s a snapshot of where different channels stand:

Step 2: Quality-adjusting

For quality adjustment, there are three factors I introduce: the quality adjustment for the audience, Qa, the quality adjustment for the fidelity of the message, Qf, and the alignment of the message, Qm. So the overall equation is:

Quality-adjusted viewer minute = Views × Video length × Watch % × Qa × Qf × Qm

Quality of Audience (Qa)

This quality adjustment is meant to capture the average quality/importance of the audience. The default audience is set to 1 as if the average person watching is an average person (totally random sample from the human population). If an audience member is more influential, has more resources, or is otherwise more impactful on the world, this number goes up; vice versa if it goes down. At the extreme, you might think of a value of 0 for someone who is about to die alone on their deathbed right after they watch the video, and perhaps as high as 1,000,000,000 for an audience of just the President of the US. Other things that make a viewer valuable are things like being at a pivotal time in their career, being extremely intelligent, etc.

This is comparable but not directly correlated with CPM (cost per thousand views, aka how much advertisers would pay to advertise to this person). This value could, in theory, be negative (it'd be better for this person not to watch this video/we'd pay for a person who was about to watch the video to have their internet shut down and the video not load), though I’d suggest ignoring that.

Normal values for this factor will be between 0.1 and 100, but should center around 1-10.

Fidelity of Message (Qf)

This refers to how well the message intended for the audience is received by the audience, on average, across all viewer-minutes. It attempts to measure the importance of the message and how well the message is conveyed for your goal. If your goal is to explain instrumental convergence or give the viewer an understanding of mechanistic interpretability, how well the video conveys the message is what is being measured here, alongside how important that message is.

An intuitive way to grasp this metric is to consider how much you’d rather someone watch one minute of a certain video compared to a reference video. For now, I am somewhat arbitrarily setting the reference video to be the average of Robert Miles’ AI safety videos. I’m seeking a better reference video; perhaps one that is more widely known or is simply considered to be the canonical AI safety video. If you would trade off X minutes of watching the video in question for 1 minute of the average Robert Miles video, then the Qf factor is 1/X.

Normal values of this number for relevant videos will be between 0.01 and 10.

Alignment of Message (Qm)

This factor refers to the message being sent relative to your values. This value will range from -1 to 1 and where a value of 1 is “this is the message I most want to get across to the viewers” and a value of -1 is “this is the exact opposite of the message I want to get across to viewers. For example, if your most preferred message to get across is “change your career to an AI safety career” and the message a particular video portrays is “pause AI” which you prefer half as much, Qm for this video is 0.5 and if the exact opposite message you want to portray is “accelerate AI as fast as possible”, you’d give a value of -1.

Importantly, this is perhaps the most subjective of the factors and depends greatly on your values

Results

Here are my results:

Feel free to make a copy of the Google sheet, insert your own values, play around with it, and compare things, as you can with Givewell’s spreadsheets. I intend to update the master spreadsheet with info as I receive it (costs, viewer minutes, etc.), so I don’t recommend you change non-subjective values. I’ve added comments to explain the estimates I made in the Google Sheet. I’m seeking recommendations for the best way to capture estimates vs. reported data, or any other suggestions.

Observations

The top cost-effectiveness comes from creators that are monetizing their content (AI Species and Cognitive Revolution) as well as well-produced videos of typical YouTube length (5-30 minutes), and not from long podcasts or short-form videos.

We’re seeing that for 1 dollar, good AI safety YouTube channels generate on the order of 150-300 QAVM (or about 2.5-5 hours).

Some things to compare this to.

• Global average income is about $12k/year or about $6/hr. We could pay people this wage to watch videos; this would be 10 viewer-minutes per dollar.

• We could pay to promote existing videos on YouTube. CPM on YouTube videos is about $0.03/view of about 15 seconds, which would be $0.12/minute or about 8 VM/$

• We could pay to show video ads in other locations (eg, Super Bowl ads). A 30-second ad on Super Bowl LIX would have cost $8M and been shown to 120M US people, or ~8 viewer-minutes per dollar - with perhaps a 3x audience quality adjustment for primarily US viewers, you get 24 QAVM/$.

Of course, these comparisons don’t include the cost of video production itself.

Other notes from this exercise:

1. Over the course of this, I asked a lot of people, formally and informally, who they thought of as AI safety communicators, YouTubers, etc. Essentially, everyone said Robert Miles was the first that came to mind, and a few said that they have careers in AI safety, at least partially due to his videos. This led me to make the “audience quality” category and rate his audience much higher.

2. Here I am measuring average cost effectiveness, though we probably want to be doing this at the margin. While this is a different exercise, I think it should still be illuminating and should serve as a good benchmark we should be aiming for.

3. Where is Dwarkesh? Austin and I argued for a while on this — Austin thinks Dwarkesh should be included as someone whose channel reaches important people, while I think that very few of Dwarkesh’s videos count towards AI safety. I informally surveyed a bunch of people by asking them to name AI safety YouTubers, and Dwarkesh’s name never came up. When mentioned, he was not considered to be an “AI safety YouTuber”.

4. Many creators seemed to want me to include their future growth. I think this is perhaps a bit too subjective and would introduce a lot of bias. These view counts are a snapshot at this point in time, but when deciding what to fund, you should also consider growth rates and look to fund things that could reach a certain cost-effectiveness bar.

5. I don’t think that this post should cause large-scale shifts in what gets funded or what people do, but I do think cost-effectiveness is one of the things people should be looking at for the projects they pursue

How to help

The main thing I need is data, both of metrics and of costs, including the salaries of people spent making these videos. A lot of data is particularly hard to come by, and otherwise, I make estimates. Thanks to all the people who responded to my emails and text messages and gave me data already!

For next steps, I am planning to expand this into all media of AI communications (podcasts, books, articles, signed letters, article readings, website visits), collect data on metrics and costs, and come up with quality adjustments. If you make content or work for an org that has data, please message me at marcus.s.abramovitch@gmail.com.

The end goal of this remains to cross-compare different “interventions” in AI safety, like fieldbuilding (MATS) and policy interventions (policy papers, lobbying efforts,) and research (quality/quantity of papers). Stay tuned!

Appendix: Examples of Data Collection

Rob Miles

The Donations List Website indicates Rob has received ~$300k across 3 years from the LTFF. Since Rob has been making videos for ~10 years, I am estimating it takes ~$100k/year for a total of $1M. I used my scraper and estimated here of ~33% average watch time of video per view for ~31.6M viewer minutes. I will update this one as soon as Rob responds to me, since Rob is considered the canonical AI safety YouTuber.

AI Species (Drew Spartz)

Drew told me he spent $100k for all the videos on his channel, and including his time (~1 year at $100k/year), $200k would be fair for total resources spent. I used my script to pull views for each video, and estimated that about 50% of each view was a full watch. I sent this to Drew, and he told me 30-35% was more realistic, and then he gave me his raw data. This was very helpful since it allowed me to calibrate % watch time for various video lengths.

Rational Animations

I looked at Open Phil grants, which totaled $2.785M. FTX FF gave a further $400k. I then added ~10% for other funding from individuals or other grantmakers that didn’t show up. I scraped their data for views from the YouTube API and used a 33% watch-through rate.

After this, Rational Animations confirmed to me that they in fact spent more and received more grants, which summed to $4,395,132.

AI in Context

Aric Floyd told me that 80k spent $50.75k on the video and that staff time on the video summed to ~$75.2k for a total of $126k. He also sent me his watch data for the video.

Cognitive Revolution

I asked Nathan, and he said I was approximately correct that $500k ($250k of which is on production and ~$250k paid to him as salary) is about what has been spent over the last couple of years for production. He also shared his YouTube data and suggested that he gets more engagement from audio-only. (Not to worry, Nathan, I’m analyzing this next. It’s just hard to get data.)

Cognitive Revolution is unique in that they have very substantial revenues because of sponsorships and YouTube views. This somewhat breaks my formulas for cost-effectiveness since donations aren’t required. In other words, Cognitive Revolution makes money. Therefore, I’m conservatively estimating the value of Nathan’s time here to be a bit higher, given his experience. Thus, $250k is spent on production, and $250k/year is spent on Nathan’s time. For 2 years, I have been considering the total cost of Cognitive Revolution to be $750k with $500k in revenues, though I am very open to changing these numbers. I don’t know the best way to treat Nathan’s podcast given all these variables.

Personally, I’m skeptical that pausing AI would be good, and also don’t know whether these protests are effective for achieving that goal. But Manifund aims to be a neutral platform, supporting different viewpoints on how to do good, recognizing that we’re not always sure what is the right thing to do. In that spirit, I wanted to share this list of protests with you all. Enjoy!

Why I’m writing this

Like Cate, I (Rachel Shu) think it’s inevitable that public opposition to AI will continue to grow quickly over the next two years. In particular, I think there is a medium-sized-but-salient chance (33%?) of some flashpoint in the next two years during which millions of people worldwide will be angry enough to lend their voice to protests against increasingly powerful AI. Most likely this flashpoint will center around large-scale labor replacement, or possibly a mismanaged government surveillance initiative, rather than around existential risk.

When such flashpoints happen, they create political momentum that advocacy groups, labor unions, civil rights organizations, and other actors can mobilize to advance their agendas. The organizations best positioned to capitalize on this moment will be those with existing infrastructure, clear policy demands, and the resources to scale up quickly. Mapping out these potential actors, and the actions they’re taking, will prove essential for understanding and potentially influencing how such a movement develops and what it ultimately accomplishes.

What this series will cover

To build this awareness, I’m writing this as the first of a series of posts to track global protest movements against AI and AI companies, both those protests concerned with existential risk from AGI and also of more immediately pressing concerns.

The scope of this series is specifically mass advocacy: for example, demonstrations, strikes, phone banks, petitions, and the like; as opposed to higher-level/insider political work, which I don’t have any comparative advantage in explaining. The series will cover both actions specifically protesting AI and also actions against AI organizations for reasons that aren’t centrally about AI. Each post will also devote some to explaining the context and history of specific issues or players. Hopefully this roundup will be useful to policymakers, journalists, historians, and protest organizations themselves.

I’ll analyze of some recent actions and the organizations behind them, and at the bottom will list upcoming actions and meetings I know of for anyone interested in getting involved!

Recent demonstrations (since June 1, 2025)

July 15, Pittsburgh, USA: Indivisible, Sunrise Movement, ACT UP — “Stop the Summit”

Hundreds of people attended a series of demonstrations which took place against the Pittsburgh Energy and Innovation Summit at CMU, in which President Trump gave a speech, jointly organized by various advocacy groups. Riot police were deployed to disperse the protests. The summit was organized around the creation of AI datacenters in Pennsylvania, and in response the protests featured speeches discussing datacenter environmental impacts and the perceived prioritization of the international AI arms race over local concerns. From the signage, it seems the protests were also a fairly generalized anti-Trump, anti-corporate affair, rather than specifically about investment in AI.

July 14, Seattle, USA: Jewish Voice for Peace (JVP) — “Purge Palantir”

Jewish Voice for Peace Seattle staged a demonstration against Palantir’s Seattle headquarters, and about 120 people attended. Other JVP chapters and sympathetic organizations staged concurrent protests at Palantir locations around the US, including Denver, New York, and Palo Alto, each of which drew dozens. JVP is an organization of Jewish-Americans who oppose Israeli actions in Palestine and support a ceasefire.

Palantir Technologies provides surveillance services to federal, military, and police organizations, including Immigration and Customs Enforcement (ICE) and the Israel Defense Forces (IDF). Given that these two organizations have been the targets of recent large demonstrations in the US, it’s unsurprising that Palantir is also under fire for supporting them.

Previously six protestors were arrested at Palantir’s NYC offices on June 26 during a protest organized by climate justice group Planet Over Profit and immigrant rights group Mijente.

July 13, Mexico City, Mexico: Mexican Association of Commercial Announcements (AMELOC) — “Creative industry workers united for the urgent regulation of artificial intelligence”

Dozens of media professionals attended voice actor protests in Mexico City for the regulation of AI voice cloning. The inciting event was the unauthorized and unattributed use of a deceased voice actor’s voice by the National Electoral Institute, a Mexican public agency.

According to this Instagram post from the organizer, there are proposed regulations on the table in the Legislative Assembly of Mexico City (a federal district) which they are agitating in favor of. These would require consent, attribution, and royalties for the use of voice likenesses.

Previously in 2023, SAG-AFTRA and the WGA led 6-month strikes in the USA during which one of the major themes was AI use in media. SAG-AFTRA subsequently led a second strike of voice actors on the issue of AI use in video games.

July 8, Geneva, Switzerland: Boycott, Divestment, and Sanctions (BDS) — “End UN Partnership with Genocide-Enabling Tech in ‘AI for Good’ Conference”

The AI for Good summit is organized by several dozen UN and sponsored by many major tech companies, many of which have partnerships with the Israeli military or civilian government. Israel controls one of the most technologically advanced militaries, and deploys AI tools for mass surveillance and surgical precision strikes in Gaza. Around 100 activists gathered at Geneva's Broken Chair monument for this protest. Anadolu Agency quotes an activist as saying “The UN is delegitimizing itself with this conference, with hosting big tech at an AI for Good conference during an AI-powered genocide.”

July 7, San Francisco, USA: Sunrise Movement Bay Area — “Billionaire Bailout Bill vs The People #bullytherich”

A small gathering of Sunrise Movement members protested outside of Sam Altman’s house, in particular against Sam Altman’s support of the Big Beautiful Bill. Sunrise Movement is nominally a climate change focused organization, but it runs campaigns in support of a wider base of left-coded causes, such as racial and economic justice. Minor themes of this particular action included protesting AI datacenter energy use and AI job replacement.

Another Sunrise Movement chapter also featured in the Pittsburgh protests linked above.

July 2, USA: No Azure for Apartheid — “When a child dies in Gaza, we get paid for aiming the air strike”

No Azure for Apartheid claims that Microsoft, especially through its Azure division and partnership with OpenAI, has deep investment in Israel; it seems that much of the Israeli military relies on Azure as a platform. No Azure for Apartheid, founded in October 2024, has staged a string of in-person and online protests against the sale of AI weaponry to Israel and broadly attempting to get Microsoft to divest from Israel. Microsoft workers taking part in these actions are more or less signing up to get fired.

June 30, London, UK: Pause AI —”AI companies are less regulated than sandwich shops”

Pause AI led a protest against Google DeepMind for releasing Gemini 2.5 Pro without honoring previous commitments on how they would go about risk assessment for newly released AI models. The organization’s name might be misleading to some; they are not specifically in favor of an AI moratorium like the widely-known 2023 petition of a similar name, but rather support members who take a broader range of positions from increased regulation to full cessation of AI development. About 60 people took part in this protest.

Pause AI has previously led protests against other frontier AI labs.

June 12, Annecy, France: Belgian association gathering authors and creators of animation (ABRACA) — “GenAI Seeks Not To Support Artists, But To Destroy Them”

Annecy is host to the world’s premier animation festival, taking place this year June 8 - June 14. Animators are increasingly under pressure due to advances in AI generated videos, and animation unions have been nearly unanimously in opposition to the widespread deployment of genAI in the entertainment industry. This protest was led by ABRACA (Animation workers union of Belgium) and supported by over 20 other European media and entertainment industry labor unions.

May 22-July 1, USA: Grassroots advocacy against AI state regulation moratorium clause

A federal moratorium on state AI laws was proposed by Ted Cruz, but ultimately removed from the One Big Beautiful Bill Act before it was signed into law. The Senate removed the provision in a 99-1 vote, enabling states to continue regulating AI. This has been covered better elsewhere, and in my understanding was mostly a higher-level policy push rather than a grassroots effort, so I’ll just point at some advocacy groups that helped with the push by phone banking: Pause AI, and Encode Justice, among others.

Upcoming demonstrations and actions

A protest being listed here doesn’t constitute an endorsement. Contact the author if you want to be listed.

July 25, San Francisco: Stop AI — “Close OpenAI!”

Self-description:

“OpenAI is threatening our jobs, our lives, and our loved ones' lives. JOIN THE FIGHT! […] We demand that the US government shut down OpenAI, close any other company building AGI, and permanently ban the development of AGI.”

Upcoming publicly advertised meetings and other events

San Francisco, CA, US:

Stop AI: Bar meeting every Wednesday 6-8 pm at Kiitos Sports Bar in San Francisco.

Berkeley, CA, US:

Stop AI: Bar meeting every Sunday 6-8 pm at Raleigh's Pub in Berkeley.

Bay Area, CA, US:

Sunrise Movement: Monthly virtual hub meetings.

If you’d like to help with this effort, please comment, reach out in a private message, or message me on signal: @wearsshoes.77. I’m looking both for tips about upcoming or ongoing actions, and also advice on how to shape this series going forward! I’m planning to publish this at a monthly cadence initially, faster when start to things pick up.

Thanks to Rachel Weinberg, Caithrin Rintoul, Ross Rheingans-Yoo, Gavriel Kleinwaks, Venki Kumar and others for assistance.

On June 25, U.S. Representative Raja Krishnamoorthi opened a Congressional committee meeting by talking about AGI.

“Basically,” Krishnamoorthi’s opening statement explained, “it's AI that meets or exceeds human capabilities and can take action without human intervention. […] Whether it's American AI or Chinese AI, it should not be released until we know it's safe. That's why I'm working on a new bill—the AGI Safety Act—that will require AGI to be aligned with human values and require it to comply with laws that apply to humans.”

This “AGI Safety Act” has not been formally introduced, and we don’t know what it might ultimately contain. But Krishnamoorthi’s statement, among others, clearly shows that some members of Congress are open to talking about and legislating on AI safety. This could be a key moment for AI policy, and so it might also be a key moment for donors who want to support AI policy orgs, especially orgs that work on building relationships with members of Congress. In this post, I want to explain why individual donors are especially important to those orgs, and why large foundations can’t just fund all of the policy work that they think is important. If you are an American citizen interested in donating thousands or tens of thousands of dollars towards AI policy and governance, this opportunity might be for you! (The rest of us will just get a little insight into American politics and policy.)

The fundraising grind

Members of Congress have full schedules of committee hearings, briefings, floor votes, meetings with their constituents, meetings with other Congresspeople, etc, but their schedules are also shockingly full of fundraising. It’s been reported that sitting members of Congress spend as much as half of their work time fundraising. They’re on the phone for hours, going down long lists of potential supporters, calling them one after another to ask for donations. A 2016 CBS segment got some horrifying quotes from sitting Representatives:

“[Before 2010] I'd have to put in about an hour, maybe an hour and a half, at most, two hours a day into fundraising. And that's the way it went until 2010, when Citizens United was enacted. At that point, everything changed. And I had to increase that to two, three, sometimes four hours a day, depending on what was happening in the schedule.” - Rep. Steve Israel.

“Both parties have told newly elected members of the Congress that they should spend 30 hours a week in the Republican and Democratic call centers across the street from the Congress, dialing for dollars.” - Rep. Rick Nolan.

As Rep. Nolan’s quote suggests, this extreme fundraising schedule isn’t purely an individual decision for each politician. Members of Congress are expected to raise large sums of money for their parties, not just their own campaigns. And these “party dues” are higher if you are on the most powerful legislative committees – so, in practice, you won’t be assigned to the most powerful legislative committees unless you can raise significant amounts.

This all means that if you give money to a politician’s campaign, alongside whatever effect you may have on their (re-)election odds, you can save them a lot of time and grief, and even enable them to actually serve in their current elected office.

Not all dollars are created equal

Some donation dollars are much more useful to a candidate than others. Under the current laws governing federal elections, a donor can give a maximum of $3,500 per election directly to a candidate’s campaign. (State-level elections have different rules in each state, but 38 of the 50 states also have this kind of donation limit.) Political Action Committees (PACs) can give a bit more: a qualifying PAC can contribute up to $5,000 per election. Other entities, like corporations and 501(c)(3) nonprofits, can’t contribute to campaigns or to PACs at all.

These legally-limited contributions, from individuals and PACs directly to a campaign, are the most valuable form of support: the highly desirable “hard dollars”. Candidates are allowed to use this money for any legitimate campaign expense. Most importantly, they can run ads to reach and convince voters.

Famously, “super PACs” can raise and spend unlimited amounts of money. However, the candidates themselves don’t control this kind of spending. By law, candidates and their campaigns can’t be involved in the creation of a super PAC’s messages in any way. So from a politician’s or a party’s perspective, while they appreciate super PAC spending on their behalf, they would definitely prefer “hard dollars” that they can use to run their own ads with their own messages.

This means that a new donor who can give, say, $3,000, is much more valuable to a politician than an additional $3,000 from someone who has already given significant amounts. This in turn opens up an opportunity for smaller donors to collectively have an impact that’s not available to a single large donor. Networks of small donors, who care about an issue and are willing to coordinate, can be very influential.

The power of coordination

Donors who are more willing to coordinate will have an easier time building relationships with politicians and leading the discussion on a policy issue. For example, with no coordination, individual donors might each independently contribute to a campaign, and each independently tell the candidate that issue X is important to them. This approach leaves it up to the candidate and their team to proactively notice that many different voters and donors care about X; then, even if they do notice, there will be no obvious way to learn more about X and discuss possible next steps. But if donors work together, they can collectively host X-themed fundraisers; they can unite behind specific experts and spokespeople, who can then build ongoing relationships with politicians; they can pursue a strategic mix of supporting new candidates who would champion issue X if elected, and also advocating for issue X with more established politicians.

Examples of successful donor networks:

• OutGiving and the Gill Action Fund. The Gill Action Fund was founded in 2005 by Tim Gill to advocate for marriage equality and related LGBT issues. OutGiving, the network of LGBT and allied donors that Gill had founded a decade earlier, partnered with the Action Fund to direct their individual donations to pro-equality candidates and ballot measures. After same-sex marriage was legalized nationwide in 2015, and with a majority of Americans supporting marriage equality, the Gill Action Fund wound down its operations, and the OutGiving network moved away from direct political donations.

• The American Israel Public Affairs Committee, or AIPAC, is a pro-Israel lobbying organization. It has a large pool of affiliated donors, including “club members” who commit to contributing at least $1,800 per year. These contributions are bundled and directed to candidates across the country. Until 2020, AIPAC hosted a large annual policy conference; past speakers include Presidents Trump, Biden, Obama, and Bush. AIPAC’s policy positions have been influential in Congress for decades.

• EMILYs List is a PAC that aims to elect pro-choice Democratic women to office. Each election cycle, the organization endorses candidates in local, state, and federal races across the country. Affiliated donors can support these candidates by giving to EMILYs List, which bundles the contributions and sends them to the campaigns. EMILYs List also recruits and trains potential candidates, and runs an affiliated super PAC called Women Vote.

Giving opportunities

Concretely, I know of two AI policy orgs that would benefit from a network of individual donors concerned about AI risk. The AI Policy Network (AIPN) advocates for policies to help the United States prepare for rapid advances in AI capabilities, focusing on national security and on maintaining human control of AI. Americans for Responsible Innovation (ARI) advocates for policies in the public interest on a range of AI issues, including current harms (e.g. nonconsensual deepfakes), national security (e.g. enforcing export controls), and emerging risks (misuse and misalignment).

You can reach out to these orgs if you are interested in supporting their work or hearing about upcoming donation opportunities. Note that AIPN and ARI are both 501(c)(4) nonprofits, so donations to them, like donations to political campaigns, are not tax-deductible. ARI also has a 501(c)(3) sister organization, the Center for Responsible Innovation, which is recommended by Founders Pledge.

Some cautionary notes

When thinking about whether you should make some of your donations as part of an AI policy donor network, consider:

• Some people who work on AI safety have worried that getting involved in politics will result in the idea of AI risk becoming (ahem) politicized. In particular, you might worry that AI safety will become associated with one party, and then the other party will be negatively polarized against the idea.

• On the flip side, if a donor network works with politicians all across the political spectrum, that will probably include supporting politicians that you, personally, strongly disagree with on other issues. You may not be willing to do that! (You can still participate in a donor network even if you're not willing to support all the politicians they might work with, but the more politicians you put in that category, the less valuable your participation in the network becomes.)

• Remember that the point of building relationships with politicians, talking to them about AI risks, and electing more people who care about this issue, is ultimately to write and enact helpful legislation. I would only recommend being part of an AI policy donor network if you agree that there are specific policy asks that would mitigate AI risks.

Last time, I wrote about some considerations for AI safety grant evaluation, but didn’t actually ship a cost-effectiveness model. Since then, Austin, Nishad, and I have:

• Developed Manival, an LLM-powered grant evaluator

• Demoed it to an audience at Manifest

• Written and applied our own grantmaking criteria—we’ll see if Manival can replicate our taste

How Manival works

This is effectively a form of structured ‘Deep Research’.

First, we specify the fields that matter to us when evaluating a grant. These might include ‘domain expertise of project leads’ or ‘strength of project’s theory of change’. We have RAG-based ‘data fetchers’ (Perplexity Sonar) scour the internet and return a score, with reasoning, for each of these fields. We then feed these into an LLM synthesizer (Claude Opus), which provides an overall evaluation.

This is a pretty janky LLM wrapper compensating for the lack of Deep Research API. We’re aware of various RAG and Deep Research alternatives, and expect our evaluations to improve as we plug better models in.

Customizing the criteria

Different people have different ideas of what should go into a grant evaluation config. Austin cares deeply about how great a team is; I’d like mine to consider counterfactual uses of a team’s time.

With Manival, you can apply any grant evaluation criteria of your choosing (go to Configs → AI Generate). Here’s one we made just for fun:

What’s next for Manival?

Manival has lots of potential uses. Here are some main ones:

1. Estimating marginal cost-effectiveness: We could write a config that estimates how much of a difference marginal $ <x> would make.

2. Predicting impact market cap: Right now, our configs evaluate projects on a scale from 0 to 10. In the real world, project size varies: some established projects seek 6-7 figures like a ‘Series A’; others seek 4-5 figures in ‘seed’ / ‘pre-seed’ funding. Can we use Scott Alexander’s impact valuations to estimate a project’s ‘impact market cap’?

3. Improving project proposals: Grant applicants can run their project proposal through Manival to understand what might need clarifying.

4. Project comparison: We can use Manival to rank a category on Manifund, funnelling its most underrated projects to the top of your feed.

5. Recommendations: We can use Manival to recommend new projects to grantmakers based on projects they’ve already supported.

6. Solving the ‘adverse selection’ / ‘funging’ problems: Grantmakers can estimate how likely a project might be to get funding elsewhere, or better understand why it hasn’t been funded when that’s the case.

It might be valuable to simulate how other grantmakers you respect might evaluate a project when deciding whether to make a grant. For example, here’s a simulation of Joe Carlsmith’s thinking:

These ‘simulated scores’ might differ from how a grantmaker actually thinks. Accordingly, we plan to develop configs that are maximally faithful to our own thinking over the next week.

For now, I expect a lot of Manival’s value to come from ‘flagging potentially great projects to look into’, rather than being something people defer to.

We’re excited for you to try Manival, and eager to know what you think, especially if you’re a donor, grantmaker, or someone else who cares a lot about evaluating grant proposals. Schedule a call with us to chat this through, or let us know in the comments!

The premise: EA was founded on cost-effectiveness analysis—why not try this for AI safety, aside from all the obvious reasons¹? A good thing about early GiveWell was its transparency. Some wish OpenPhil were more transparent today. That seems sometimes hard, due to strategic or personnel constraints. Can Manifund play GiveWell’s role for AI safety—publishing rigorous, evidence-backed evaluations?

With that in mind, I set out to evaluate the cost-effectiveness of marginal donations to AI safety orgs². Since I was evaluating effective giving opportunities, I only looked at nonprofits³.

I couldn’t evaluate all 50+ orgs in one go. An initial thought was to pick a category like ‘technical’ or ‘governance’ and narrow down from there. This didn’t feel like the most natural division. What’s going on here?

I found it more meaningful to distinguish between ‘guarding’ and ‘robustness’⁴ work.

Some reasons you might boost ‘guarding’:

1. You think it can reliably get AI developers to handle ‘robustness’, and you think they can absorb this responsibility well

2. You think ‘robustness’ work is intractable, slow, or unlikely to be effective outside large AI companies

3. You prioritize introducing disinterested third-party audits

4. You think ‘guarding’ buys time for ‘robustness’ work

Some reasons you might boost ‘robustness’:

1. You want more groups working on ‘robustness’ than solely AI developers

2. You think ‘guarding’ work is unlikely to succeed, or be effective against advanced models, or is fragile to sociopolitical shifts

3. You prioritize accelerating alignment / differential technological progress

4. You think ‘robustness’ work makes ‘guarding’ efforts more effective

Finally, a note on ‘robustness’. I don’t expect safety protocols that work on current models to generalize to more capable models without justification and concerted effort. Accordingly, I think it makes sense to separately classify orgs whose theory of change (ToC) focuses on superintelligent systems.⁵

Doing so—and categorizing other helpful work as ‘facilitating’—we get a typology roughly as follows:

As you can see, ‘technical’ and ‘governance’ orgs fall all over the map. Some orgs are particularly hard to categorize—e.g. CHAI has outputs that plausibly fall in all four quadrants—so I’ve tried to focus on orgs with narrower remits. Redwood, GovAI, and RAND are placed solely for the agendas stated. Some work (like METR’s) facilitates work east or north of it. In this diagram, which quadrant / quadrant boundary an org belongs to carries meaning, but where it lies within that does not. The horizontal axis may collapse depending on your beliefs.

I think this typology is very much open to contention. Its main point is to convey how I arrived at exploring the funding landscape for ‘robustness’ orgs. I’m interested in the effective giving prescription regarding these, particularly those tackling superintelligence risks, because (in roughly decreasing order):

1. I think their work is less legible to outsiders than ‘guarding’ or ‘facilitating’ work.

2. I think evaluation of their work is comparatively neglected, given other funders’ current foci.

3. I endorse differential technological development, with safety outpacing capabilities.

4. I’m concerned about the fragility of a guarding-only strategy, particularly in fast takeoff scenarios.

   1. I think having concrete plans for a ‘pause’ may strengthen the case for one.

When I searched for nonprofits working on superintelligence (ASI) safety ‘robustness’, I got a list as follows:

FAQ:

Why are Redwood included despite being categorized as ‘prosaic focus’ above?

They have several public stories of how their work might contribute to ASI alignment. I think more orgs should have public stories like this!

Why is <org> included / excluded?

I categorized based on what I could easily find online. This method reflects what an external donor might see. It’s also imperfect, and I probably missed some orgs. Corrections are appreciated, particularly if I missed orgs that are currently fundraising.

Why is there so much theory?

The amount of compute required for empirical ASI work favors for-profit hosts. Yes, it’s possible this is the work that matters most. Sought / upcoming post: a theory of what safety-relevant goods it’s nonprofit orgs’ comparative advantage to provide.

—

Because we’re looking for effective giving opportunities, we want to focus on the ‘Actively seeking donations’ bucket. (We also want to make it easier for orgs to share when they’re in that bucket!)

Of the six such orgs I identified, all but Orthogonal have sought funds on Manifund⁶. Indeed, I only identified Coordinal Research, Luthien, and SaferAI because of this. Most of these are small orgs (<10 FTE). I don’t claim this list is exhaustive: if told of another org addressing ASI threat models, I’ll include it.

I’m pretty unsure how to rank the six highlighted orgs actively seeking donations. High-quality, transparent evaluation and comparison would help.

Hang on, isn’t this OpenPhil’s job?

A major concern of would-be donors is that if OpenPhil / Longview / SFF / LTFF / … chose not to fund / recommend these projects, they might not be worthwhile⁷. I’d like to dig into this assumption.

Will OpenPhil fund all worthwhile ‘direct work’ orgs seeking six figures for operating expenses? I can see that they funded Timaeus, several months after Manifund supported Timaeus with an initial start-up grant. They seem to fund a decent amount of academic work, as does Longview.

As a prospective donor, I’d like to understand why OpenPhil has funded Timaeus but not the other orgs on this list (it’s possible the others haven’t applied). I’d also appreciate a fuller list of Longview’s grants: it’s hard for me to understand their decision-making from only three featured AI grants.

It’s unfortunate that OpenPhil doesn’t share more about their grantmaking process: the director of CAIP, an org denied OpenPhil funding, claims not to know “what criteria or measurement system they’re using.”⁸

If OpenPhil and Longview could share more of the reasoning behind their grantmaking, that could really help other donors make effective giving decisions. It could also help orgs reflect more critically on their own activities!

—

Manifund is well-positioned to help great new projects get off the ground, as shown in the case of Timaeus. It also has its own limitations—often the only information available to prospective donors is an org’s own funding call, and sometimes the fact they haven’t gotten funding elsewhere.

Over the course of this investigation, it became apparent to me that Manifund lacks the resources to evaluate every org manually. With that in mind, here are some feature ideas I have to help address the ‘adverse selection’ issue. Manifund plans on implementing some of these – I’m also curious for you, reader, to share what’s missing!

1. ‘What the experts say’ sidebar:

• Explicitly invite expert commentary and highlight endorsements

• Offer verified anonymous input channels via:

  • Fully anonymous posts

  • Aggregated feedback (visible only to donors or the fundraising org)

2. One-click granular reacts (with optional elaboration):

   • “Right idea, wrong team”

   • “Best this falls, making space for something new”

   • “Low-probability, high-upside bet: support”

   • “Accelerates capabilities at least as much as alignment”

   • …

3. ‘Why Manifund [rather than / in addition to other funders]’:

   • Invite orgs to explain why they haven’t found funding elsewhere

   • Invite other funders to publish their assessment criteria

4. Recommendations welcome!

I hope this will make it easier for donors to find the best giving opportunities, and for great projects to get funded.

Thanks Austin Chen, Justis Mills, Nick Marsh for feedback.

Some ‘robustness’ orgs currently fundraising: CAIS, Coordinal, Luthien, CORAL.

Some ‘guarding’ orgs currently fundraising: SaferAI, CAIP.

Some ‘facilitating’ orgs currently fundraising: Asterisk, AIFP, Apart, BGP.

¹Long feedback loops, high downside risks, to name a few.

²Some others’ work on this topic: OpenPhil (criteria only), Larks (org-by-org), Longview (criteria only), Zvi Mowshowitz (org-by-org), Ben Todd (spotlighting orgs).

³This is an obvious limitation: lots of safety work happens within for-profit companies. Sought / upcoming post: a theory of what safety-relevant goods it’s nonprofit orgs’ comparative advantage to provide.

⁴’Robustness’ is an overloaded term. But I wanted to describe ‘making the underlying systems safer’: calling this ‘solutions’ would’ve been a disservice to ‘guarding’. Tough semantics.

I call ‘guarding’ everything outside the blue line, which includes white-box evals, black-box evals, audits…whereas the name might imply everything outside the red line, which includes control and hardware mechanisms. I think my classification makes sense, because some safety/control mechanisms may be either ‘baked into’ a model or ‘external’ to it. Everything within the blue line could be shown to an evaluator to argue that a model is safe.

⁵I categorized based on what I could easily find online. This method reflects what an external donor might see. It’s also imperfect, and I welcome corrections.

⁶I tried to minimize selection effects, but I expect I missed at least one org, which makes this statement weaker.

⁷Thanks, Larks, Nanda, and Linch for articulating the ‘adverse selection’ problem.

⁸Source

This is why Manifund runs our regranting program: to quickly fund promising projects, by delegating budgets of $50K-$400K to experts. As we’ve just announced our 2025 regrantors, now is a good time to review past regrants -- some we think were great, and others that weren't such good fits.

We think our regranting program is great for donors who want to seed ambitious new projects, care about moving fast, and appreciate transparency. If you want to fund our 2025 program, please contact us!

About the author: Jesse Richardson recently joined Manifund after working at Mila - Quebec AI Institute, and also has a background in trading on prediction markets. This post is basically Jesse’s low-confidence, hot takes.

Three awesome AI Safety regrants…

What makes a great regrant? We look for early-stage projects that need quick funding, opportunities OpenPhil might miss, and chances to leverage our regrantors' unique expertise. Some of our favorites are:

• Scoping Developmental Interpretability to Jesse Hoogland - the first funding for Timaeus, accelerating its research by months

• Support for deep coverage of China and AI to ChinaTalk - reporting on DeepSeek, ahead of the curve

• Shallow review of AI safety 2024 to Gavin Leech - quick regrants inducing further funding from OpenPhil & others

Scoping Developmental Interpretability - the first funding for Timaeus, accelerating research by months

This regrant was made in late 2023 to Jesse Hoogland and the rest of what is now the Timaeus team, for the purposes of exploring Developmental Interpretability (DevInterp) as a new AI alignment research agenda. Four regrantors made regrants to this project totalling $143,200: Evan Hubinger as the main funder, alongside Rachel Weinberg, Marcus Abramovitch and Ryan Kidd. Evan had previously mentored Jesse Hoogland as part of MATS, and therefore had additional context about the value of funding Hoogland’s future research. This is the sweet spot for regranting: donors may have the public information that Evan Hubinger is an expert who does good work, but regranting allows them to leverage his private information about other valuable projects, such as DevInterp.

Regarding the grant itself; success for this project looked like determining whether DevInterp was a viable agenda to move forward with, rather than producing seminal research outputs. I recommend reading more about DevInterp if you’re interested, but my shallow understanding is that it aims to use insights from Singular Learning Theory (SLT) to make progress on AI alignment through interpretability, focusing on how phase transitions in the training process lead to internal structure in neural networks.

I’m not well placed to form an inside view on how likely DevInterp was/is to succeed, but this proposed research agenda had numerous things going for it:

• it was novel; the application of SLT to alignment was largely unexplored prior to this work,

• it seemed to be well thought out; the LessWrong write-up included plenty of detail about why we might expect phase transitions to be a big deal and how this would relate to alignment, as well as a solid six-month plan,

• it had an element of “big if true” i.e., it may be unlikely that the strong version of the DevInterp thesis is true, but this research has potential to make meaningful progress on AI alignment if it is

These are all markers of projects I am excited to see funded through Manifund regranting. In addition to the agenda itself, I also think this was a good team to bet on for this kind of work; they seem capable and have relevant experience e.g. ML research, and running the 2023 SLT & Alignment Summit.

This regrant is a strong example of where Manifund’s regranting program can have the biggest impact: being early to support new projects & organizations, and thereby providing strong signals to other funders as well as some runway for these organizations to move quickly. In this case, Manifund’s early funding helped Hoogland’s team get off the ground, and they subsequently started a new organization (Timaeus) and received significantly more funding from other sources, such as $500,000 from the Survival & Flourishing Fund. It’s probable that they would’ve gotten this other funding regardless, but not guaranteed, and I’m happy that Manifund helped bring Timaeus into existence several months sooner and with increased financial security. Hoogland notes:

Getting early support from Manifund made a real difference for us. This was the first funding we received for research and meant that we could start months earlier than we otherwise would have. The fact that it was public meant other funders could easily see who was backing our work and why. That transparency helped us build momentum and credibility for developmental interpretability research when it was still a new idea. I'm pretty sure it played a significant role in us securing later funding through SFF and other grantmakers.

In terms of concrete outcomes, there’s a lot to be happy with here. Timaeus and its collaborators have published numerous papers on DevInterp since this regrant was made, and it seems that DevInterp’s key insight around the existence and significance of phase transitions has been validated. My sense is that the question of whether DevInterp is a worthwhile alignment research agenda to pursue has been successfully answered in the affirmative. It’s also nice to see strong outreach and engagement with the research community on the part of Timaeus: November 2023 saw the first DevInterp conference, and they’ve given talks at OpenAI, Anthropic, and DeepMind.

Support for Deep Coverage of China and AI - reporting on DeepSeek, ahead of the curve

In 2023 & 2024, Manifund regrantors Joel Becker and Evan Hubinger granted a total of $37,000 to ChinaTalk, a newsletter and podcast covering China, technology, and US-China relations. ChinaTalk has over 50,000 subscribers and is also notable for the quality of its coverage and the praise and attention it receives from elites and policymakers.

Before this regrant, ChinaTalk had been run by Jordan Schneider and Caithrin Rintoul, both part-time, on a budget of just $35,000/year. What they were able to accomplish in that time with those limited resources was impressive, and I believe merited additional funding, even just to allow Jordan to work on this full-time. More funding would also have meant ChinaTalk bringing on a full-time fellow who, per Jordan, “would be, to my knowledge, the only researcher in the English-speaking world devoted solely to covering China and AI safety.” ChinaTalk has since received further funding and is in the process of growing to five full-time employees, but we would’ve loved for this to happen sooner through an expanded regranting program.

Even putting aside the specific track record of ChinaTalk, it seems clear to me that the intersection of China and AI safety is an incredibly important area to cover, and at a high level it is valuable to fund organizations that are doing this kind of work. It can be hard to imagine plausible scenarios of how the next decade goes well with respect to AI that don’t run through US-China relations, and I am persuaded by Jordan’s case that the amount of energy currently being expended on this is grossly inadequate.

Since the first regrant, ChinaTalk’s Substack audience has grown from 26,000 subscribers to 51,000 and they’ve put out regular high-quality content, including an English translation of an interview with DeepSeek CEO Liang Wenfeng, coverage of chip policy, and what important 2024 elections in the US and Taiwan mean for China. The ChinaTalk team has expanded to six people, allowing for a greater diversity and quantity of coverage, including YouTube videos. Jordan has also announced plans for launching a think tank—ChinaTalk Institute—this year, in a similar vein to IFP.

Among their varied coverage, I was particularly impressed to see how ChinaTalk was ahead of the curve in covering the rise of DeepSeek, while most of the West seemed to be taken by total surprise in January 2025. As a trader and forecaster, this advance insight might have been worth a lot of money to me through anticipating the market freakout, suggesting I should pay more attention to ChinaTalk in the future.

ChinaTalk has continued on the strong trajectory it was on in late 2023, and it was great that Manifund was able to support ChinaTalk in this success. For more information about why this grant was likely good ex ante, I encourage you to look at regrantor Joel Becker’s comment on the subject. Joel’s detail about why ChinaTalk was at the time insufficiently funded

Philanthropists are scared to touch China, in part because of lack of expertise and in part for political reasons. Advertisers can be nervous for similar reasons… Jordan was hoping to support this work through subscriptions only.

makes me more optimistic that this regrant was the kind of thing the program should be doing: plugging holes in the funding landscape.

Shallow review of AI safety 2024 - quick regrants, nudging OpenPhil & others to donate

Gavin Leech co-wrote Shallow review of live agendas in alignment & safety in 2023, which was well-received and considered a useful resource for people looking to get a top-level picture of AI safety research. Given that it was intended to be a shallow review, this post has a lot of helpful detail and links for various research agendas, e.g., the amount of resources currently devoted to each, and notable criticisms.

Last year, he sought funding to create an updated 2024 version of this post. He received $9,000 from Manifund regrantors Neel Nanda and Ryan Kidd, as well as $12,000 from other donors through the Manifund site.

Big picture, I believe there should be an accessible and up-to-date resource of this kind; for people who are starting out in AI safety and don’t know anything, for funders trying to get a sense of the landscape, or for anyone else who might need it. In 2022 I was at a stage where I wanted to contribute to AI safety but didn’t know anything about it and was unsure where to start, and I would’ve likely found Gavin’s review useful, along with the other resources that existed. Based on this, Gavin’s record in a variety of fields, and the quality of the 2023 version, I think this regrant looked promising.

The new post (Shallow review of technical AI safety, 2024) came out in December 2024 and appears to be similarly comprehensive to the 2023 version, although it has gotten less attention (~half the upvotes on LessWrong and not curated). That’s probably a bit worse of an outcome than I would’ve hoped for, but I still would have endorsed this grant had I known the result in advance. Presumably the updated version is less eye-catching than the original, while still being necessary.

The funding of this project also shows the advantages of the Manifund regranting program. Gavin asked for between $8,000 (MVP version) and $17,000 (high-end version) and was quickly funded for the MVP by Neel and Ryan. He then got an additional $5,000 from OpenPhil, after Matt Putz learned about this proposal via our EA Forum post; and a further $12,000 from other donors. I am happy with how the regranting program is both able to provide the small amount of funding to get a project off the ground, and increase visibility of that project so that other donors can step in and fund it to a greater extent. A couple of small negatives: (1) regrantor Neel Nanda is less optimistic than I am that this was a particularly good grant and (2) the high-end version was supposed to include a “glossy formal report optimised for policy people” which didn’t get made (OpenPhil opted against funding it), however the excess money is instead going towards the 2025 edition. I look forward to it!

… And three that maybe weren’t a good fit for regranting

• Pilot for new benchmark to EpochAI - a limited-info regrant that might have accelerated AI capabilities

• AI Safety & Society to CAIS - cool new project on AI safety writing, less exciting for regranting specifically because the regrantor was also leading the project

• General support to SaferAI - an org doing important work in AI policy, but might be growing too fast

Pilot for new benchmark by Epoch AI - unfortunately confidential

This regrant consists of $200,000 from regrantor Leopold Aschenbrenner to Epoch AI to support the pilot of a new frontier AI benchmark. Epoch is a research organization that does some great work on forecasting AI progress, AI hardware, the economics of AI, and more. For this reason, they are by default a good candidate for funding through our regranting program. [Editor’s note: in fact, we’ve asked Tamay to serve as a regrantor for 2025!] However, there are a few reasons why I am less excited about this regrant than many of the other projects that could’ve been funded.

Firstly, this proposal presented very little information, for confidentiality purposes. While I understand why this was necessary, the lack of transparency is not ideal from Manifund’s perspective. Part of the value we hope potential grantees can get from the regranting program is increased awareness of their project, so that other funders can support them, and hopefully to improve understanding of what work is being done in AI safety. The lack of detail here makes that difficult.

Secondly, I think the program works best when regrantors give smaller amounts to many projects, compared with the (relatively) larger sum given to Epoch here. We generally want to be taking risks on getting smaller projects off the ground, with the hope that other funding sources can take over if/when their funding needs exceed what Manifund can offer. Part of our comparative advantage lies in the flexibility and speed of the regranting program, so I place more value on regrants that lean on that advantage. On the other hand, this is a pilot, so the proposal does fit the loose remit in that sense. I would also be happier about the relatively large grant if it came from multiple regrantors on the site (as with Timaeus), as that’s further evidence that a proposal is worth funding according to multiple different experts.

Thirdly, related to the above, Epoch has already received significant funding from OpenPhil and SFF, suggesting they wouldn’t benefit most from Manifund’s support. Aschenbrenner remarks that Epoch has other funding, but not for this project. I wonder whether that’s a signal that this project is not as valuable as Epoch’s other work, in the eyes of the Epoch leadership team and their previous funders?

Finally, while I highly value a lot of Epoch’s forecasting work, I think there is growing evidence that benchmarks of this kind help accelerate AI capabilities. The new reasoning-via-RL paradigm for frontier LLMs means that one of the biggest bottlenecks to AI progress is now high-quality reward signals to train on, which new benchmarks may assist in providing. This is evident in Epoch’s FrontierMath benchmark, which OpenAI commissioned and has sole access to (except for a holdout set). While the math nerd in me thinks FrontierMath is really cool, it seems to me that OpenAI thought creating this benchmark would be to their benefit, either through directly speeding up their progress or for marketing reasons. I find this concerning. We don’t know exactly whether this new benchmark would accelerate AI capabilities, or what the ownership structure of it would be, but that ultimately comes back to difficulties stemming from lack of transparency. I would be more confident about the positive value of this regrant if there were more information about this thorny question. There is significant disagreement about whether contributing to AI capabilities is in fact negative for the world, and I want to defer some to that uncertainty, but my personal view is that it does increase the probability of global catastrophe and I am therefore more pessimistic about this regrant. I would’ve preferred to see our regrantors fund a wider array of projects, as well as projects that are more transparent and ideally carry less downside risk.

AI Safety & Society - a direct donation may have made more sense

AI Safety & Society (AISS) was the proposed name for a new platform by the Centre for AI Safety (CAIS) designed to foster high-quality AI safety discourse, filling a gap between social media and formal academic publications. This platform launched two weeks ago, under a new name: AI Frontiers. It received a regrant of $250,000 from Manifund regrantor and CAIS Executive Director Dan Hendrycks, who is also advising the project and is now Editor-in-Chief of AI Frontiers.

There’s a lot here that I’m excited about: I agree with the need for more good AI safety writing, and ensuring that AI safety ideas reach a broader audience. I also trust Dan and the CAIS team to execute this well, and they have a lot of the right contacts and experience to get this off the ground. Looking at the articles published on AI Frontiers in the last two weeks, I’m impressed with the quality and scope.

Despite the positives, I’m unconvinced this was a great outcome for the Manifund regranting program, because it involved a regrantor using their budget to support their own organization’s project. To be clear, I think this was a reasonable use of money. It makes sense that Dan Hendrycks’ reasons for supporting this project professionally also lead him to think it’s a good place to direct funding. Yet this doesn’t strike me as making valuable use of the regranting program, compared with other funding mechanisms. Consider the two main ways a hypothetical donor might want to defer to Hendrycks’ judgment as an AI safety expert:

• they might specifically value the work he’s doing with CAIS and projects such as AI Frontiers, or

• they might value his ability to discover and rate AI safety work more generally

The former situation should probably lead that donor to just donate to CAIS directly; the latter is a case that the regranting program was specifically designed to assist with. This regrant feels more like the former, which I don’t think is a problem in itself, but I would feel less optimistic about continuing the regranting program if this is the main way in which it was used. That being said, I hope this project succeeds and if AI Frontiers does have a big positive impact, I might end up thinking that the specifics of how it got funded were less important than getting it funded at all.

This post should not be seen as knocking AI Frontiers— a nascent project that could be highly impactful— but rather articulating a view on the best use cases for regranting. Looking forward, I wouldn’t want Manifund regrantors to rule out funding projects they’re involved in, but such a decision should perhaps merit some extra skepticism about whether this is the best use of a regrantor budget.

[Editor’s note: we reached out to Dan about this grant, and he disputes this inclusion. In particular, he notes that AI Frontiers has only been out for a couple weeks, and that his previous grant to the WMDP benchmark would make a better example, as that grant has similar characteristics but has had more time to stand on its merits.]

General support for SaferAI - growing too fast?

This $100,000 regrant was made by Adam Gleave to SaferAI, an organization that conducts research on AI risk management. Similar to Epoch AI and CAIS, I have confidence that SaferAI is doing good work , such as their ‘risk management maturity’ ratings for leading AI labs, featured in TIME Magazine.

Adam Gleave continues to support the grant, and is excited about further support to SaferAI:

1. They’ve done good work, and are one of really very very few policy-focused orgs in the EU that have gained any traction there. They’re the only one I can think of in France, which is such a key actor.

2. They can’t take [OpenPhil] money. This really limits the number of other sources.

However, I am concerned that SaferAI's leadership team is relatively inexperienced for their rate of growth; a concern shared in Adam’s comments and SaferAI’s proposal. They intend to grow and take on further projects in a way that might require a lot of coordination and administrative overhead—tasks the leadership may not be fully equipped to handle. I am not personally familiar with the SaferAI team, so I’m taking this as given based on what these guys have written, but it seems to me that this may limit the usefulness of additional funding to SaferAI at this point in time. In other words, SaferAI seems like an organization that may be bottlenecked along multiple axes (funding, experience, organizational capacity) such that fixing one bottleneck alone is less valuable than making grants towards another effort that is solely bottlenecked by funding. It may be that SaferAI was not a strong candidate for a Manifund regrant at the time (February 2025) but will be in the future as their ability to usefully absorb more funding increases.

On the other hand, one of the ways that this grant may be used is to allow Siméon, the Executive Director, to start drawing a salary, which seems a more straightforwardly good use of money, assuming he doesn’t have perpetual funding from another source. It’s the bringing on of additional (junior) staff that gives me more pause, although I must stress that my judgment here is low-confidence given my limited information on SaferAI’s capacity for expansion.

Beyond the organizational capacity issues, I also note that SaferAI has “multiple institutional and private funders from the wider AI safety space,” which, while certainly not disqualifying for a regrant, is another reason this regrant may be lower value than many others made through this program.

Finally, I commend SaferAI for the 'What are the most likely causes and outcomes if this project fails? (premortem)' section of their proposal, which was especially helpful for thinking about the value of this grant. I imagine it can be quite easy to write a premortem that doesn’t really get to the nub of a project’s potential downsides. I hope to see SaferAI continue to have positive impact and receive support from the AI safety community, but given their other limitations, I am not sure it was the best fit for the regranting program at this stage.

Austin again. I’m seeing some common patterns with these examples:

1. A large regrant ($100k+),

2. Made to established orgs with lots of money already,

3. By a single regrantor,

4. Who is extraordinarily busy

So these are heuristics for regrantors to keep in mind; and for Manifund to think about when designing our future program.

That said, I don’t think that pattern-matching to one of these heuristics is necessarily bad! Because:

1. Sometimes it’s actually right for a regrantor spend all their money on a single bet. This year especially, Manifund is diversifying the program with more regrantors, across more fields, with smaller per-person budgets

2. Sometimes orgs with good track records will have a lot of funding, but can still use the marginal $ better than a new org

3. Most regrants are made by a single regrantor; it’s rare to get contributions from multiple regrantors, or the general public.

   • In fact, I remember a bit of “regrantor game of chicken” happening with orgs like Timaeus and Apollo — everyone thought they were obviously good, but it wasn’t clear who should spend their budget on it. IMO, funders should think like angel investors — getting early “allocation” into good projects is good!

4. Regrantors who are busy are generally busy for good reason: they’re competent & well connected. As the saying goes, “If you want something done, give it to a busy person.”

   • Also, it’s good for Manifund to have famous (aka busy) people associated with the regranting program — it makes fundraising easier for us, and makes the program seem more legit. But we don’t want to overweight this; we definitely want regrantors who will actually make good grants too.

So overall, it’s really hard to say 🤷. Making good grants is a hard problem, one we’re always trying to get better at. Once again, we’re very grateful to every one of our regrantors for finding and funding the opportunities that they feel will most help the world!

For 2025, we’ve raised $2.25m so far, and are excited to announce our first 10 regrantors:

• Neel Nanda — DeepMind

• Lisa Thiergart — SL5 Task Force

• Lauren Mangla — Constellation

• Aidan O'Gara — Longview

• Gavin Leech — Arb

• Marius Hobbhahn — Apollo

• Thomas Larsen — AI Futures Project

• Tamay Besiroglu — Mechanize

• Richard Ngo — Independent

• Joel Becker — METR

We deeply respect what our regrantors have each accomplished in their fields, and are excited to see what they choose to fund!

Why does regranting matter?

Regranting fills a critical role in the AI safety funding ecosystem:

• Fast, low-friction grants: when needed, we can go from “grant recommended” to “money in grantee’s bank account” in <1 week.

• Hits-based giving: regrantors can make grants solo, and are directly responsible for the quality of the grant. This encourages more speculative grants, and avoids problems in review-by-committee.

• Proactive funding, not reactive: regrantors can offer funding, knowing that they have the money to back their promises. Most other grant processes (OpenPhil, LTFF, SFF) are application-based.

• Support beyond money: regrantors form closer relationships with grantees, and can provide feedback, intros, mentorship, and publicity

  • Startups will often accept small angel checks for the prestige, connections & advice; regrantors can provide similar support to grantees

  • In contrast, OpenPhil program officers or LTFF fund managers are responsible for hundreds of grants and thus are often too busy to help

• Efficient markets in grantmaking: regranting provides more influence (dollars as ”votes”) to AI safety experts with good track records

  • Similar to how tech startup exits create angel investors, who “vote” on the next batch of startups by funding them

  • Similar to how prediction markets provide more “votes” to good forecasters, becoming more accurate over time

  • Regranting budgets themselves are similar to retroactive funding for past good work!

• Regranting is a decentralizing force, seeding new independent orgs, instead of concentrating talent within AI labs, or large charities with strong fundraising operations

• Our regrants are transparent, with project descriptions and grant writeups in public

  • This provides a public track record and social proof for grantees, which helps them with future fundraising and hiring. For example, early public comments from regrantors Evan Hubinger and Ryan Kidd likely helped Timaeus raise more from SFF.

  • Transparency also helps the community understand which kinds of work are respected and funded. Manifund regrants provide more detail per grant compared to every other AI safety grantmaker.

• Regranting builds up the Manifund network, helping us meet great grantees

  • This gives us a picture of the landscape of AI safety,

  • We can follow up with intros, referrals, funding down the line

  • Many of the 2025 regrantors started out as Manifund grantees from past regrants (eg Tamay Besiroglu from Epoch, Lisa Thiergart from MIRI, Marius Hobbhahn from Apollo)

What makes a good regrantor?

With regranting, the most important choice Manifund & our donors face is: “who gets these $100k+ budgets?” Here are some of the criteria we look for, when deciding who to invite:

• Good taste in projects and people

  • Regranting is a similar skillset to angel investing. Good regrantors have a track record of being early and right; of finding undervalued opportunities.

  • Regranting is also similar to hiring: both require identifying great talent. Standard advice for investing in startups is to focus on the founders, not the ideas — we think this holds true for regrants, too. “Fund people, not projects”.

• Has capacity to regrant, proactively

  • That is, regrantors should not too busy to actually find, consider, and make grants

  • Willing to research potential grantees, reach out and form relationships, follow up with mentorship and support

  • Regrantors would ideally spend 5h+/month talking to grantees & writing up the grants

    • It helps if a regrantor’s day-to-day work already provides them with good leads and ideas, eg they are more of a manager than a researcher

• Extends coverage of AI safety funding. One goal of regranting is to spot new opportunities — a wide net helps with this. Some kinds of coverage we think about:

  • Geographic coverage: Bay Area vs DC vs London vs China

  • Subject matter coverage: evals vs mech interp vs AI policy vs fieldbuilding

  • Different kinds of orgs: labs vs thinktanks vs startups vs charities

  • Different competing orgs: Anthropic vs GDM vs OpenAI

• Doesn’t already have easy access to funding

  • People who are already grantmakers, or are well connected to and can recommend grants to philanthropists, might use regranting budgets less counterfactually. We mostly don’t invite existing grantmakers to regrant.

  • There are exceptions though! It depends on specific circumstances. For example, Aidan O’Gara already makes grants at Longview, but we were still excited to have him regrant.

    • A Manifund regranting budget provides Aidan with more flexibility and risk appetite (can make regrants that Longview may be unwilling to publicly endorse), and is more lightweight (regrants are fast and can go out in small amounts, whereas Longview rarely considers grants of <$200k).

  • Grantmaking may be a skill that improves with practice, so budgets may go farther in the hands of experienced grantmakers

We’re still looking for more regrantors; consider applying here!

For regrantors: what makes a good regrant?

Good regrants are often local, small & fast:

• A local grant is made within a regrantor’s unique network, without prior knowledge from OpenPhil or others — our donors aren’t excited to funge against OpenPhil dollars (neither are we!). At the same time, it’s important not to overthink this criterion. There are plenty of cases where OpenPhil sees an opportunity but can’t fund it due to e.g. reputational risk, and also cases where they may be lacking information or are just wrong about the value of a proposal.

• If the org receiving the grant isn’t small (raised $1M+), it probably already has a public track record and a fundraising team, and should therefore be on the radar for OpenPhil and others. As an example, Evan Hubinger’s grant to MATS itself is less exciting to us than grants to his own MATS mentees, who have less funding and less visibility.

• One strength of regranting is that it can be fast, i.e., time sensitive grants, as Manifund can move dollars to grantees within days of the recommendation. Types of funding that might fit into this category include funding for compute or travel expenses, or bringing on world-class talent, ASAP. On the other hand, yearly planning for orgs is better captured by OpenPhil and SFF, who move more slowly but have a lot more money available.

Some examples of regrants that we particularly liked (full review coming soon!):

• Scoping Developmental Interpretability by Jesse Hoogland - the first funding for Timaeus, accelerating its research by months

• Support for deep coverage of China and AI to ChinaTalk - reporting on DeepSeek, ahead of the curve

• Shallow review of AI safety 2024 to Gavin Leech - quick regrants inducing further funding from OpenPhil & others

Ultimately, regrantors have a lot of discretion for making grants, and we encourage you to use it! We expect many great regrants to come not just from looking at new proposals as they appear on the Manifund website, but from proactively finding them. This could look like asking your friends or colleagues for leads, launching a request for proposals, or a prize contest.

See also “Some fun lessons I learned as a junior regrantor” from Joel Becker.

For donors: why fund regrantors?

If you’re looking for a turnkey way to seed great AI safety projects, consider funding our regranting program! We think it’s a good fit for donors who are:

• Earning to give, and donating $50k+ a year

• Concerned about AI safety, but less familiar with the landscape, and not interested in spending a lot of your time getting up to speed

• Excited to back new, undiscovered opportunities

For such donors, regranting provides a “part-time program officer, as a service”. It’s often hard to find great AI safety program officers because their opportunity costs are high; those with the appropriate context could just do direct work. But regranting doesn’t take an expert away from their day job. Instead, they can keep an eye out for good opportunities, while continuing to work in the field.

So rather than spinning up your own foundation, with the overhead that imposes, you can tap into the expertise of someone you trust — and have Manifund handle the logistics of sending out grants. We do ask large donors to contribute an extra 5% towards Manifund’s operational budget; in exchange, Manifund works closely with each donor to understand your giving objectives.

We’re quite flexible with how to distribute budgets:

• You can fund the regrantor program as a whole, allowing Manifund to allocate the budget

• You can increase the budgets for specific regrantors you trust

• You can nominate your own regrantors to join the program

If you’re interested in donating via regranting, please reach out to austin@manifund.org!

Thanks to Jesse Richardson and Leila Clark for input~

I spoke with leaders at Constellation, Lighthaven, FAR Labs, OpenPhil; nobody had a good answer. Everyone said “yeah, an SF hub makes sense, I really hope somebody else does it”. Eventually, I decided to be that somebody else.

Now we’re raising money for our new coworking & events space: Mox. We launched our beta in Feb, onboarding 40+ members, and are excited to grow from here. If Mox excites you too, we’d love your support; donate at https://manifund.org/projects/mox-a-coworking--events-space-in-sf

Project summary

Mox is a 2-floor, 20k sq ft venue, established to bring together EA & AI safety folks with the SF tech scene and labs. Since launching 6 weeks ago, we’ve onboarded 40+ coworking members and hosted 20 events: hackathons and bootcamps, dinners and retreats.

We’re now raising funding to expand Mox into a premier hub. We’re inspired by what Constellation, Lighthaven, and FAR Labs have achieved in Berkeley, and intend to build upon their example, in San Francisco: the city that is ground zero for transformative work.

What are this project's goals? How will you achieve them?

The main elements of Mox:

• Coworking & offices: We host daytime members, who use Mox as their primary workplace. Currently our members are small teams and individuals, with a mix of EA orgs, AI safety researchers, and startup founders. We’re also speaking with “anchor” orgs like Epoch AI to situate their offices here.

• Community space: We’re positioned as a “weekend office”, for folks at eg Anthropic, OpenAI, and METR to work and mingle. We encourage member-run gatherings like blog club, paper reading groups, lightning talks and yoga.

• Public events: As a large, central venue with easy access to both SF and East Bay, Mox is ideal for hackathons, speaker talks, happy hours, unconferences and the occasional party. We organize our own events, and also rent our space to aligned organizers.

• Project incubation: as a medium-term goal, we’d like to host external fellowships or incubators (eg for MATS, FLF, or Apart), or run our own in-house accelerator.

Our north star is approximately: “bring together people who are insanely great.” In pursuit of this, we’ll move fast, stay flexible, try out many approaches, and double down on whatever shows promise.

Mox isn’t a WeWork; an explicit non-goal is to profit by selling coworking space. While we do charge for memberships and events, we do so at subsidized rates, to ground the value we provide. If Mox itself ends up becoming fiscally profitable, that will likely be through other models, eg equity from incubating amazing projects, YCombinator-style.

How will this funding be used?

All figures are very rough. We’d guess that philanthropic funding & investment will cover between 20% to 50% of Mox expenses, and to make up the rest via memberships, events, and other program revenue.

• Minimal budget (for 3rd & 4th floors): $1.6m/year

  • $200k in setup costs (furnishing & labor)

    • We inherited ~$100-200k worth of furnishing and labor from Solaris, the previous tenants; helping us to hit the ground running

  • $120k/mo * 12 mo of ongoing costs

    • ~$40k in fulltime benefits for team of 3

    • ~$80k in rent, utilities, services, materials, snacks1

  • Aim to support ~150 members

  • Organize ~1 tentpole event a month

• Ambitious budget (for all 4 floors): $3.6m/year

  • $800k in setup costs

    • +$600k to build out the 2nd floor to support housing for ~100

  • $240k/mo * 12 mo ongoing — doubling the ongoing values above

  • Support ~350 members, with a fulltime team of 6

  • Organize 3-4 tentpole events a month

Who is on your team?

Our current core team is:

• Austin Chen: CEO of Manifund

• Rachel Shu: Space Manager for Mox

• Mattie Reyes: Operations for Mox

• Ara Hao: Interior Designer for Mox

Manifund also employs ~3 other FTE for other projects, including Manifest.

What's your track record on similar projects?

Since launching 6 weeks ago, Mox has:

• Onboarded ~40 members, contributing $8k per month. Our members include folks from Anthropic, METR, FLF, and numerous startup founders and AI safety researchers.

• Hosted ~3 events a week, with ~$10k in revenue to date. Notable visitors include Aviv Ovadya (AIDF), Ben Goldhaber (FLF), David Rein (METR), Deger Turan (Metaculus), Dylan Patel (Semianalysis), Gideon Lichfield (Wired), Jan Leike (Anthropic), Joel Becker (METR), Jonas Vollmer (Macroscopic), Josh Morrison (1DaySooner), Jueyan Zhang (AISTOF), Leo Gao (OpenAI), Marci Harris (PopVox), Oliver Habryka (Lightcone), Owain Evans (Truthful AI), Paul Christiano (AISI), Tamay Besiroglu (Epoch), Tao Lin (METR), William Saunders (ex-OpenAI)2.

• Organized the AI for Epistemics hackathon, along with Elicit. Andreas Stuhlmüller (CEO of Elicit) remarks:

Mox was an amazing space for the AI for Epistemics hackathon. I loved the thoughtful layout with central communal gathering areas and quiet work areas around the edges, easily enough space for the 40+ participants. Austin was extremely helpful as well - the event was super smooth. I didn't have to think about anything, couldn't be happier. I'd definitely co-host again and am excited to do other projects together

Outside of Mox, Manifund has:

• moved >$5m to projects in AI safety, biosecurity, animal welfare, and other EA and charitable causes

• run AI safety regranting in 2023 & 2024, with regrantors like Neel Nanda, Leopold Aschenbrenner, Dan Hendrycks, Adam Gleave, Ryan Kidd, and Evan Hubinger; and have raised funding for a larger 2025 version

• partnered with Scott Alexander on ACX Grants in 2024, distributing $1.5m to early science & developmental projects

• organized 3 premier conferences of 200-500 people: Manifest 2023 & 2024, on prediction markets & forecasting; The Curve 2024, on bridging the AI safety and SF tech scenes

Previously, Austin started Manifold, a prediction market platform which raised >$4m in EA & venture funding; hosts ~160k markets and ~7m bets; attracts 5000 MAU; and engaged the NYT, Nate Silver, Paul Graham, Sam Altman, Elon Musk, and countless blogs.

What are the most likely causes and outcomes if this project fails?

While thinking about whether to do Mox, I made this prediction market, including reasons it might be a bad idea. Some that are still top of mind:

1. Coworking spaces are bad businesses, as far as I can tell

   • Nowhere near the margins of software

   • The upsides mostly flow to the tenants I think?

     • Maybe the answer is "charge more" but I'm somewhat allergic to that

2. Many other event/coworking spaces have failed or are failing

   • E.g. Lightcone Offices, Atlantis, Solaris AI, Wytham Abbey if you squint

   • It's not obvious to me that SF Commons or Constellation are currently doing well (at least well enough to make me go "yeah there's no point in me starting my own")

3. "Coworking" might be actively harmful (bad for focus, lead to groupthink).

   • Famously, Paul Graham refused to offer coworking to YC startups

   • I do think the Constellation setup of "provide lots of private offices" might be good

4. Physical spaces mostly serve human users; maybe there's more upside in serving AI users

   • Though, as a human, I like humans, and probably will for a long time

5. Maybe most interesting work in SF happens inside of labs, so there's less need for this kind of space

Though: in just 6 weeks, Mox has already proven its value in counterfactual events hosted, connections made, and projects incubated. “Failure” now would be a matter of degree, aka ending up less than maximally amazing as a space and community.

How much money have you raised in the last 12 months, and from where?

Mox has not received any external funding at this time. Manifund started Mox with $300k for the initial deposit, rent, and operational costs; we have approximately 2 months of runway without further funding. We’ve applied to, and are still waiting to hear back from EA Infrastructure Fund (app), and OpenPhil (app).

Even without funding, we are committed to operating Mox for remaining duration of the lease (~12 months), at the least. We’ll be more interested in finding clients that can pay well for coworking or events, rather than clients aligned with our mission. We’ll also need to focus more on the fundraising side of Manifund, to generate operational revenue. In the worst case, Austin may make substantial personal donations towards Manifund & Mox.

Impact certificates in Mox

It would be wise to view any donation to Mox in the spirit of an investment; specifically, as a purchase of an impact certificate for Mox, at a premoney valuation of $8m. Your donations now entitle you to a share of the credit for whatever Mox accomplishes. And, extremely speculatively, you may be able to sell these impact certs for more donation ability in the future.

There isn’t much precedent for impact certs of this size, but here are some comparisons to ground this $8m valuation:

• Constellation received $20m in funding from OpenPhil in 2024. In a typical investment round, investors might buy 10-20% of the total company; this heuristic would value Constellation at $100m-$200m.

• Lighthaven made $1.8m in revenue in 2024; Claude and ChatGPT ballpark Lighthaven’s for-profit valuation at $4-8m based on revenue multiples; this metric excludes Lighthaven’s significant impact, which I would put a further 3-4x multiple on for $20-$30m.

• A typical YC-backed startup might raise a $2m seed at a $15m-$25m valuation after demo day, 3 months after starting.

• Manifold raised at a $15m postmoney valuation in our seed round in Mar 2022, 3 months after launching our website.

To be clear, impact certificates are a very new concept, without any of the ecosystem surrounding for-profit startups. It’s very possible that your donation to Mox simply ends up as a one-time donation.

But we’re hoping that through our efforts and with your support, Mox will grow to be much more valuable than it is today. And if Manifund can establish a robust ecosystem of impact certificate purchases, buybacks, and retroactive funding (big if!), then donors today may be rewarded with a large return in charitable credit, to donate towards future projects that matter to you.

Donate to Mox

PS: apply for Mox!

Want to join Mox? We're located at 1680 Mission St; our website is here. Fill out https://moxsf.com/apply if:

• you’re looking for a space to work from. We support memberships for hot desks, fixed desks, private offices and occasional visitors.

• you’d like to run an event. We’ve hosted hackathons, parties, dinners, reading clubs and weird events of all kinds.

• you’re interested in working for Mox or Manifund! We’re looking for great people for a variety of roles, from events coordinator to back office & finances to “Director of Mox”

(or if you know someone who’d be a good fit, send them the link!)

Resources

• See the project showcase: https://moxsf.com/ai4e-hacks

• Watch the recordings: project demos, opening speeches

• See the outline of project ideas: link

  • Thanks to Owen Cotton-Barratt, Raymond Douglas, and Ben Goldhaber for preparing this!

• Lukas Finnveden on “What's important in ‘AI for epistemics’?”: link

• Automation of Wisdom and Philosophy essay contest: link

Why this hackathon?

From the opening speeches; lightly edited.

Andreas Stuhlmüller: Why I'm excited about AI for Epistemics

In short - AI for Epistemics is important and tractable.

Why is it important? If you think about the next few years, things could get pretty chaotic. As everyone rushes to integrate AI systems into every part of the economy, the world could change more rapidly than it does today. There's significant risk that people and organizations will make mistakes for relatively uninteresting reasons—simply because they didn't have enough time to think things through.

If we can make it easier for people to think clearly and carefully, that's really important. People will use AI tools to help them make decisions either way; eventually unassisted decision-making just won’t be competitive anymore. This is a lever: the more these tools actually help people make wise decisions, or help them figure out whether they're right or wrong about something, the better off we'll be.

AI for Epistemics is also tractable now in a way it wasn't before. We're just reaching the point where models are good enough and cheap enough to apply at scale. You can now realistically say, "Let's analyze all news articles," or "Let's review all scientific papers," or thoroughly check every sentence of a document, at a level of detail that wasn't feasible before.

Given good ideas for epistemic tools, the implementation cost has dropped dramatically. Building significant products in hackathons has become much easier. You can basically copy and paste your project description into Cursor, type "please continue" five times, and you'll have a working demo.

The key challenge we'll need to think about today is: how can we tell if we're actually making things better? What evidence can we see that would lead us to believe a tool genuinely improves people's thinking, rather than just being a fun UI with knobs to play with?

I'm really excited about this hackathon. This is the event I've been most excited about for quite a while. I'm very grateful to Austin for creating this space for us.

Austin Chen: Why a hackathon?

Andreas first talked to me a couple months ago, saying we want to do more for the AI for Epistemics field. We were thinking about some ideas: “oh, maybe we should do a grants program, or a fellowship program, or something like that”.

But I have a special place in my heart for hackathons specifically. So I really sold him hard: we're gonna do a hackathon. We can do all that other stuff too later, but: first things first. (Andreas, wryly: “I was very hard to sell.”)

I like hackathons for a lot of reasons:

• Hackathons are a sandbox. They're a place where you can play with an idea a little bit. You don't have to worry about whether this thing will be great down the line, or even live past the end of the day. So it gives you a chance to be a bit more creative, try riskier things.

• It's a blank canvas. You don't have to worry about what your current users will think, or “will this make money?”. You can just… do stuff.

• Hackathons are a forcing function. There's the demos in eight hours. We're all gonna get up there and present and talk about what we did. So you have to sit there and actually build your idea. You can't just keep spinning your wheels, thinking forever.

• And it's a chance to meet people. It's a filtering function to find people who care a lot about this particular niche. Right now, AI for Epistemics is a tiny field. All the people who care about it are maybe in this room right now (plus a few others who are remote). But hopefully, it will grow down the line. And this is your chance to meet each other, talk to each other, share your ideas, build stuff out.

Those are some of the reasons I'm excited about hackathons. I'm glad that Andreas and the Elicit team are happy to host this with us today.

Meet the projects

We asked the participants to share more about their project after the hackathon ended. Comments are mostly Austin’s.

Question Generator, by Gustavo Lacerda

Demo: Starts 6:18

Description: This is a browser extension that generates forecasting questions related to the news page you are visiting.

Comments: Good exploration of a promising form factor (chrome extension to make personal flow easier). I like that it ends with “create a Manifold question”, as a concrete thing to go next. I’m not sure if the questions were actually any good? But with LLMs, maybe it’s always a brainstorming aid, or LLM generate and humans filter (as with imagegen).

Symphronesis, by Campbell Hutcheson (winner)

Demo: Starts 14:34

Description: Automated comment merging for LessWrong; finds disputes between the comments and the text and then highlights the text with the disputes; color coded and you can mouse over and then jump to the comment.

Why did you build this?: I’m interested in how LLMs will enable highly personalized UI/UX. One of my main contentions is that software became mass produced because the cost of development is very high and so it was prohibitive to create artisanal software solutions for individuals - but that LLMs - because they make software cheaper - give us the opportunity to return to a more artisanal software experience - where our interface to software is created dynamically. Moreover, as the cost to benefit ratio of software was even worse in design than elsewhere - good design has been something essentially limited to software companies that aggressively focus on it as part of their core value prop (e.g. Apple, Notion, Linear). But, this can now change, and we can have better, more personalized, richer experiences.

What are you most proud of for this project?: It worked. It has nice bells and whistles. It enables me to have more control over a document as an organic thing.

Source: https://github.com/chutcheson/Symphronesis

Comments: Interesting & pretty UI, reasonable concept. Lots of audience questions about how it was implemented. Lukas: “Unfortunate to do this for Lesswrong which is the website with the most support for this already”

Manifund Eval, by Ben Rachbach & William Saunders

Demo: Starts 1:38

Description: Screen all Manifund projects to identify ones to look into more to consider funding. Also identifies the grant’s story for having an impact on transformative AI going well, so you can review that to save time in your evaluation. Makes it feasible to quickly sift through the large number of Manifund projects to find promising ones to consider.

Demo link: https://manifundeval-zfxpigvo8jemehaybdwwsw.streamlit.app/

Comments: Of course, soft spot in my heart for using the Manifund API. Pretty important and impactful project (Andreas: “I actually need this.”) Not sure if the final scores outputted, or reasoning were that good though; didn’t seem that great by my lights. Might be biased — I’d tried something similar (for giving feedback to potential new projects) and it was only okay. But def worth more experimentation. I think I might want to issue a bounty to solve this problem for Manifund.

Detecting Fraudulent Research, by Panda Smith & Charlie George (winner)

Demo: Starts 20:59

Description: There’s a lot of research. A lot of it seems bad. How much? We use language models to try and detect retraction-worthy errors in published literature. We purely reason from first-principles, without using meta-textual information.

Why did you build this? Panda: At Elicit, I spend a lot of time thinking about people’s info sources. I’ve also read metascience blogs for a long time. I assumed there would be some fraud/bad papers that modern reasoning models could catch pretty easily. (I didn’t think there’d be so much!)

What are you most proud of for this project? Panda: Very happy with doing a mix of “research” where we ran the numbers on how effective our technique was, but also prototyping and making something people can get their hands on

Source: https://github.com/CG80499/paper-retraction-detection

Demo link: https://papercop.vercel.app/

Comments: Had the most “wow this is fun to play with” factor, also “I can see this going viral”. I particularly liked that they had some semblance of evals (taking 100 papers and running it through), rather than just one or two demo cases; with LLM stuff it’s easy to focus on one or two happy cases, and I’m glad they didn’t.

Artificial Collective Intelligence, by Evan Hadfield

Demo: Starts 29:13

Description: ACI is a consensus-finding tool in the style of Polis / Community Notes, simulating a diverse range of perspectives. LLMs play the role of extra participants, submitting suggestions and voting on entries.

Demo link: https://aci-demos.vercel.app/

Comment: Most ambitious IMO — an entire platform of sims. With more time to develop this, I could see this as my favorite entry. Unfortunately, lost points for not having a live working demo :(

Thought Logger and Cyborg Extension, by Raymond Arnold

Demo: Starts 35:26

Description: I have a pair of products: – a keylogger, which tracks all your keystrokes (except from apps you put on a blocklist), and exposes it on a local server – and a “prompt library” chrome extension, which lets me store fairly complicated prompts and quickly run them, while pulling a website or the keylogger logs into context.

For demo day, I worked on a “useful personal predictions” prompt for the prompt library, which takes in my keylogs from the past 2 days, extrapolates what projects I seem to be working on, and generates prediction-statements about my project, that help guide my strategy. (i.e. “I’ll get at least 3 positive reports from users about my product helping them, spontaneously, in the next 2 months.”). When I see ones I like, I enter them into Fatebook.

Why did you build this? The general idea of the keylogger + prompt library is to set me up to leverage AI in all kinds of customized ways over the next couple years. I want to be an AI poweruser, and to have an easy affordance to invent new workflows that leverage it in a repeatable way.

I think “decision-relevant predictions” is a good tool to help you get calibrated on whether your current plans are on track to succeed. But operationalizing them is kind of annoying.

Source: The tools aren’t public yet, but message me at raemon777@gmail.com if you’d like to try them out.

Comments: Interesting set of work, I like the keylogger idea, picture of “record everything and have LLMs sort it out”. In practice had the flavor of people optimizing their personal setup a bit too much, and being hard to scale out (see also: complex Obsidian thought mapping, or spaced repetition)

Double-cruxes in the New York Times’ “The Conversation”, by Tilman Bayer

Demo: Starts 41:17

Description: “The Conversation” is a weekly political debate format in New York Times “Opinion” section between conservative(ish) journalist Bret Stephens and liberal(ish) journalist Gail Collins, ongoing since 2014. I used Gemini 2.0 Flash Thinking to identify double-cruxes in each debate, with the aim to track both participants' shifts over time.

Why did you build this?: Double-Cruxes are a somewhat intricate epistemic concept that so far doesn't seem to have made it very far beyond the LessWrong sphere. I wanted to explore whether one could use current LLMs to apply it at scale to a (non-cherrypicked) corpus of political debates aimed at a general audience.

What are you most proud of for this project?: After some experimentation, found a prompt+model combination that holds up quite well in vibe tests so far.

Source: Presentation slides from the hackathon

Comments: Unclear to me if double-cruxes is important epistemic tech, esp whether it has broad reach. Didn’t really have a working demo, sadly.

Trying to make GPT 4.5 Non-sycophantic (via a better system prompt), by Oliver Habryka

Demo: Starts 47:01

Description: I tried to make a system prompt for GPT 4.5 that actually pushes back on things I say and I can argue with in productive ways. It isn’t perfect, but honestly a bunch better than other experiences I’ve had arguing with LLMs.

Prompt: link

Comments: Many points for directly trying something out of the Owen’s spec. And for having the bravery to do a “non-technical” hack — as LLMs do more of the technical work, what’s left for humans is prompting well, imo. And for something that is immediately usable!

Squaretable, by David Nachman (winner)

Demo: Starts 55:27

Description: To assist a user in decision-making, the app uses LLMs to help the user come up with weighted factors, possible options, and factor values for each option. The UI consists of an always displayed table of the factors, options, weights, and values. The final score for each option is computed symbolically as a weighted sum based on the values and weights.

Comments: Great UI, information is pretty well laid out and yet compact, love the colors. Unfortunate that David didn’t seem to think that the LLM’s results were that good. Andreas: “maybe better UX if you add columns incrementally, easier to spot check”. Makes sense, kind of like git diffs or what Cursor does in chat mode.

What went well

• Hacks were pretty cool! Especially given that they all represented ~8 hours of work

  • Many are minimal versions of products I really want to exist, and play with more

  • Almost all of them felt like a worthwhile showcase, exploration of something interesting, and relevant to this field

• Lots of great people came for this! Very hard to think of more central folks for AI for Epistemics:

  

• From left to right: Rafe Kennedy, Oli Habryka, Evan Hadfield, Kirill Chesnov, Owain Evans, Charlie George, Panda Smith, Gustavo Lacerda, Andreas Stuhlmüller, Austin Chen, David Nachbach (virtual), Lukas Finnveden, Tamera Lanham, Noa Nabeshima, Campbell Hutcheson, Keri Warr, Xyra Sinclair, Tilman Bayer, Raymon Arnold, Chris Lakin, Ozzie Gooen

  Not pictured participants and viewers: William Saunders, Ben Goldhaber, Deger Turan, Vishal Maini, Ross Rheingans-Yoo, Ethan Alley, Dan Selsam, Stephen Grugett, David Chee, Saul Munn, Gavriel Kleinwaks and many others…

  • Some of the goal of the hackathon, as with any event, is just to bring people together and have them stay in contact

• Good conversations at the beginning while participants were ideating, and throughout. Both on AI for Epistemics and other topics.

• Overall event felt smooth and cohesive, especially for being pulled together on not that much organizer time

  • Pretty happy with the continuing artifacts that we produced out of this hackathon (the showcase page, the video recordings, this writeup)

  • Somewhat more effortful to do all this compared to typical hackathons Austin has run, but hopefully worthwhile when trying to incubate a new field

• Mox seemed to be a good venue for this event! This was just our third week of operating, but I think our venue supported the hackathon well.

  • One participant remarked:

Something I like about your office is that it seems to naturally create the Cal Newport Deep Work architecture, where the further in you go the more deepworky it is

What could have gone better

• Fewer submitted hacks than we’d hoped for

  • Had ~40 people around but only ~10 submissions. Ideally more like 15-20?

  • Maybe we should have promoted this event harder, or cast a wider net?

    • There’s a tradeoff on average participant quality vs number of submissions.

    • But maybe projects are hit-based, so having the best projects matters more than having a high average quality

  • Maybe try to get higher commitment from folks, if we run this again

• Hoping to have discovered more people from outside our current networks, who are excited for AI for Epistemics

  • 2 of the 3 prizes went to teams from Elicit

    • (Which says something about how great the Elicit team is, in case anyone out there is thinking about finding a new job…)

• Unclear path to deployment for these projects, or continuing impact

  • Admittedly, this is a standard problem with a hackathon form factor, especially when the hackathon isn’t housed within an org/for specific product features

• Not sure we made great use of the ideas doc & categories that Owen/Raymond/Ben compiled?

  • But hopefully, their work will set a stage of “this is what AI for Epistemics is about”

  • Perhaps having such specified categories was too confusing for participants

    • One participant asked, “do I have to do something in these categories?” (A: no, but it’s bad that this wasn’t clear)

• As judges: hard to give great judgements and feedback in short amount of time, by just looking at demos and asking questions

  • The format of “demos in front of an audience” do bias towards presentation ability and flashiness, over usability of a core product

  • Might change up the structure for next time

    • More time for judges and audience to play with hackathon demos?

    • Open up voting to the public, so it’s more democratized?

Final notes

Overall, we’re very happy with how this hackathon turned out. Building a new field from scratch is difficult, high-dimensional problem, and this is just one step along the way; but I think we made meaningful progress, with ideas we brainstormed, hacks we demoed, and people we gathered.

After the end of the hackathon, a few of the judges and participants continued to discuss: “What’s next for AI for Epistemics? How does one build a nascent field? Is ‘AI for Epistemics’ even a good name?” We’ll try to share more on this in the coming days; until then, if AI for Epistemics excites you, leave a comment or reach out to us!

Today, Manifund is mostly a remote endeavor; we interact through website comments, emails, and Discord. So one thing I’ve always wanted to do is to bring y’all together and meet face-to-face. I’ve really enjoyed doing this with the Manifold community at our premier festival, Manifest; but that takes months of work to organize, and Manifund’s not yet at that scale. What would a more minimal, 80/20 festival look like?

And so: minifest! We've booked out Lighthaven for a single day: this coming Saturday, Dec 14. We'll have unconference-style sessions, a smattering of events like charity poker, and ample time to hang out and chat. We're running this on a shoestring budget, so that everyone can come -- eg expect a homecooked dinner instead of catering. It's a cozy, lowkey, experimental event; I hope to see you there!

Get a ticket here: https://minifest.is

— Austin

In other news: we’re in talks to start a 100-person office & coworking space in San Francisco next year — markets say 70% likely. If you’d like to get involved as a funder, tenant, or organizer, let me know (austin@manifund.org)~

• Local: Creators have a good track record in the EA or AI Safety community

• Modest: The amount requested is not large; $5k would be meaningful

• Overlooked: Not already backed by large institutional funders like OpenPhil

If you’re a small donor or earn to give, consider giving to projects like these:

1. Feature-length documentary on SB 1047

By Michael Trazzi — $16k raised of $55k

Michael has already recorded interviews with the main characters of SB 1047: sponsors like Scott Wiener and Dan Hendrycks, proponents like Zvi Mowshowitz and Holly Elmore, and opponents like Dean Ball and Timothy B Lee. Now he needs the funding to turn it into a 1-hour feature documentary. This is a rare chance to sponsor a high-quality video narrative, and share it beyond our existing ecosystem. I’ve personally donated $10k towards this, and expect Michael to be able to effectively use much more.

More info & donate here: https://manifund.org/projects/finishing-the-sb-1047-documentary-in-6-weeks

2. Overview of AI Safety in 2024

By Gavin Leech — $8k of $17.6k raised

Gavin Leech is a forecaster, researcher and founder of Arb; he’s proposing to re-rerun a 2023 survey of AI Safety. The landscape shifts pretty quickly, so I’d love to see what’s changed since last year.

As I was writing this, regrantor Neel Nanda funded it to the minimum $8k ask! Neel adds:

I think collections like this add significant value to newcomers to the field, mostly by being a list of all areas worth maybe thinking about, and key links (rather than eg by providing a lot of takes on which areas are more or less important, unless the author has excellent taste). Gavin has convinced me that the previous post gets enough traffic for it be valuable to be kept up to date.

More info & donate here: https://manifund.org/projects/shallow-review-of-ai-safety-2024

3. Podcast series on Effective Altruism’s values

By Elizabeth Van Nostrand — $1.3k raised of $2.6k

Elizabeth & Timothy’s initial podcast was very well received, drawing extensive, thoughtful comments from a variety of folks. I’d be excited to see them continue this series, especially if they bring in folks involved with steering the EA community (like Sarah Cheng, who has extensively engaged with their points)

More info & donate here: https://manifund.org/projects/elizabeth-and-timothy-podcast-on-values-in-effective-altruism

4. Sentinel, a foresight and emergency response team

By Nuno Sempere — $16k raised of $90k

Nuno has long been one of our community’s most outspoken forecasters; now he’s working with Rai Sur to spin up an emergency response team (think: Army Reserve Corps, but for responding to existential risks). They’re already putting out a useful weekly report on biosecurity, geopolitics and other such topics.

More info & donate here: https://manifund.org/projects/fund-sentinel-for-q4-2024

5. Research on co-occurence of sparse autoencoder latents

By Matthew A. Clarke — $0 raised of $6.4k

TBH, I don’t know much about the merits for or against this line of research; I’m highlighting this grant because it’s overseen by Joseph Bloom, a past Manifund grantee who I and others have been very impressed with. If mechanistic interpretability is your jam, check this one out!

More info & donate here: https://manifund.org/projects/salaries-for-sae-co-occurrence-project

What else have we been up to?

It’s been a quiet couple of months, but here’s what Rachel and I have been busy with:

• Organizing The Curve, a premier conference on transformative AI

• Wrapping up EA Community Choice and paying out grants (writeup to come, someday…)

  • Did you see that Marc Andreessen donated $32k to one of the projects 😮

• Filing our taxes Form 990 for 2023. Last year, we raised ~$3m and disbursed ~$2.6m!

• Waiting on Survival and Flourishing Funds to get back to us on our funding request

• Hacking on a system to spin up web apps from a single LLM prompt

• Dreaming about a new coworking space like Constellation or Lighthaven, but in SF

• And of course, taking some time off to care for our newborn baby, Ada

Cheers,

Austin

PS: want to make a larger grant? Manifund can facilitate donations via donor advised funds, crypto, and bank transfers.

Claim your funds

Recap on how this works

• Phase 1: Project registrations open (Aug 13)

  • 50+ projects signed up to participate in this round, so far

    • Project signups will remain open through phase 2; you can list your project here,

  • 350+ members of the EA community registered their interest

• Phase 2: Community members claim funds & donate (Aug 21 - now!)

  • Fill out this form about your roles in EA to claim your funds.

  • Then, look through the Community Choice projects here; donate to your favorites!

  • You can also leave comments on each project proposal. This is a great way to ask questions to the project organizer, or share thoughts with rest of the EA community.

• Phase 3: Funds matched and sent to projects (Sep 3)

  • Projects and donations will be locked in on September 3rd. Then, all money donated will be matched against a $100k quadratic funding pool.

  • Note: we’ve pushed back the dates a bit to account for Labor Day.

More questions? Check out the announcement post FAQ, or hop in our Discord!

Sign up here to get notified when the projects are live, or read on to learn more!

Timeline

• Phase 1: Project registrations open (Aug 13)

  • Project organizers can now create a page for their projects here, to raise funding as part of this EA Community Choice round.

  • Community members can sign up for updates here, or recommend projects to sign up here.

• Phase 2: Community members receive funds (Aug 20)

  • We’ll give everyone $100 to donate; more if you’ve been active in the EA community. Fill out a 2-minute form to claim your $100, plus bonuses for:

    • Donor: $100 for taking the GWWC 🔸10% Pledge

    • Organizer: $100 for organizing any EA group

    • Scholar: $100 for having 100 or more karma on the EA Forum

    • Volunteer: $100 for volunteering at an EAG(x), Future Forum, or Manifest

    • Worker: $100 for working fulltime at an EA org, or full-time on an EA grant

    • Senior: $100 for having done any of the above prior to 2022

    • Insider: $100 if you had a Manifund account before August 2024

  • You can then donate your money to any project in the Community Choice round!

  • You can also leave comments about any specific project. This is a great way to share your experiences with the project organizer, or the rest of the EA community.

  • Funds in Phase 2 will be capped at $100k, first-come-first-served.

• Phase 3: Funds matched and sent to projects (Sep 1)

  • Projects and donations will be locked in at the end of August. Then, all money donated will be matched against a $100k quadratic funding pool.

  • Unlike a standard 1:1 match, quadratic funding rewards a project with lots of small donors more than a project with few big donors. The broader the support, the bigger the match!

  • Specifically, the match is proportional the square of the sum of square roots of individual donations. A toy example:

    

  • Learn more about principles behind quadratic funding by reading Vitalik Buterin’s explainer, watching this video, or playing with this simulator.

What is an EA community project?

We don’t have a strict definition, but roughly any project which helps you: learn about EA, connect with the EA community, or grow your impact in the world. We’re casting a wide net; projects do not have to explicitly identify as EA to qualify (though, we also endorse being proud of being EA). If you’re not sure if you count, just apply!

Examples of projects we’d love to fund:

• Community groups

  • Regional groups like EA Philippines

  • Cause-specific groups like Hive

  • University groups like EA Tufts

• Physical spaces

  • Coworking spaces like Lighthaven, Epistea, and AI Safety Serbia Hub

  • Housing like CEEALAR and Berkeley REACH

• Events

  • Conferences like Manifest, EAGx, LessOnline, and AI, Animals, and Digital Minds

  • Extended gatherings like Manifest Summer Camp or Prague Fall Season

  • Recurring meetups like local groups or online EA coworking

  • Tournaments like Metaculus Tournaments or The Estimation Game

  • Essay competitions like EA Criticism & Red Teaming Contest

• Software

  • Tools like Squiggle, Carlo, Fatebook, or Guesstimate

  • Visualizations like AI Digest

  • Datasets like Donations List Website

• Educational programs

  • Incubators like AI Safety Camp and Apart Hackathons

  • Course materials like AI Safety Fundamentals

• Information resources

  • Websites like AISafety.com

  • Youtube channels like Rational Animations, Rob Miles, and A Happier World

  • Podcasts like The 80k Podcast, The Dwarkesh Podcast, and The Inside View

FAQ

• What is Manifund?

  • Manifund is a platform for funding impactful projects. We’ve raised over $5m for hundreds of projects across causes like AI safety, biosecurity, animal welfare, EA infrastructure, and scientific research. Beyond crowdfunding, we also run programs such as AI safety regrants, impact markets, and ACX Grants.

• Why are you doing this?

  • We want to give the EA community a voice in what projects get funded within our own community. Today, most funding decisions are centralized in the hands of a few grantmakers, such as OpenPhil, EA Funds, and SFF. We greatly appreciate their work, but at the same time, suspect that local knowledge gets lost in this process. With EA Community Choice, we’re asking everyone to weigh from their own experiences, on what projects have helped with their personal journey towards doing good.

• Why these criteria for donation bonuses?

  • We chose these to highlight the different ways that someone can contribute to the EA movement. EA Community Choice aims to be more democratic than technocratic; we want to ensure a wide range of activities get recognized, and that a broad swathe of the EA community feels bought in to these donation decisions.

• Why quadratic funding?

  • Quadratic funding is theoretically optimal to distribute matching funds towards a selection of public goods (and we’re suckers for elegant theory). The crypto community has pioneered this with some success, eg with Gitcoin Grants and Optimism’s Retroactive Public Goods Funding rounds. Closer to home, the LessWrong Annual Review is an example of a quadratic voting system in practice, which produces pretty good results.

• Where did this $200k come from?

  • An anonymous individual in the EA community. Manifund would love to thank them publicly, but alas, the donor wishes not to be named for now. (It’s not FTX.)

• Can I direct my funds to a project I work on or am involved with?

  • Yes! We ask that you mention this as a comment on the project, but otherwise it’s fine to donate to projects you are involved with.

• How should I direct my funds? Eg should I fund projects based on their past work, or how they would use marginal funding?

  • We suggest based on how much value you have gotten out of it (aka retroactive instead of prospective), but it’s your charity budget; feel free to spend it as you wish.

  • We’d appreciate if you leave a comment about what made you decide to give to a particular project, though this is optional.

• Can I update my donations before Phase 3?

  • Yes! If later donations or comments change your mind about where you want to give, you can change your allocation

• If I think a project has negative externalities, can I make a “negative vote” aka pay to redirect money away from it?

  • TBD. This may be theoretically optimal and has been used by other projects, but leaning no because of bad vibes/potential for drama and additional complexity it introduces.

• Can I contribute my own money towards a community project?

  • Yes! You can make a personal donation to any project in this community choice round; these donations will also be eligible for the quadratic funding match (as well as a 501c3 tax deduction, if you’re based in the US).

• How about contributing towards the matching fund?

  • Yes! We’re happy to accept donations to increase the size of the matching pool for this round. Reach out to austin@manifund.org and I’ll be happy to chat!

  • Or, if you’re excited by this structure but want to try a different focus (eg a funding round for “technical AI safety projects” or “animal welfare projects”), let us know!

Get involved!

As the name “EA Community Choice” implies, we’d love for all kinds of folks in the community to participate. You can:

• Register your project to receive funding in this round

• Recommend a project to join the round

• Sign up to get notified when funds get sent out

Excited to support the projects that y’all choose!

Thanks to Rachel, Saul, Anton, Neel, Constance, Fin and others for feedback!

Neither of us are experts on OpenAI or have insider information; we’re just hashing out our different worldviews in a casual setting. If you listen or follow along, let us know what you think!

Topics covered:

• 00:33 OpenAI's contradictions

• 02:44 NDAs and tech vs rationalist culture

• 13:51 Is there enough criticism of OpenAI? Too much?

• 17:15 Superalignment's 20% compute promise

• 21:50 The EA departures from OpenAI

• 26:10 ScarJo

• 34:41 Should AGI be pursued by a for-profit company?

• 46:10 A hypothetical AGI "Manhattan Project"

• 51:45 Government vs company services

• 54:08 Is OpenAI democratic?

Note that this podcast was recorded May 22, before Kelsey Piper’s expose on NDAs; subsequent NDA walkbacks by OpenAI; and reports that Sky was indeed commissioned before the ScarJo reachout

Link with video & transcripts: https://share.descript.com/view/1N6kvfXFLr9

1: OpenAI

Austin: [00:00:00] So we're here today to talk about OpenAI a subject that Linch and I have talked about in some context before.

I think I, coming from a more tech startup background, have views that maybe are a little bit more alien to the OpenAI. What I think of as a standard EA like AI safety, maybe Doomer ish views, which maybe Lynch represents a little bit more of.

So I'm excited to have a chance to hash this out with you today, Lynch.

Linch: Yeah, happy to do I think if anything, I'm probably more negative about like OpenAI than the average like EA person, like for example, the OpenPhil cluster viewpoints. Interesting.

[00:00:33] OpenAsteroidImpact & OpenAI's contradictions

Austin: And when you created Open Asteroid Impact, right?

This is like a satirical thing, trying to get people to pay more attention to OpenAI and like ways that it could go. Badly. And I was working with you on that. I think in my mind, I didn't even think of this as " Oh, this is because we want to draw attention to like things, OpenAI is doing badly."

Rather like it just seems funny to me.

Linch: Yeah. Yeah. I think badly is like one conception of it. I think contradiction is another conception. Like I was trying to highlight the ways in which I thought the public messaging, or at least Yeah. Public messaging as understood by people I know conflicted with a lot of the actions.

Austin: Yeah, maybe that's a great place to start actually.

With the overall sense of messaging being contradictory, can you say a bit more about that?

Linch: Yeah. These things are like. Oh not necessarily super obvious, but I think there's definitely a very weird needle or like a very odd thing.

So one example is that they talk about, so we talk about this in Open Asteroid Impact, but they say a lot of the value they brought is like bringing attention to AI or AI risk by making like really splashy like AI demos and like basically building the thing that's potentially on the ramp to being dangerous.

I think that just like an odd approach. I think it's like a pretty unusual approach. I couldn't think of many like historical precedents for doing something like this.

Austin: That's not necessarily contradictory though or when you think, when you say contradictory, I think that usually implies like they're saying one thing and doing another or saying two different things at the same time.

Linch: Sure. I guess like talking about openness is like a pretty odd thing a lot of the time from like the software perspective. To have obviously not like really open source for that don't open source most of their work, but also like open means many things right from like open societies perspective, you might think that people are free to talk about what they want to, or they're able to voice opinions.

I know for many people in the, during my issue, I safety crowd are worried about technological capabilities leaks for other Specific technical things that are like dangerous to say, and that's why you might need like NDAs for other ways to cover that. But it seems, as we now know, but I think it was pretty obvious before they don't say a lot of things, period.

There's a lot of stuff that they could talk about. They like employees basically can't. And that's pretty odd. I think.

[00:02:44] NDAs and tech vs rationalist culture

Austin: Yeah. So I guess jumping to one of the topics of the day, the NDAs I, my high level of thought is that I feel like people in the area are making a mountain out of a molehill, so to speak.

It seems unlikely in my head to be the case that like the NDAs are significantly restricting speech. And I have more of you that are like, Oh, these are like relatively standard. Maybe there's a few ways in which not that standard. I don't think, I especially don't think there was like a lot of like ill intent, like trying to suppress disparagement or disclosure of like things that beyond what is like.

Roughly typical in Silicon Valley and yeah I guess I have some kind of model of if there were like important things that the people who have recently left OpenAI would have wanted to say they would be saying those things and it's not they feel, oh, I have signed this contract and as a result, can't say these things I don't think the signing of the contract makes a huge difference to their yeah.

Internal sense of whether it's worth blowing a whistle on like bad behavior at OpenAI. So I just think there's like too much of a conspiratorial view on like how much these NDAs actually affect the speech actions of the people who have left, or people who are still at OpenAI.

Linch: Yeah. So for context, for people who don't understand, know this, like they have, so there's like NDAs, which are non disclosure agreements.

Those are fairly common in industry and there's non disparagement agreements, which are like Somewhat common, but not super common where you can't say that it's not, you can't say like important information or like trade secrets or anything, but you just can't say bad things about a company. And my understanding is many, if not most, for all OpenAIX employees have signed like a non disparagement agreement, which is much more significant to us.

And those are lifelong. So if you sign it, like legally, you're not supposed to ever criticize OpenAI. Which is pretty big, and third, my understanding is that these agreements were signed on the fear of losing their equity that they've already owned, which makes it much a higher bar than I think is common for other cases I've heard of.

Austin: Yeah. So thanks for that context. I think one other thing that I wanted to draw attention to was that somewhat recently, I think there was like a kerfuffle, maybe a small discussion on the, on LessWrong perhaps, where I believe Wave who is like a company that does mobile payments in Africa.

[00:05:00] They're known for being like pretty EA affiliated. I think many of their like top employees are familiar with that. Wave mentioned they also have non, I believe disparagement agreements for people who leave. And this caused some, I think like clash of cultures between like I think the wave founder Lincoln Quirk was like, Oh, this is like a pretty normal thing.

And then I think about Oli Habryka of LessWrong, I was like, no, this is changing speech norms and like very much against the like rationalist culture. And I also took the like tech maybe like Lincoln Quirk line here, which is this seems like pretty fine and normal. And I guess to expand on that a little bit I do think the non disparagement agreement is there for some reason mostly that when you have an employee leave, you don't want to get into a lot of bad, like ugly press battles over he said, she said things like that. And I think there's like a degree of reasonableness basically in these kinds of contracts, which is if you say like things that are like generally like factually true and important to say, like something like.

Yeah. I don't know, like this top exec at OpenAI had an affair with one of their interns and this is like a big scandal. I don't think this is the kind of thing that like, then OpenAI would go after you in a non disparagement, using the non disparagement clause, right? So again, I'm modeling these, like society, I think generally to be reasonable and like the people at OpenAI to be like to have a sense of what, which things is like disparate agreement can cover and my modeling, meanwhile, the like rationalist EA crowd of taking these things like too literally if you look at the literal text of the contract, maybe it says oh, you can't say anything bad about this.

But literally, if you try to do this and an OpenAI tried to go after you with the contracting, I think they would lose in the court public opinion and they understand this. And everyone understands this.

Linch: It's just obviously a chilling effect though. If you're like on the fence of whether to say something or not, you're just not going to say it.

And like in aggregate, like it's often the case where you have an elephant and everybody sees one piece of elephant, but they don't see all of it. And, and especially in like. Cases where there's like bad behavior, but there's no smoking gun. It's just like pretty common to make it very hard to know, like without all the pieces of information out there.

Austin: Yeah but I really think the like non disparagement agreement is such a like tiny portion of the like reasons people don't speak up. I think like people don't say things for yeah, mostly things like. Wanting to be nice or not thinking that it is a worth raising or things like that.

And not one

Linch: of the most powerful people in Silicon Valley, may not like you and may go after your employee, your future employers. There's a ton of stuff like that. Like this is like one piece of that angle that we see we don't know all the information we don't see.

Austin: Yeah. Again, that seems Kind of conspiratorial.

I guess I don't know Sam Allman, for example, in person. And I know some people say things like, oh he could be, like, vindictive, or some people are, like, concerned of retribution or things like that. Maybe I'm just much more of an optimist, or I believe in people in general, like Sam Bankman-Fried sorry, Sam Altman and Sam Bankman-Fried Included.

There were some other things said about SBF that he could be, like, vindictive, and people were, like, afraid to say things. And I think all this is Roughly BS roughly it wasn't that this or

I guess I don't want to say, put words in people's mouths, maybe or put intentions in people's heads or whatever possibly they actually believe this, but I just, I don't know, think that actually, there wouldn't have been, like, important retribution or something like that, or

Linch: yeah.

Yeah. A lot of people seem to believe this about Sam Bankman-Fried, like the, the former CEO of FTX US, I think, for example, had a bunch of specific things like he was served by lawyers and stuff like that. Sued by lawyers? Served. Served, I see. I think. Like, where like he, sorry, that's not exactly it.

There was like internal disagreement about FTX US policy. And then instead of resolving it the way that like. You might expect like to executives to hash it out instead of like Benjamin Fried had his lawyer sent like a specific thing to him. And that's just like powerful the course, I think for like at least some powerful people who are like, on the 90th percentile of like ruthlessness or sociopathy within amongst CEOs.

Yeah, I definitely I do want to make it clear that like my opinion of Sam is not that he's like being a typical CEO here. Like I do think he's sorry some Altman. Yeah. Yeah.

Austin: Yeah. I definitely think there are many ways in which like some Altman is not typical CA, but there are many ways in which like every CEO is like not a typical CEO.

And I guess. Thus far, I have not been like particularly like concerned by like Sam Allman specifically his actions. And there was a point here around as we're talking about the NDA stuff, I think the way like OpenAI and Sam responded to this publicly on Twitter, it was like this was like a mistake or especially the, like the worst parts, the like clawback of asset equity things were not like, intended to be a thing.

And they haven't actually. And they're like, trying to fix the like contracts or whatever, or make things right to their employees. And I think that's reasonable a fix I think one thing you might say is that Oh, they only do this in response to like large public outcry or something.

And it's like evidence of like bad behavior that they need to be policed more. But I guess I wouldn't take that line. And I would also take the there's a hundred things to get right when you are like an executive at a company. And oftentimes this kind of minor detail you don't pay that much attention to your, like lawyers tell you this is a standard, like non disparagement agreement.

I'm like, cool. Okay. That's I [00:10:00] really

Linch: doubt this is like what his lawyers dreamed up of without somebody in the exact team. Probably him, but at least somebody like. Yeah.

Austin: And I guess I haven't read the text of this non inspirational agreement very closely, or None of us have.

None of us have. Oh, is it not like a agreement that the non disparagement agreement itself is like private or?

Linch: Yeah. None of us has. Yeah.

Austin: The whole self sealing NDA thing. But to use like a closer example, I think Lincoln Quirk, the CEO of Wave, also had this non disparagement agreement in their contracts and do you model him as having the same amount of I really think it's just like a, Oh, you like you're filling out like a template.

Like I do this a lot for manifold, like contracts. I just go to a website. There's a template. I like check some boxes. And I think in the moment I'm making like should I make them sign a non disclosure agreement? Yeah, sure. This seems better for the company. It seems safer.

I don't see why I wouldn't do this. And I did click that and move on. And that was like, not even a considered decision.

Linch: Yeah. If this was like the type of. That's in that 90 percent of commodity does. I really think that there just isn't a case like in fact, for the Lincoln Cook thing, I think I had a dialogue like one of those were less on dialogues with Ruby where I was like this thing's bad, but it's like very reasonable from different cultural norms.

You shouldn't really help expect it to be held up to the rationalist ones. But I think A lifelong non disparagement agreement with like your best equity being called back as far as I could tell, I don't know about this. It seems very unusual and it's also silencing. And I don't know, this is just all pretty unusual.

And I don't know how unusual it is because obviously because they're all self silencing, not a lot of people have come up with saying, oh, this is like really common, like lots of companies do it. It's I've heard it's Yeah. Almost none actually do this.

Austin: You don't think they get credit for walking it back or being like may a couple of I think Sam actually is like, yeah, this is my mistake, like an oversight as and I was, this is my fault, I am like into charge and they're going to try to fix it.

I'm like,

Linch: I think morally they don't, because this is like pragmatically probably one of the better options they could take. They do get like some competency points.

Austin: Yeah. Yeah. I don't know. I guess I'm just like occupying like a very different view, which is Oh, these people are like probably pretty like good hearted in general, or it's have a good heart in general.

Yeah. And are trying to like, They have a million things to manage and like this one was like a big issue. Whoops. They're gonna try to fix that and like they're gonna move on and try to do like

Linch: other things Yeah, so the details of their walk back is also interesting, right? They didn't say that like they will nullify the previous ones will nullify the specific condition of non discrimination previous ones Or anything like that, right?

They will talk to me with his response to if you have a disagreement with it Which is you know, an art thing and Kelsey has And I think others, especially Kelsey, I'm sure he's like seeing Kelsey's comments have pushed them multiple times, both, like official communication and on Twitter to just say that they were like, not go after people who disparage the company.

Um, signing the original agreement for, retracted this, original agreement and, just silence from them so far.

Austin: Yeah. I guess one thing I am keeping my eyes out on is the Daniel Kokotajlo prediction market. There's a manifold market on will he get his equity back?

I think for listeners he posted on the EA forum that. He'd given about he's given up equity equivalent to about 80% of his network, which means like if he currently had like

Linch: his future network

Austin: Yeah. His future net worth. Yes. Yeah. If he currently had, let's say like a million dollars, he possibly gave up $4 million yeah.

Of worth of OpenAI equity to be able to not sign that non-disparagement agreement.

Linch: Yeah.

Austin: And I think a lot of people probably rightfully lauded him as oh, this is like a good thing to do. This is a very like brave maneuver. But OpenAI putting its money where its mouth is would be like reinstituting that kind of equity to people like Daniel.

We'll see how that turns out. That will be like I think at least like some evidence to me to be like, Oh yeah, they are trying to do the right thing. And this is like a misstep.

[00:13:51] Is there enough criticism of OpenAI? Too much?

Linch: Yeah. Yeah. I think that would be some evidence to me. I think also if there was, how do I put this?

If more people like start like criticize OpenAI and it's just not much Out there afterwards when like it feels like safe then obviously that feels better to me I feel like there's probably just a lot of pressure to not criticize open for various reasons like I know within EA like it's pretty common for people to want to get jobs within like labs and stuff and Often they just don't criticize the labs as much as they could because of that especially for like technical safety researchers.

Austin: I think that's like a thing that probably happens to some people. I feel the opposite though. I feel like on places like the EA forum, there's like too much nitpicking or trying to find flaws with OpenAI in a way that's not very charitable. And I think there might be like the opposite effect where like you gain more like status points or something like that within the community if you are seen as being like hard headed or pushing for the right thing or something like that and I don't think these cancel each other out.

I think these are probably very different people actually, who are like staying silent, the like researchers who hope to work at OpenAI versus the like people [00:15:00] who are more in the Doomer camp and trying to push against it. But yeah,

Linch: I think that it's like the more informed people choose to be silent roughly.

And Less informed people who just see from a distance that this whole thing is like really fucked and like foggy and like lots of black ink.

Austin: Do you have examples of people who you think of as like particularly informed who are like notably not saying much?

Linch: Almost all the time. ex-OpenAI employees.

Austin: Like the ones who left in certain Anthropic and also the ones who have left recent or the ones who have left recently, or I guess, yeah, both

Linch: Both camps, like you expect like more, like you expect more of the Anthropic people to at least have said something about now, like most likely the split of acrimonious so those are the obvious ones, I think, let me see.

Yeah, I think, I also know people who work at like safety non profits who like say less than they probably should.

Austin: Yeah,

Linch: Again, it seems although that said, I think if anything critiqued by Anthropic is even more hard. We're like, Like more

Austin: hard to come by? Like critique of

Linch: Anthropic? Yeah, we're like, people are even more self silent, within the EA circles, like even more self silent in critiques of Anthropic than of OpenAI, would be my guess.

Austin: Yeah, I think that's like plausible. I don't know. Again, I am operating from like some other model where like people aren't saying much because there isn't much to say is like my best guess. They're There's not there may be like some kind of small things that like, didn't quite seem right that they like put out there, but I dunno, I'm like thinking of my like recent, like leaving a manifold, for example, which like, you could imagine somebody that could look at that and see my public statements and think there's gotta be like a bigger story than this, or there's gotta be like, bitter acrimonious things that like Austin isn't like saying, I don't really think there is, I don't know.

I think that

Linch: You're like one of the most patently transparent people out there. And you also have criticized them when you left and stuff. And that seems like fine. I don't think I don't know. I think it just looked like a pretty different situation from the outside.

Austin: Sure, but I know Jen Leakey, I hope I'm pronouncing his name right.

He did have this Twitter thread that was like a critique of the way, do you think he has like many things that he's not saying as well, or? I think he has

Linch: less things that he's not saying would be my guess, but he probably, there's like standard NDA things that cover like some of the things he can't say.

[00:17:15] Superalignment's 20% compute promise

Linch: It's known about He hasn't publicly said much about the compute stuff. I'm going back on the computer agreements. It's only through like press leaks, which may have been from him. I have it for other people that we need like more details about the compute stuff. I don't know. There's probably other things like that.

Austin: Yeah. The compute stuff is interesting because that is like a concrete example. OpenAI committed 20 percent of the training compute up till the point when super alignment was launched

Linch: to

Austin: the super alignment team.

Linch: Yeah.

Austin: It roughly seems like they didn't get that compute or maybe open, I didn't quite honor the agreement.

I imagine in the In the heads of the people who are like managing open, I, they were like, Oh, we will we promise this, but we don't, we can't deliver all of it right now. Maybe you have to maybe next year, maybe the year after or something like that. So maybe it doesn't feel like a broken promise to them, but it definitely seems a bit of a broken pro or at least insofar as the like super alignment team, is saying Oh, we didn't get enough compute. I probably mostly take them at their word. I do feel like compute is tricky though, because like compute is basically like money or like like influence in some sense in this like world of OpenAI, I imagine.

Where it is like one of the most like scarce resources that they have and deciding where to spend your compute is I don't know how to express it. It's like spending, like allocating your headcount, allocating your like, yeah, it's one of the most

Linch: important things. That's why it's not sounding like a strong promise and incredible one. We're like a different way if a country says with, um, I don't know if a politician says we're gonna give 20 percent of money into Social Security or Medicare or something and they end up giving like 1 percent and that obviously is a broken promise, right?

And money is a very important thing for the US government.

Austin: Yeah, I agree. It seems a bit like a broken promise, but like looking at the super alignment team case, right? Yeah. At some point, like if the people you promised to all decided to leave, like relatively early on like super alignment has only been around for a year or so, is my understanding.

And I assume this like promise of 20 percent of the commute was not like, oh, and we will give this to you right now if you need to like, stop all of what OpenAI is currently operating on and just do super alignment stuff, right? I don't know, I still have I do feel basically like this is like one of the like less good things, one of the clear, like clear broken promises, perhaps but still like I see lots of ways in which like there's like room into ways for two teams to talk to each other, the super alignment team within the OpenAI team

Linch: and

Austin: both sides feel like they're like doing their best, but

Not quite getting to the right thing.

Linch: Yeah, I think that I think they didn't break the explicit wording of the promise, which is I think we also are getting at this at 20 percent compute secured over the next four years that they could have meant like one possible interpretation is that they have 20 percent per year or so, up to 80 percent annual quota from that year and then they're going 2026.

That's one possible interpretation. Yeah. And that would in fact be the cheapest way for OpenAI to honor that commitment, given exponential trends [00:20:00] and things getting cheaper. But that is not how most people in the public interpreted it. That is not how I assume people within super alignment interpret it.

Austin: I guess I would have interpreted it as 5 percent per year for each of those four years or something like that. If it's, 20 percent over four years.

Linch: That's not how it works though. It's not 20% of the annual quota. I know

Austin: it like, but they had fixed the amount of compute that they promised.

20%. So I think like a rough, very reasonable thing was to like dole it out equally over the four years time. That's probably, yeah. So I would've expect knife that would

Linch: be like 20%. So to be clear, that would be 20% of that amount that year over five, four years, right? Than 5%.

Austin: Or I'm thinking

Linch: of computers not analyzed tone by default computers like continuous usually we think of like literal GPUs and 20 percent of GPUs being reserved for that time. And then as you get exponential trends that like as a percentage of total stock of GPUs that gets less and less, but it's not like 20 percent of that, like 20 percent running per one year. And then that's split into 5 percent over four years.

Austin: I realized this is a thing I actually don't know very much about. I had assumed that it was like there are a certain number of floating operations that have been used to train. All of OpenAI models up to this point, one fifth of that moving forward is and maybe that is like a billion floating operations, but not floating operation per second.

Just like total like operations. That was my understanding or my, but I haven't, I don't know what the like agreement, I haven't looked into this as a thing, I guess we could like

Linch: research. I haven't thought about it that way, but that also makes sense. Yeah. Yeah, I was thinking of the physical. Thing, but I guess, yeah, I guess like compute, like it's pretty reasonable to find compute as the, yeah, flop.

Yeah, the flop.

Austin: Okay. Yeah, I guess in any other case, like this is like getting into the weeds somewhat, but yeah. Overall, I think this is like an important point where it seems like if the super alignment team didn't get the compute that they thought was according to them based on this promise, that is like a broken promise in terms of like OpenAI trying to help align each guy.

[00:21:50] The EA departures from OpenAI

Austin: Cool. I think another point and we're already started talking about this is like the impact of all the people leaving OpenAI lately. It's, I forget who it started with but at this point there's Ilya and Jan the two leads of super alignment. A bunch of Daniel Kokotajlo, William Saunders, like Leopold Aschenbrenner.

A few other people, I think .

Linch: Yeah.

Austin: Who are kinda like somewhat related tied into the EA space. Other people who are like, roughly in my social circles or like friends of friends or something like that.

Linch: Yeah.

Austin: Who yeah. Have like suddenly left in the last like week or so, or have been leaving over the span of the last couple of months.

Linch: Yep.

Austin: What do you make of this.

Linch: Yeah, for the world, pretty neutral I assume this is already baked in they couldn't get anything done yet. OpenAI, or couldn't get much done at OpenAI, so they left maybe they will kick out, maybe they won't.

It was pretty voluntary, but yeah, hopefully

like the best case scenario here is that Ilya and Jan weren't very good at politics and weren't able to do the thing, but like deep in their hearts Sam and Greg and the rest, like really wanted to care about safety and would work with safety on somebody with a group of people who are better at politics.

And. Maybe John Schulman, I think is the new person. Maybe he would be that person. Maybe they'll find somebody else. That's the best case scenario. There's most of it was just like political ability within the company. I was scenario is of course that like most of the safety work was like ultimately just trying to do some food safety washing in a way that's not super blatant.

And no matter who's in charge there, like it's just not gonna get much done.

Austin: Yeah. Maybe I'm like narratizing this a little bit too much, but I do like roughly feel like it is like a sad thing that once upon a time or not even that long ago, there was like At least somewhat like a good faith effort. It seems to have people working in the same company and like talking to each other and like sharing ideas there.

And now that like split is much more like visible, something like the split between AI safety and OpenAI or EA and OpenAI or something like that.

Linch: Yeah. Yeah. I think some of their point that came out was that, I don't know about AI but yeah, OpenAI, I think that was somewhat engineered from OpenAI side.

Like they didn't like the. Depressing them as like a EA associated company for various reasons, including like the FTX labs and stuff. So it's understandable why they feel that way. And it also makes sense. They aren't much of a EA company. So it makes sense that they don't want to be associated as one.

Austin: Yeah, this is the kind of ties into another issue which is something around like many places wanting to turn that turn away the EA brand at this point, like viewing it as like negative, which I think is a whole other discussion, but we won't try to go like too deep into that. I do think I think it's

I feel like this can't have been that big of

what, what gives us a sense of like the, like brand of EA being negative like negative assets. Yeah. was a major contribution to like OpenAI kind of separating from the recent people.

Linch: 5 percent maybe 3%. I'm not sure.

Austin: Sorry. You think about 3 percent of the like rationale for that was, yeah. Okay. So maybe then when [00:25:00] we're pretty aligned like, I don't think it was like that big of a deciding factor. That sounds right.

Linch: Yeah. They were like, yeah. According to, I think the New York Times, like while they're reporting, yeah, they were like, internal discussions within OpenAI of like how to distance themselves from EA.

But yeah, from like just senior people, but I don't know, like you said, a startup CEO works on a hundred people, a hundred things, probably wasn't that big a priority. Sure.

Austin: Yeah I guess I, maybe I'm just like very conflict averse or something, but I just like, like it very much when like people like get together and talk and work out things and don't like much when like things are more like, oh, clear like lines in the sand.

We're going to get into a fight. I haven't been doing this building like e/acc versus EA, like narrative as oh, this is seems like it's going to worsen the quality of the discourse and make a lot of people mad and get a lot of clicks for not very good reason. Things like that.

Linch: Yeah. I see that. Yeah, all else equal. I would also rather be in a world where like people are like, Like actually talking to each other, understanding each other's points. I don't notice myself being, like, less smart for less capable of clear reason when, more tribal impulses take me.

That's not great, I like to be smart and capable of good reasoning.

[00:26:10] ScarJo

Austin: I guess moving on from this then into the third, I think topic du jour Scarlett Johansson's? What is it? Not sure how to pronounce these names. Yeah. I'm sure that Scar Jo is how I've been seeing her easier way.

Yeah. I think it's

Linch: Johansson. Yeah. Johansson. Yeah. I don't really

Austin: know. Was this a thing that you also had like strong feelings about with regards to OpenAI? I think a lot of people are painting this as, OpenAI choosing to use Scarlett Johansson's voice or voice alike in Skye as like a broken promise or like another example of them not being able to be trusted.

And this once again seemed to me to be a bit like catastrophizing or a bit like, making a mountain out of a molehill. Interesting.

Linch: I don't think broken promise is quite the framing I would have. I don't think they promise much to her. I think it's more I think there's a few things. I think impersonating people is kinda sus but I'm not sure. I I could be convinced otherwise here. I think that's one thing. I think that I'm saying that it's, I don't know for sure, but it seems like a pretty odd coincidence if it's coincidence.

Austin: Yeah. I listened to a very short clip of, Skye versus Scarlett Johansson's voice and in her, and they seem similar, but like similar in the way that like two women coming from like similar backgrounds might have like similar voices.

It might seem similar, especially like through the lens of a somewhat robotic like screen filter or something like that in the case of Sky's voice. And

Linch: yeah, I think also like the model has probably iterated like multiple times over time. So it would be hard to find the ground truth or it would be hard to I would probably want to look at like a lot of clips to know and try to pinpoint the exact time that.

Scarlett claims that her friends like said it sounded really similar as well.

Austin: Yeah. So there's one question which is around are they like telling the truth or lying, and that would be like a big one. I think probably one that we may like not find out, may never find out whether Sam Altman or the people working on a sky voice, like intentionally.

Try to make this similar to the scarred your voice. I really hope we do find out, but it might be pretty hard. Yeah. Yeah. I think like absent of that just reinforces your belief in if you were suspicious of the OpenAI people, you'd be like, oh, this is another example of them, like doing sketchy things, especially like breaking a promise or again, not a promise, but like infringing on ScarJo's rights or something like that.

Linch: Yeah.

Austin: And you said earlier that you weren't so sure about is this kind of impersonation even bad? Yeah, for one, it's not exactly impersonation. I think I would typically think of impersonation as pretending to be. A person, and this is something more like the whatchamacallit, the like mid journey artist issue.

Do artists and like voice actors and I guess writers have a right to their style to the way they draw their work, to the way they sound in this case, to the way they write. And I guess I also Some of my sympathy for OpenAI is I don't really know that I believe that creators should have this right.

This is like possibly a very controversial take. And I know like in many places, especially in like video game development, for example, many video game artists are strongly of the belief that this, all this AI generated art stuff is really bad. And you'll see like things like Steam reviews when they use an AI generated game, they'll get like downvoted to oblivion because people are like, no, we want to support the artists.

And not pitch in on this side of the corporations, like shilling AI. I kinda take the opposite side, I take the corporation side, or the AI side in this, where yeah and maybe this is a part of what gives me more sympathy for the, again, OpenAI camp over the ScarJo camp.

Linch: Yeah. Yeah,

Kinda, I, yeah, I don't have strong opinions with here. I feel like I don't have very good intuitions here, and this feels like the type of thing that I don't know, hopefully like reasoned discourse and public opinion and like some form of public philosophy and like people like trying to express their preferences honestly, ideally people who aren't like, super economically or socially incentivized to say things.

Hopefully we'll get to a point where this is like the type of thing that society can agree [00:30:00] on. I don't really feel like this type of thing was like a clear truth of the matter, so to speak like absent social context, way that, like murder is like probably bad in almost all contexts where it's not bad.

Austin: Sure. It's one of those like issues that is probably gonna get more like salience over time and let the legal system and like society around it needs to think a little bit more about what like copyright means and what an artist write to their likeness feels like those things are much easier to generate and produce.

Linch: Yeah.

I do feel like, at some level I feel more strong intuitively that if somebody writes in my style, that's flattering but if somebody tries to talk in my style or act like me, even if they disclaim in loud letters, they're not me, that feels I don't know.

Off creepy.

Austin: Interesting. There's a thing where like parody for example, is protected under like us law, at least. Aside from that, if somebody tries to use my voice, I guess I don't have a very strong association with my own voice. I never thought of my voice is like core to my identity, but I can see how maybe somebody like Scar Jo cares a lot more about that.

That is like her part of her, like life's work. How she makes her like living.

Linch: And it's also relevant that most. I don't know, many of us have read like "Reasons and Persons" or similar work. Many of us don't tie that much stuff to our personal identity within EA. In a way that I think is probably almost the opposite for like actors, for I think knowing people somewhere in between and things for like professional actors are definitely on the extreme end of they just tie a lot to their personal identity.

Like for various reasons, including that it's like very professional and relevant to them.

Austin: Yeah. And I almost wonder if there's one place where our intuitive like we see an actor, they are like a monkey the way we're a monkey and we don't like the monkeys. It leads us a little bit astray in terms of what should be the right thing to do from like a overall, like social economic perspective or something like that.

Where we think yeah, the actors deserve their privacy rights, like image likeness rights.

Linch: Yeah.

Austin: Whereas like in the opposite world where like, people just didn't like anyone could like make like deep takes of anyone. Yeah. There would just be much more creative, like media and people like consumers would be a lot more satisfied.

I think there might be something like that. I'm just thinking about this for the first time. This is not like a very like,

Linch: well

Austin: recent argument,

Linch: I think. I

Austin: mean,

Linch: I guess we'll see. I guess I'm not convinced that you have that much better out because of this. Especially yeah, if you exclude parody and stuff, but I don't know.

So it is If you think about outer science, and I think like some of it like discovery, it does feel weird that the first person to discover a style gets to really own it fully. Yeah. But I think a lot a lot of copyright doesn't look like this, right? Token their own works for the idea of good visibility.

Obviously not a hero's journey or anything, and I think I don't know, like tying your identity to your voice, it's like a pretty reasonable thing, like in history to do it's it doesn't seem that extreme. It's one thing. And then my pre deception point, I'm still like really weirded out by OpenAI saying that they can't single the voice actors out because of privacy.

That, speaking of privacy, that does feel a bit odd to me.

Austin: Yeah, definitely if I were OpenAI, I think I would be like asking the voice actress who voiced Skye to like, give up those privacy rights, maybe pay them extra money and be like, can we like tell people this was you?

And it'd probably be good for this person's career,

Linch: yeah. Yeah. On priors, like how frequent is it that actors don't want to be credited for their work? Yeah. Come on. It just feels very odd. And

Austin: I like

Linch: Open, I managed should find the few actors in the world who happen to really care about their privacy the same way that they ex-employees, they just care so much about privacy that they can't complain about Open.

I, yeah I, I dunno man.

Austin: I mean it would definitely make sense to me as like a, just a pure business move. Like probably as a business open, I want be to think of sky as opposed to as like X person's voice. Yeah. It's

Linch: but except for this current situation, it's much lower liability if one of 'em, do something shitty, or speaks out against OpenAI. People don't want to identify this as Sky speaking out against OpenAI. Yeah, where they do something shitty, like they, like, when he had DOI, they don't want to. Yeah. Obviously, there's lots of reasons why You don't want that.

Austin: Sure. But yeah I guess like then there's some thinking of are they again, like lying about it being a privacy thing? Could they have like relatively easily gotten that unlocked and are just choosing not to do so because it would be bad? Is it like actually secretly Scarlett Johansson's voice like in disguise?

I think these are all like,

Linch: Yeah, we're like the scholar Johansson famous scholar Johansson impersonator that I hired.

Austin: Yeah. And I guess I just like, once again don't really take the like, conspiratorial view on these things. I just feel like, Oh yeah, let's like, maybe some of these things.

I don't know.

Linch: Yeah. Famous conspiratorial, but I feel like this is just straightforwardly like a, amoral business doing amoral businesses. If they could get away with it, they would.

[00:34:41] Should AGI be pursued by a for-profit company?

Austin: Okay. Maybe moving back away from the stuff going on today and just like bringing it back out to a more overall how do you feel about OpenAI's approach?

How do you feel about like OpenAI's approach? Like overall to date?

Linch: Yeah. I think it's like better than some worlds I think it's one of the worst. Among the worst words were [00:35:00] possible words out there. I think like somebody in that situation Sam Altman is like person, personality aside, a company in the position of opening the eye Can in fact do a lot worse, can in fact be a lot worse, I think they do talk about safety quite a bit.

They they try to be reasonable a lot of the time They like, You know, spend some, not a lot, of resources, but some resources on public goods and safety. They, talk about it from the beginning. They care about near term ethics issues and spend non trivial resources on that and so forth.

In that position, I think they do a fair number of reasonable things. I think that I talked about this with you, and I, Briefly mentioned this on the EA forum. I think that putting themselves in that position is somewhat blameworthy that I think

Austin: as a, like a for profit corporation

Linch: trying to deal with AGI.

Austin: And alternatively, like they could have found this as they did originally find this as a nonprofit, but you could have also imagined it as like a government sponsor initiative or like an academic initiative or something else. Yeah.

Linch: A lab within a company like not doing AI stuff at all.

There's many options.

Austin: Yeah. I think that was like a very interesting point when you brought that up that like they had different, or one could have imagined trying to push to create AGI not inside of the API. For profit company

Linch: specifically a for profit company dedicated to AGI. Yes, dedicated to AGI.

Austin: And I guess this is where I think something like in my world, which is very much influenced by tech startups being as like, very like good. I feel path of a startup doing a thing, it's just so much more likely to succeed or execute well on a thing compared to the government quote unquote doing a thing for an academic group or a bunch of researchers like doing a thing.

I think that's like on one hand. On the other is the like fundraising considerations where open eyes started like actually as a nonprofit. And I think like a famously, they decided to switch over to more of a for profit model and started raising like billions of dollars when they realized, okay actually to make AGI work, we just need the funding and the nonprofit, we're not going to get enough in donations to be able to pay the cost of all this compute.

So we are going to have to raise from venture backed raise venture back funding from like for profit like investors. You, do you what would the alternative one be maybe like try to shut down.

Linch: Yeah. If it, if you can't do something morally, then don't do it at all.

It's like the obvious. Yeah, a nonprofit. Don't like, don't think yourself as though, people building AGI, that's one of the groups that people like. Helping to make a GI safe. There's lots of stuff you could do. They could become a policy nonprofit, they could do safety research the way that like file does what red work does.

There's just like a lot of options if if Yeah. Get acquired.

Austin: Yeah. I guess I, I feel like these are all like, like pretty unimpressive slash like very bad alternatives in some way that like, I guess I

Linch: care more about the thing being good than the thing being impressive, which I think is a big difference between like my like internal ethos and that of Silicon Valley.

I think Oppenheimer was not like morality tale about Oppenheimer being a great person and everybody should aspire to be like him.

Austin: Yeah, I hear that. I think that Trying to be good is like pretty important. Probably like more important than being impressive. But at the same time, I think like one thing, the like altruistic mindset, the good mindset kind of undervalues is just the degree of like uncertainty in the world, the like you can't get to, I think, really good outcomes just by Trying to do good things.

You actually need feedback loops. You actually need the signal that comes from running a thing at larger and larger scales. And there's the classic, like where has a lot of the good in the world come from and libertarian pill people tend to think a lot of like why we are like happier and like rich and like prosperous and like many as a, like human beings.

Is because of like capitalism or like the liberal, like world order or something, the ability for people to start things with capital and grow those and scale those up. And that creates most of the good in the world. I think this is pretty compelling. I think it's I think many people agree with this take already.

Yeah. Yeah. And then there's like a specific sub question of like, for the issue of AGI, should you use the same engine of like capitalism to create this thing? Or should you try to do something else? And I guess I feel like most of the alternatives you named would like something like not do a very good job of getting to AGI.

And then that being the case, like some other for profit might do that better job. And I guess if you're sitting in Sam's position or the founders of OpenAI's position like maybe the very reasonable thing is okay we will be the ones to do it instead.

Linch: I guess like for large scale scientific projects.

I don't really buy this

Austin: Okay,

Linch: like I feel like there are many large scale scientific projects that are not like Built by a company that's like dedicated to doing it.

Austin: I think like once upon a time, yes, but I think the track record of these things lately has been quite bad. Like lately, for example, I think of SpaceX as a much more [00:40:00] impressive achievement in getting things like, into space than like NASA where once upon a time I was like, Oh yeah, the

Like shut space, like shuttle system got like astronauts to the moon.

Cool. But like now, like not so much. It's not

Linch: true. It's I don't know, there's like motion rovers and there's like far flung like space things and like deep space probes and stuff that all come from not SpaceX. Yeah, I think they think it exists. SpaceX is mostly like scaling up the thing that like we do.

Where do you know how to do and doing a lot cheaper and a bunch of other things that are good.

Austin: But then the doing a lot cheaper part is actually like quite important. Oh, I agree. Yeah. Or it enabled like cheapness enables like a qualitative shift in what is possible and enables things like Starlink.

And I think we'll enable sustainable, like Mars travel or something like is the goal of this SpaceX launches followed by Starship development.

Linch: Yeah.

Austin: Yeah. So I don't know like how closely you follow the space. I'm not like, the most ardent like SpaceX fan, but I have a really strong sense that like a company in this case was a much better way of like working on like the scientific achievement of at least like reusable rockets and things like that.

I think like maybe things that the government seems to do better than like for profit. I was going to say something like fusion, but actually I'm also not sure on this point. Like historically, a lot of the research into fusion reactors or things like that was like government funded.

I know Sam Allman has invested in like companies like Helion, and I don't know enough about the state of progress here to see which of these is more impressive or will be.

Linch: Yeah. Yeah. That's fair. I think a lot of basic research is still funded by the government for like offense Government funded things in academia eventually spin out once the science is established.

Austin: I definitely agree with that, but the the basic research to me, it's not very clear that what HDI looks like, especially with, if the scaling hypothesis is true, is what it requires to scale it versus engineering work, stuff that seems to progress much faster inside of large companies.

Yeah,

Linch: so my guess is that if you wanted to build HDI within a government lab, it would be more expensive. Maybe it's 10x more expensive, maybe it's 100x more expensive. Okay. I'm just like, this is such an important thing that it's like worth paying the cost to do it well which in a way that I would not say for most other things.

I would not say this for spaceships. I would not say this for fusion probably.

Austin: Yeah. And I think like being very expensive I think has like other costs time to like deployment time to like us being able to use the thing. But also just It doesn't strike me that if the government had done 100x as much money people at the goal of making AGI, the resulting AGI would be more aligned, or something like that, or better by our lights.

It's not very non obvious. It could be the case, but it's not obviously the case.

Linch: I'd be interested in looking at the empirical record of how often disasters happen from More government y things versus non government y things. I've seen that, especially in governments within democratic regimes I would guess that the governments are safer.

Suddenly you could go into the communist territory and be, like, bad, but, yeah, I think it's just I think I just have the intuition that the government run things are, like, in fact safer. Because, they sacrifice other things for safety. And this is even more true if you control for competency of people.

Austin: I think this is where like safety becomes too broad of a concept to be like very useful, right? There's one kind of safety, which is do people like die or something like that as a result of the work here? But I don't think this is the like useful kind of safety when it comes to AI safety.

It is like something a lot more like complicated, like weird will this like thing that we are summoning out of the ether, hold our beliefs and act the way we want it to. And I think it's I won't expect much of a difference, like for that kind of, for do humans die as a result of AI from GPT 4, GPT 5 level of like work, whether that happens in like OpenAI or like X government.

Linch: Yeah. I don't think there'll be necessarily be a huge difference, but a couple of reasons to think that there might be some. When is that like alignment research might look. Less like Mech Interp stuff or like the scaling hypothesis and look a bit more like fundamental research or like somewhere in between like even Mech Interp feels like a Bit less you could just like still compute and make a thing happen. Although maybe you could especially at higher levels of capabilities I don't necessarily bet on it at least.

The other thing is just are you willing to make the thing stop? Are you willing to slow down? If things look scary? And my claim is that like the incentives for a for-profit company that's focused on ai, it's like basically the worst possible one here.

Not literally the worst. Like you could have two, two countries like that are really racing and, war, hot war against each other, maybe. Even then, I'm like, I don't know, we have like nuclear treaties and we have de escalation before, I'm just like, like a company voluntarily shutting down its main product because of like speculative claims, or that just seems like insane, that just seems like really hard you have to like, Be extremely honorable.

Never risk, either risk or really honorable and well meaning person. And I guess I don't think of any of the AI labs as that. In fact, very few are that. It just seems like a really weird and insane incentive structure to place yourself in.

Austin: Yeah, I can see that. I [00:45:00] guess if I try to think of OpenAI as like a climate global or something like that they have this like particular profit incentive to produce a lot of oil and as a result, they do things like lobby like Congress or try to spread disinformation about what climate change is going to look like first.

Then that would make me, that makes me a little bit like more sympathetic to the idea that like, oh, maybe the for profit like structure for, OpenAI is not that good.

Linch: Yeah, I think that's the main thing I want to emphasize. But then a secondary point is that it's, that's the main thing they're doing.

It's not they're like a, they're like Microsoft or Google or some other company that has a bunch of product lines. It's it's not like asking Google to cancel their I don't know, the YouTube line. Because it's like going poorly and like bad for people. It's like asking Google to cancel like the ads or like asking Facebook to cancel social media.

Just like a very high bar to ask for a pause or a slow down.

Austin: It's a good point. But I feel like the societal external response to this kind of thing, like regulation and protests and things like that will probably do a reasonable job of keeping things in line. And I think this particular structure of like company tries to do a thing with a very clear mission and a profit incentive.

[00:46:10] A hypothetical AGI "Manhattan Project"

Austin: And insofar as there are negative externalities, That is socially handled by other aspects of society designed to try and control for that. That seems pretty okay. Compared to some hypothetical this was like a Manhattan Project style initiative where lots of government people hired the smartest researchers to work for the government instead.

That doesn't seem better, I guess.

Linch: Yeah. Why not? You have a Manhattan style of like projects that people, because people think AGI is going to be really good for the world and lots of people have mixed opinions. Unlike the actual Manhattan project, this was not done because of the Nazis, either because the U. S. is way ahead of China or because they figured out enough international agreements such that any other possible contenders also have their arm in the pie. You had this and. I don't know. Civil society is talking about the risk, like people debate this regularly, like it's a topic of conversation among other intellectuals, like things keep progressing and that really gonna there's no way to stop it?

Austin: I think the structure of the Manhattan Project was like much more responsible towards the U. S. government or maybe like a few like specific like generals and people inside the U. S. government.

Linch: Yes.

Austin: And this feels like a much less broadly likely to be good for society incentive reporting structure than what OpenAI has, which is yes, like responsible to the shareholders, but also it's customers and it's employees and like a lot more society.

I also feel famously Manhattan Projects was very secretive. Like they don't tell very many people about what they were doing. I think the OpenAI as it is set up and the way most corporations are set up, they are much more Open in the sense of letting people know what they're doing and trying to appeal to the public opinion in a way that I think government projects do not have to win.

Linch: I guess the version of the Manhattan Project for AI imagining is one that's much more public to

Austin: Much more open than Open is today

Linch: yeah, like to be like responsive to civil society, like it's not going to be like a secret line in the government's budget because under war, it's going to be pretty clear. We spend like X dollars on this thing. If it's like trillion dollars or more of spending, like it's going to be a type of thing that's covered in presidential debates.

It's a type of thing that like, how the intellectuals in nearby areas like talk about regularly. They're going to have papers in Science. They're going to be position papers. They're going to specific details of if it turns out it's there's like algorithmic breakthroughs that are scary rather than just scale, then like specific things will be hidden. But a lot of the details are going to be public.

Austin: I guess,

Linch: It might look more like Apollo, for example. Sure. I can, the American public did not want Apollo to happen.

Austin: I feel like, I'm not sure if this is just a bias because I'm again, closer to this Silicon Valley world, but I have much more of a sense of how to get like my own preferences. And I think a lot of other people's preferences, embedded into the way OpenAI currently operates.

Compared to if it were like any like government like initiatives like an Apollo, right? If Apollo is doing something, I think it's very dangerous. What do I do? Write a letter to somebody maybe is like my, like the first thing I think of.

Versus with OpenAI go to Twitter. Like you can try to do a public campaign. You can like change how you pay for things. Like being like a consumer of this thing very much effects.

The touch points between like me and most of I think society and a company. Feel much better developed than the touch points between a government program and like society.

Linch: It's just really wrong to me. I think like for OpenAI specifically, I think like underlying reason is that they're like not that far away from a social group. And if I want, if I really want to complain about OpenAI, I have a very specific technical proposal that I would like him to implement. I could just ask I could find somebody, I could just ask them to do it.

And that's probably true for you as well, and many people in the audience. But this is not like normal civil society. This is not the majority of people who aren't like,

Austin: Bay

Linch: Area tech adjacent.

Austin: Maybe if you thought about it, it's like Walmart versus like Kansas, like [00:50:00] city government or something like that.

Yeah. I think

Linch: if you're somebody in Kansas, you probably have Is there a way to change like Walmart's behavior, change like Kansas city's behavior than Walmart's?

Austin: I'm not sure. I do feel like Walmart is a lot more like responsive to your like requests or something like that. If you are like unhappy, you can talk to somebody and Walmart, like a Walmart like store manager is going to listen to you and tell you like what they can do about it.

That kind of thing.

Linch: Yeah. Walmart is like a very consumer focused company. Sure. Supposedly I don't like what Palantir does. Okay, maybe that's not a good example because they really partner with the government. Blackwater which is a military, like they're like not really a military contractor, but they it's not because of like laws that's like making them scary.

What do I do?

Austin: Yeah, I guess it's hard for me to know because I don't really know what Blackwater is, but I agree that there are many companies in this reference class. But, OpenAI does not seem to be in this reference class. They seem to be in the try to put a thing in front of many people

Linch: kind of

Austin: reference class.

And that I think gives them like a lot more like direct incentives to, do things right by their customers. I think that's actually like a pretty important point that I'm developing this idea like right now, just as we're talking, but I actually think that's like pretty important as far as they're trying to create this AI that like serves the world, they actually have a very strong incentive towards keeping their users like happy and providing them like a valuable service in a way that like, yeah, It's a very powerful engine of corporations and does not exist nearly as much with like things like NASA or the Manhattan Project.

Linch: Yeah. Manhattan Project especially. Yeah. But yeah, interesting. Yeah.

I do agree that you get a lot more like feedback from reality, more data more ways to like to see how like customers want your product to be done.

[00:51:45] Government vs company services

Linch: How much of that is a government versus company thing? Would you say that Like Fox, okay, maybe that's not, like CNN say, do you think that's like more aligned than the BBC, for example?

Okay.

Would you say that?

Austin: Oh, sorry do I think CNN is more aligned than the BBC? Yeah. I don't even know very much what the differences are. The BBC is like a British, like broadcasting company.

CNN is like an American one. They're both like relatively neutral in terms of the political spectrum.

Linch: Yeah. And BBC is owned by the British government and CNN is not.

Austin: Yeah. I have very little Sure, sure.

Linch: Yeah, that's fair. That's fair. What examples that we might both be familiar with? NPR, any chance?

Maybe media is not a good example. I don't know. Um, E versus if the government's running it.

Austin: Doesn't the government run PG& E, I actually don't really know. They seem very close to the state monopoly. I guess I agree that the services are not very good or something like that.

Linch: Yeah, I don't know. UPS versus USPS.

Austin: Sure. Or like Amazon delivery versus UPS or something like that. Or

Linch: Amazon delivery runs on UPS and USPS

Austin: among many other I think Amazon has it own, like driver and things like that. I guess I feel like generally a higher level of quality service from like the Amazon private delivery service than from like public, like US Mail or something like that.

But Okay.

Linch: Yeah, I feel like they're similarly, sorry, Amazon specifically now, because like it's trying to do a different thing, but you're trying to get one of my packages from like, point A to point B. I'm not seeing like a big difference between USPS and UPS. I think I agree with that.

Yeah. Yeah, but maybe that's not fair. I know that SOEs in general do worse somewhat. State owned enterprises? Yeah. Like this is true in China at least, and I assume it's true across the world. But yeah, that's exactly it though. I think the thing that, I guess the thing that matters to me is if I'm not a consumer, can I get something to stop?

And it feels like I have more leverage for government things than corporation things, especially, and one reason I don't is because governments are more powerful, but, like we are like kind of conditioning against that if we're like actually have AGI.

Austin: Cool. Did you have any I think we're like running at like about an hour or so. Did you have any other things you wanted to chat about on the OpenAI side? Things that you thought are like pretty important to your worldview of, like, how you think about OpenAI that you didn't get a chance to express?

[00:54:08] Is OpenAI democratic?

Linch: I guess one thing, I don't know how strongly I feel this because I'm confused about how much I care about democracy and stuff. But I do think there's an important sense in which like OpenAI is like just a very strongly like Sorry, the mission of creating AGI, one of the most powerful technologies of all time and Doing not sure what with it That just feels very anti democratic

Austin: What about it feels anti democratic?

Linch: I guess it's like a lot of people did not consent to this danger being imposed on them. Assuming you have non trivial pDoom, which my understanding is that OpenAI leadership at least publicly said they do.

Austin: Yeah, I agree that they have like non trivial pDoom. I again think that like they expect them working on it lowers pDoom compared to [00:55:00] them not working on it compared to some other competitor working on it. Or

Linch: yeah,

Austin: I guess they'll do the main thing or like

Linch: Yeah, I think making that decision on behalf of humanity feels a bit odd and maybe that's the best choice they could make like I'm not like unsympathetic to like situations where like, you just got to do this in and wait for other people to catch up But it does feel like as a society we should be asking questions or something

Austin: Maybe. I think many things about OpenAI seem like quite democratic to me or the way they are opening up the access of chat GPT to many people is like the kind of practical democracy that matters a lot. Anyone can use like the cheaper versions of the models for free in a way that is helpful to their own lives.

Linch: Yeah,

Austin: I think that kind of giving things to lots of people and trying to help them is I don't know if it's democratic exactly, but along the vision of trying to help many people as opposed to concentrating the like good things that come out of AI to a few to the people who pay a lot for it, or people in charge of it.

Linch: Yeah. I think that's like a framing of morality that many EA's or like it's a framing of like altruism or like justice. Sure. It's not like a framing of like participation. It's not like a framing of giving people the agent agency to choose whether to, if things are happening.

Austin: I think like giving people like AI in their pockets is a kind of like democracy can mean one kind of thing. Ability to vote for, legislation or vote for people to enact legislation. I think the spirit of democracy goes beyond that. It is something like, giving people the ability to flourish pursue happiness, and I think they're doing quite well on that regard.

On the question of is the process by which OpenAI, makes their decisions democratic. Again, going back to the, many people are consumers of OpenAI, they pay OpenAI money for a service that is provided. There's a kind of democracy, more correctly, it is a kind of like a plutocracy, maybe like the people with the money in this like ecosystem, like influence the decisions of OpenAI.

But I think like practically it is not just like who is paying, but also like their free uses probably matter a lot to them. The. Like OpenAI as an entity is trying to serve these people well, because like in the future, some of them might actually turn into paying customers, that kind of thing.

Linch: Yeah, I think this is a type of experiment that if you don't have massive negative externalities or massive externalities period, then I'm happy to, this is just part of being a good civil society democracy isn't everything freedom is, the underlying thing that matters more than democracy, and having lots of companies choose to do their own thing, and serve their customers and so forth, that enhances a lot of freedom.

I think the question does become that once you're all like, talking about percentage points of doom, but much, much less, honestly. It's the same type of thing that you really want, free exploration ideas and I don't know maybe you should maybe that's like the safest path forward You know, in any light and perspective, the most Oh freedom has forward.

Austin: Yeah. The like standard way by which like democracy will have it say in AI is that like our elected leaders in like Sacramento or Washington are going to sit down and think hard and talk to like lobbyists paid for by like many sides of the equation to think like what kind of legislation should we enact on how it, how the like development of AGI happens.

And. I guess they could OpenAI could be more democratic if they like literally conducted like polls or something like that or like surveys of many people in the US around the world about how they feel. Are you thinking like that's the kind of thing that is missing from? What a responsible steward of AI would be doing?

Linch: I don't think that in the incentive structure to pick themselves, I don't think that being super responsible. I think that there's been some resources I'm trying to figure out how to govern AI that obviously not enough, but, they have many constraints. They are, like, thinking about how to get more inputs and so forth.

I think that's a benefit. I think, like, some of the things they do are like weird, like lobbying, like bunch of the lobbying things they do, or saying that they care a lot about safety, but then like secretly lobbying against, but like they try pretty hard to lobby for LLM exception in the EU, like AI laws, for example, stuff like that, it's weird, but overall I'm not like mad massively.

problematic in the position they put themselves in. I think more it's like just as a civil society, do we want this type of thing to happen? Do we are we excited about like individual companies making choices for things that might like, Do myself and obviously there's a probabilistic, right?

Like a tiny probability that like microplastics will be really bad for humanity or whatever. Sure. We should probably, that should not be enough to like, stop plastics companies from like refining petroleum stuff. At least like not the information we have.

Like maybe we should spend a bit more resources into researching or something, but I feel like at anywhere close to level, like anywhere close to like risk of car accident, killing off humanity, I don't know, it feels odd that civil society is not like putting more pressure and maybe rethinking whether we want this to be done at [01:00:00] all by private companies.

Austin: Yeah, I do expect this kind of pressure to just continue to ramp up over time as like people like start seeing what AI is capable of doing and tying it to, their very normal takes on how bad it would be if very smart AI, walk the earth. Yeah.

Other than that, I guess the critique of OpenAI as somewhat anti democratic doesn't really resonate with me, but I can see yeah, there's something in this reference class maybe of Oh, a nuclear weapon or but much more dangerous maybe that should be a thing that like democratic processes have some kind of, yeah, control, decide

Linch: how it happens and whether it should happen at all and when it happens and stuff like that

Austin: are like, yeah, the drivers in the space, as opposed to right now, they're like, are looking to like, play it like a check on the space. Yeah. Cool. Cool.

I think that's a good place to end it. Thanks. That was a great conversation. Yeah. Thanks for chatting.

Linch: Yeah. Hope there are five people or however many people like to view this.

Austin: You five people, you are awesome. Yeah. Awesome.

Linch: Yeah. Yeah. One of my favorite people. I'm sure. Yeah.

Overall, we’ve learned a lot and are happy with the projects we’ve funded, but are less excited by impact certs than before — it’s been hard to get investor interest, and we still haven’t found a use case where certs led to better funding decisions. For the next quarter, we’re trying out a bunch of things (including Manifest, regranting, and prize challenges) while looking to get product-market fit.

What we’ve been working on in Q1

ACX Grants 2024

Manifund hosted the 2024 round of ACX Grants, which included both direct funding to some projects and an impact market for any project that opted-in and did not receive direct funding. ACX directly funded 33 projects for a total of $1.35mm; on the impact market, investors funded 12 projects to their minimum bars, for ~$50k total at a combined valuation of ~$200k.

Background

Astral Codex Ten (ACX) is a blog by Scott Alexander on topics like reasoning, science, psychiatry, medicine, ethics, genetics, AI, economics, and politics. ACX Grants (ACXG) is a program in which Scott helps fund charitable and scientific projects — see the 2022 round here and his retrospective on ACX Grants 2022 here.

In this round (ACX Grants 2024), some of the applications were directly funded by Scott; the rest were given the option to participate in an impact market, an alternative to grants or donations as a way to fund charitable projects. Manifund transferred funds from donors to grantees; hosted project proposals on our website; and ran the impact marketplace.

ACXG Direct Funding

Scott evaluated all of the applications along with his team of judges, then directed a total of $1.35M to 33 projects. You can see the results on ACX or on the “Grants” tab of Manifund’s ACXG 2024 page. Example grants:

• Open source predictors for polygenic screening, by Gene Smith

• Building anti-mosquito drones, by Alex Toussaint

• Solving Far-UVC ozone generation, by Jacob Swett

This was relatively straightforward to run, as making direct grants is similar to what we did in our regranting program from last year. The biggest operational burden was collecting funding from a variety of donor-advised funds for donations to the ACX Grants program.

ACXG Impact Market

Projects that weren’t given a direct grant were eligible to participate in an impact market, and the ones that opted in are listed on the “Impact Market” tab. You can read more our announcement post about the impact market here. Example projects that were funded:

• Run a public on line Turing Test, by Cameron Jones

• Scaling Legal Impact for Chickens, by Alene Georgia Anello

• Platform for migrants to start microbusinesses, by Emily Kerr-Finell

Five retroactive prize funders agreed to participate: next year’s ACX Grants, the Survival and Flourishing Fund, the Long-Term Future Fund, the Animal Welfare Fund, the Effective Altruism Infrastructure Fund. In 2025, they’ll award prize funding to successful projects, and the investors who bet on those projects will receive their share in charitable dollars.

Some key differences between this impact market and previous impact markets we’ve run:

• Retro funding: With regard to retro funding, this impact market:

  • …was the first one we ran that has multiple retroactive funders. We’ll have a better understanding of how this “went” after the retroactive evaluations happen (next year).

  • …had the highest range of retro funding. Retro funders committed to evaluating projects retrospectively as if they were a prospective grant with a 100% success likelihood, which meant that they will be competitive with the $5-$33 million that the retro funders disburse yearly. This also made some of the investing process more difficult, as it also made the likelihood and quantity of retro funding less clear. However, the goals of the retro funders were broadly well-defined (and further collated in our announcement post). Also, attempts to make this more well-defined end up trading off against flexibility of both the retro funders and the project creators — and it will have been good to test the limits of an impact market in this direction.

  • …had the least defined criteria. We broadly said “aim for the goals of the retrofunders,” and let everything else figure itself out, rather than “here’s a specific goal, achieve it.” This also made some of the investing process more difficult, as it also made the likelihood and quantity of retro funding less clear.

• Project type & scope:

  • Since this impact market was for projects that were already rejected from a previous round, we did experience a bit of selection, both advantageous and adverse. Most noticeably, a lot of the projects that would’ve been obviously-good to fund had already received funding from the previous round, which meant that only the projects that were not obviously-good were included (which included some fantastic projects, just not the ones which were obviously fantastic upon first glance).

  • There was no central or obvious “theme” (besides being for ACX Grants, which itself is incredibly broad). Previous impact markets had been about AI worldviews, or forecasting & prediction markets, or the Manifold community. The wider theme lead to a more diverse array of projects, making it harder for investors to compare different projects.

The ACXG Impact Market overall went okay, but there were definitely a few areas that the impact market was lacking on:

• Participation: we had less participation from investors — both in terms of external $ invested and comments/interaction — than we had hoped. One of the projects directly funded in ACX Grants (Building anti-mosquito drones, by Alex Toussaint), went viral on Twitter and received a bunch of small donations. We would’ve liked to see this kind of momentum on one of our impact markets proposals, to build up a more sustained environment of donation & interaction.

  • We launched a $500 micro-regranting program to get more investors to participate, see below for more details on how that went (spoiler: pretty well!).

• Complexity/confusion: the mechanisms behind an impact market are confusing, but the auction mechanisms we used were especially confusing & complex. We’re considering ways that we can mimic the style of a SAFE or an MFN clause from VC investing onto an impact market.

• Financial operations: we ran into a few unexpected and uncorrelated issues with our payment processor, Mercury. For the vast majority of grants, there wasn’t any problem, but for a few grants it lead to delays of about 4-5 weeks

Micro-Regranting Program

Halfway through the impact market, we noticed that it was receiving less attention than we would’ve hoped. We decided to run a “micro-regranting program,” an extremely scaled-down version of the regranting programs we’ve run in the past.

We described the program in the application form:

Manifund is assembling a small cohort of charitable-funding enthusiasts to try their hands at impact investing ... If selected, you'll get to allocate $500 of charity budget to invest in impact certificates for the 2024 Astral Codex Ten Grants program; you can donate any profit from the investments to the charity of your choice. We're looking for people who are excited to thoughtfully consider how to allocate their investments, similarly to Manifund's regrantors (see examples here), and to post some feedback on the funding decisions they make.

The description “the world’s tiniest and most wholesome hedge fund” was aptly used. And overall, the micro-regranting program was a huge success! Some parts that went particularly well:

• We posted a few announcements to elicit applications, and we received 26 applications in just a few days (way more than we expected, much faster than we expected), and approved 24 of them. The high acceptance rate wasn’t reflective of an especially low bar for micro-regrantors, we just got a ton of fantastic applications!

• We moved pretty fast on this — it took us just about a week from the first announcement of the micro-regranting program to having built up a solid cohort of micro-regrantors.

• The micro-regrantors left many thoughtful comments, gave us and project creators detailed feedback over calls & in the Discord, and helped allocate about $17k in funding. Project creators seemed to enjoy chatting with the micro-regrantors in the comments, helping everyone better understand these proposals.

• The micro-regrantors overall enjoyed the process. From our feedback form (n=17), the average response to “did you enjoy being a micro-regrantor?” on a scale of 1 (hated it) to 3 (neutral) to 5 (loved it), the mean response was a 3.8, and the median was a 4. Some anonymized quotes:

  • “People talk on the [EA] Forum all the time about grantmaking, some suggesting it is wizardry for the talented few, others suggesting it should be for the democratic-EA masses. Unsurprisingly, it turns out the truth may be somewhere in between. I took on the role mainly for the educational (to me) / experiential value, and I wasn't disappointed with what I got in return for my time investment.”

  • “[I enjoyed] having to seriously think about what projects will get traction and stand a good chance of making impact, [as well as] engaging with other projects from a microgranting perspective - purpose to my engagement.”

  • “I aimed to build a small portfolio of projects and I did build it. It was fun!”

• We received a lot of helpful feedback about the Manifund site that might have otherwise been difficult to tease out of the average user.

We were really happy with how the micro-regranting program went. But some things went less well than we would’ve hoped:

• Most of the projects needed more than $500, which meant that micro-regrantors needed to coordinate with other micro-regrantors — or expect follow-on donations from external folks. Both of those happened, but weren’t done nearly so well as possible. Some ideas we had for this in the future:

  • Have both more-concerted coordination points and more coordination points overall. We had a channel in our Discord for micro-regrantors, which was really great for coordination between a few of the micro-regrantors who used it, but didn’t work well for most of them. Some possible ideas: a video call with all of the micro-regrantors to chat about projects they wanted to coordinate on; a dedicated section on Manifund for micro-regrantors to discuss grants; a shared email thread; asking micro-regrantors slightly more strongly to join & use the Discord; etc.

  • Reach out to external donors. We had at least one large donor contribute additional funding based on decisions made by micro-regrantors, but there were obvious steps we didn’t take to seek out additional large donors; e.g., email some we know, make an announcement on the EA Forum, etc. Part of the problem here was that we were pretty time-crunched, which constrained the actions we felt comfortable request of external donors.

• Some of the micro-regrantors found the structure of the impact market (including the valuations, auction mechanism, equity structure, retroactive funding, etc) extremely confusing. This was mirrored by some of the feedback we received from project creators, and we intend to reduce the confusion & complexity in future rounds.

• Some of the micro-regrantors strongly disliked the framing of an impact market as encouraging micro-regrantors to be profit-seeking. We probably could’ve framed this better in our request for micro-regrantors, and selected for those who’d be especially interested in the profit-seeking component of an impact market.

• The timing was pretty tight — we would’ve liked to give micro-regrantors a few more weeks to sift through projects and make decisions, which would’ve also let them coordinate more and let us contact some external donors.

Again — overall, we were incredibly happy with the results of the micro-regranting program. We intend to run additional programs like this in the future.

Big thanks to Anton Makiievskyi, who funded all of the micro-regrantor budgets — and allocated an extra $5k to Jason’s budget after noticing Jason’s deep & thoughtful comments on the platform and on the Discord. (And that sort of dynamic is exactly the sort of thing we were hoping would come out of the micro-regranting program!)

Manifold Community Fund

We ran an impact market for Manifold community projects from November 2023 to February 2024. We had some ideas about changes that we wanted to make for the (then upcoming) ACX Grants 2024 impact market, and thought this would be a good way to test those on a smaller scale, in addition to a way to get some cool contributions to Manifold.

In the end, we paid out $16k to 12 projects:

Here are some changes we tested on the structure of impact markets, and how we think they went:

• Ongoing evals and payouts: instead of allocating all of the prize money at once at the end of the program, we awarded prize money (i.e. made buy offers) once per month, for a total of three evaluation rounds. We think this provided useful feedback for creators, as evaluators generally wrote comments about why they valued the project the way that they did, but that the actual offers were pretty irrelevant. At each intermediate evaluation, we were only valuing work done so far, so expected valuation was always higher than the offers and no offers were actually accepted until the final round.

  • We also learned that evaluation takes a long time and isn’t that fun—participation in evals from the Manifold team went down each month, and in particular the final evals were delayed by an entire month.

• Automated Market Maker (AMM): for previous impact markets we had run, there was very little active trading on the certificates after the seed round, perhaps because the only way to trade was via limit orders so you always needed to find a trade partner. To make active trading easier, we implemented an automated market maker. This did facilitate more active trading, but overall didn’t seem worth it. For one, all of the AMMs were overpriced, perhaps because we implemented buying and selling but not shorting, which made betting against a project if you hadn’t previously bet for it impossible. Additionally, some equity and some USD was needed to seed the AMM, which meant that the seed round valuations had to be artificially inflated.

• No auction: in order to simplify the fundraising process, we eliminated the auction[link to about] from the seed funding process. Instead, creators just chose the amount they wanted to raise and made an offer at the implied valuation. This was indeed a simplier mechanism to understand, but put a lot more pressure on the creator to set their parameters wisely, because their funding minimum and their funding goal are the same number, and choosing it requires trading off probability of raising any funding at all with the chance to raise more money. On the other hand, with the auction, they can set a low minimum and let the market give them more if it so pleases. All-in-all, this change seemed to cause people more confusion, not less.

So, most ideas we tested with the Manifold Community Fund we did not end up bringing to the ACX impact market. There were, however, some really awesome projects that came out of this, specifically Case’s contributions to the Manifold codebase and wasabipesto’s updates to Calibration City.

Overall, this updated us against impact markets for this kind of use case. As with the ACXG impact market, we were disappointed with the amount of investor participation upfront, and throughout the Community Fund. The best projects to come out of this didn’t actually get any upfront funding, which suggests a simple prize would have worked just as well in this case. And making an easy-to-understand UX for both the creators and the investors in the impact market remains an unsolved problem.

ChinaTalk Essay Competition

ChinaTalk is a twice-weekly newsletter and podcast on China, technology, and US-China relations to an audience of over 35,000, hosted by Jordan Schneider. They have previously received a $17k regrant on Manifund.

Late last year, we partnered with ChinaTalk to sponsor and host their essay competition:

ChinaTalk has teamed up with Manifold to give away $6,000 in prize money to essays making a bold prediction on the future of China. We’ll choose winners in February, 2024.

The top 3 essays will win a cash prize and will be interviewed as a guest on the ChinaTalk podcast to discuss their essay with Jordan.

We’re inviting you to submit an essay making a bold prediction on a subject that ChinaTalk’s audience would connect with.

We agreed to partner on this because:

1. We wanted to try to make “impact certificates for essay contests” work. We’d previously tried this with OpenPhil’s AI Worldviews Contest; however, we didn’t receive many submissions (and none of the winning ones) through our platform, as we didn’t secure an official partnership with OpenPhil in time.

2. Chinatalk offered to take on most of the promotional and organizational work, in exchange for Manifold paying for their time to do so. As promotion and organization are two of the most expensive pieces of organizing such a prize competition (with the third being the cost of the prize itself), we were curious to see how outsourcing this would turn out

3. We wanted to commission high-quality essays that would make good use Manifold’s prediction markets. We’d seen other competitions (like the ACX Book Reviews Contest) produce extremely good essays that got a lot of organic virality, and thought ChinaTalk was well-positioned on to do this given their reach and expertise in the subject matter.

In the end, the competition was won by Lily Ottinger, with her piece "Sino-Soviet Split 2.0”. While we learned a lot from this partnership, I think that Manifund did not achieve the goals we’d set out to with this competition, for a few reasons:

• The impact certificate framing was confusing to the competitors; our site also presented some technical difficulties for essay submission (which we’ve now fixed). Moreover, we didn’t catch the attention of any investors; the use case of “speculating on the best essays” did not seem to catch on.

• We received fewer submissions than we’d hoped for, and most submissions did not make use of concrete predictions or markets in the way that we’d wanted. Jordan himself did write up a great post with concrete predictions, but this was largely orthogonal to the essay submissions themselves.

• Though the ChinaTalk team (both Jordan and Caithrin Rintoul, who handled much of the competition logistics) were professional and excellent to work with, any kind of partnership across orgs is fraught — there are always many delays in communication and nuances to work out. In particular, this competition was something like our third priority (behind the Manifold Community Fund and ACX Grants), so we were less responsive and focused on making the competition go well.

People Updates

Manifund is growing! For the first time since we started our work, we’ve brought a new full-time hire: Saul Munn, to run strategy and operations for Manifund. You might remember Saul from his work on Manifest last year, which he’s returning to organize this year; he also cofounded the OPTIC intercollegiate forecasting tournament.

We also engaged two folks to help us out as consultants:

• Dave Kasten planned and conducted a variety of interviews with charity leaders outside of EA, to see whether they might be interested in working with Manifund on impact certs.

• Lily Jordan wrote up explanations of how Manifund operates, kicked off our ACX Grants microregranting program, and improved a bunch of site UX.

Thanks to Dave & Lily for all your help! We’d love to publish the research they’ve done for us at some point; in the meantime, you can view drafts here and here.

Finally, Austin has officially left the Manifold team, to focus on Manifund and related work. On a related note, we’re considering rebranding away from “Manifund” so that the separation between Manifold and Manifund is more clear (the two orgs are separately incorporated with separate finances, and no longer share employees).

Site Updates

Throughout this quarter, we’ve generally been more focused on wrapping up funding rounds which required a lot of operational support than with site changes. Still, here were a few significant features we added recently:

• Comment reactions: you can now react to comments with emojis, or even leave tips on comments which go to the commenter’s charity balance.

  

• Better agreement display: now, the grant agreement displays the grantee’s agreement as a signature rather than simply a checkbox, and displays Manifund’s countersignature.

  

• Draft mode: for the ACXG impact market, we added a “draft mode” that projects can be in before they’re published as proposals. In this mode, the creator can save edits to their project over multiple sessions, rather than needing to publish immediately.

Where we’re headed in Q2

With the ACX Grants impact market and other rounds wrapped up, we now want to take a step back to explore, “what do we need to get to product market fit”? We have a bunch of ideas on directions we can take Manifund; over the next quarter, we’d like to get more concrete signal about which of these are promising. Some of the initiatives we already have in flight:

Manifest 2024

Manifest is a festival for forecasting and prediction markets. It’s more of a Manifold event in spirit, but it’s being planned by the Manifund team — co-led by Rachel and Saul! It officially kicks off Jun 7-9, but LessOnline and Summer Camp start the week before; we expect this to be quite busy with this in the next few weeks leading up to the event.

(Get your tickets at https://manifest.is; it’s going to be a blast!)

AI Safety Regranting

We’re renewing our AI Safety regranting program, allocating ~$1.6m to 6 world-class experts in the field: Adam Gleave, Dan Hendrycks, Evan Hubinger, Leopold Aschenbrenner, Ryan Kidd and Neel Nanda. More details about this soon, but we’re excited to be running this again and hope to scale it up, with more funding and regrantors this year!

Mixtral Challenge (with Trenton Bricken & Dwarkesh Patel)

On a recent podcast episode, Dwarkesh, Trenton and Sholto suggested the idea of having a prize challenge to better understand the open-source Mixtral model, based on the success of the Vesuvius challenge. This got memed into existence on Twitter, with Nat Friedman and others offering $45k in prize funding for this challenge. We’re working with Trenton and Dwarkesh to run this competition on Manifund, where we host the contest site and handle funds transfer, while Trenton serves to coordinate and judge.

Fiscal sponsorship

We’ve been talking with orgs like Asterisk, PauseAI, and Sage about providing a fundraising platform and fiscal sponsorship relationship (aka 501c3 tax deductability for donations to these orgs). We’re already informally sponsoring many of the projects on our site, but providing this as a proper service solves a pain point for orgs who need this to receive donations.

EA Funds collaboration

We’ve been in close contact with Linch and Caleb of EA Funds, as their work (especially on the LTFF) are the closest in nature to what Manifund has tackled. There are a bunch of ways Manifund and EAF could work together; we’d be excited to combine their grantmaking expertise with our focus on delivering great software.

Manifold Charity program

We’re still supporting mana donations to charity based out of our original Future Fund grant, but we’re likely to sunset this program once Manifold supports real money prizes via sweepstakes.

Other

People we’d love to chat with

As we scale up, there’s a lot of folks we’d like to talk to and learn from. If this describes you (or someone you can intro), reach out to austin@manifund.org!

• Creators who’d like to launch a grants program or prize challenge for their audience (eg Scott Alexander, Dwarkesh Patel)

• Grantees and orgs who are curious about working with Manifund, whether as a fiscal sponsee, impact cert founder,

• Medium-sized individual donors ($10k+/year) who find Manifund’s approach exciting and want to fund one of our programs, whether that’s regranting, impact certs, individual projects or something else

Links & callouts

• Aaron Silverbook’s Lumina Probiotics has been on a tear lately, with a very hyped launch; coverage from Scott Alexander, Yishan Wong, Richard Hanania, and other bloggers and press; and sales going through the roof — we’re glad to have been an early investor!

• Alex Toussaint’s mosquito-killing drone project recently blew up on Twitter, with a video demo of the sonar getting 1M views and attracting $2500+ in crowdfunded donations from 10 folks.

• Johnny Lin and Joseph Bloom, two separate Manifund grantees, have teamed up. Neuronpedia now provides web tooling for sparse autoencoders, using visualization tools similar to those used by Anthropic, and has since received follow-on funding from LTFF.

• Joel Tan of CEARCH (ACXG 2024 grantee) published this excellent report on the landscape of EA meta funding.

• Nonlinear Network is running a funding circle for AI safety and related fields, with hundreds of project applications, reviews from experts, and >$1m moved in donations in just a couple weeks.

• Alexander Berger of OpenPhil recaps 2023 and writes about their funding plans for 2024.

Ambitious ideas we’d be excited about

We’ve been brainstorming on a bunch of other things we could do; here are some very rough notes. If you’d like to help make these happen, comment below or reach out to austin@manifund.org!

Extending the regranting model

• “celeb regranting as a service” — manufacture more “ACX Grants & Emergent Ventures”

  • See this proposal we’d sent to Dwarkesh Patel: “Dwarkesh Grants” proposal, which led to the Mixtral Challenge collab described above

• microregrantors — crowdsourcing EA donation decisions

Building a robust impact marketplace

• Invest Anthropic & OpenAI stock — instead of certs on tiny orgs with little data, would there be more excitement for certs on larger orgs? This might just look like allowing

• Manifund DAF — medium-sized (~$10-500k) independent donations inside EA are currently a hodgepodge; we could build up an ecosystem of medium-sized donors and projects seeking funding.

• SAFIE (Standard Agreement for Future Impact Equity) — The SAFE simplified early startup investing. Can we do this for early charity funding — and make it backwards compatible with regular grants?

Funding more home runs

• Hosting amazing projects (”hits”) — in venture capital, a few hits dominate the portfolio; anecdotally, this seems to be the case in early-stage charity funding too. How do we find these projects and encourage them to come to Manifund?

• E/Accelerator — or do we try to directly create more of them, especially with a bias towards software (Manifund’s comparative expertise in the EA ecosystem?)

• EA Common App — when Manifold was first getting started, applying to a bunch of different funders for mostly the same information was kind of a pain. Like the collegiate Common App, can we make one for “EA organizational funding”?

For this round, the retroactive prize funders include:

• next year’s ACX Grants

• the Survival and Flourishing Fund

• the Long-Term Future Fund

• the Animal Welfare Fund

• the Effective Altruism Infrastructure Fund, and

Combined, these funders disburse roughly $5-33 million per year. A year from now, they’ll award prize funding to successful projects, and the investors who bet on those projects will receive their share in charitable dollars.

This post also highlights a few grants that the Manifund team are particularly excited about. And to kick things off, we’re launching a “micro-regranting” program: we’ll give you a $500 budget to invest into your favorite projects, and you get to keep the (charitable) profits.

Click here to browse open projects and start investing; click here to apply to our micro-regranting program!

[Edit: applications for the micro-regranting program is now closed. We received a lot of great applications, and are excited about doing this again in the future!]

ACX Grants 2024 Impact Markets

Astral Codex Ten (ACX) is a blog by Scott Alexander on topics like reasoning, science, psychiatry, medicine, ethics, genetics, AI, economics, and politics. ACX Grants is a program in which Scott helps fund charitable and scientific projects — see the 2022 round here and his retrospective on ACX Grants 2022 here.

In this round (ACX Grants 2024), some of the applications were given direct grants; the rest were given the option to participate in an impact market, an alternative to grants or donations as a way to fund charitable projects. You can read more about how impact markets generally work here, a canonical explanation of impact certificates on the EA Forum here, and an explanation thread from the Manifund twitter here.

If you invest in projects that end up being really impactful, then you’ll get a share of the charitable prize funding that projects win proportional to your original investment. All funding remains as charitable funding, so you’ll be able to donate it to whatever cause you think is most impactful (but not withdraw it for yourself). For example, if you invest $100 into a project that wins a prize worth twice its original valuation, you can then choose to donate $200 to any charity or project of your choice.

Meet the retro funders

Four philanthropic funders have so far expressed interest in giving retroactive prize funding (“retro funding”) to successful projects in this round. They’ll be assessing projects retrospectively using the same criteria they would use to assess a project prospectively. Scott Alexander explains:

[Retro] funders will operate on a model where they treat retrospective awards the same as prospective awards, multiplied by a probability of success. For example, suppose [the Long Term Future Fund] would give a $20,000 grant to a proposal for an AI safety conference, which they think has a 50% chance of going well. Instead, an investor buys the impact certificate for that proposal, waits until it goes well, and then sells it back to LTFF. They will pay $40,000 for the certificate, since it’s twice as valuable as it was back when it was just a proposal with a 50% success chance.

Obviously this involves trusting the people at these charities to make good estimates and give you their true values. I do trust everyone involved; if you don’t, impact certificate investing might not be for you.

As a (very) rough approximation, the four philanthropic retro funders usually disburse about $5-33 million per year. They are:

1. ACX Grants 2025

Next year’s ACX Grants round (2025) will be interested in spending some of the money they normally give out as prizes for the projects that succeeded in this year’s (2024) round. ACX Grants 2025 will be giving out prizes to people who pursue novel ways to change complex systems, either through technological breakthroughs, new social institutions, or targeted political change.

Previous rounds of ACX Grants have disbursed about $1-2 million per round, and you can find the lists of grants that those rounds gave money to here (1, 2).

2. The Survival and Flourishing Fund (SFF)

From their website:

[SFF] is a website for organizing the collection and evaluation of applications for donations to organizations concerned with the long-term survival and flourishing of sentient life.

Since 2019, SFF has recommended about $2-33 million per year in philanthropic disbursements ($75 million in total):

To find out more about the philanthropic priorities of the SFF’s largest grant-maker, Jaan Tallinn, see here. To see past grants SFF has made, see here.

3. The Long Term Future Fund (LTFF)

From their website:

The Long-Term Future Fund aims to positively influence the long-term trajectory of civilization by making grants that address global catastrophic risks, especially potential risks from advanced artificial intelligence and pandemics. In addition, we [the LTFF] seek to promote, implement, and advocate for longtermist ideas, and to otherwise increase the likelihood that future generations will flourish.

The LTFF usually disburses around $1-5 million per year, and sometimes disburses much more. You can view their yearly payout data here.

You can read more about the LTFF’s scope and expected recipients here, and find their public grants database here.

4. The Animal Welfare Fund (AWF)

From their website:

The Animal Welfare Fund aims to effectively improve the well-being of nonhuman animals, by making grants that focus on one or more of the following:

• Relatively neglected geographic regions or groups of animals

• Promising research into animal advocacy or animal well-being

• Activities that could make it easier to help animals in the future

• Otherwise best-in-class opportunities

The AWF usually disburses around $0.5-3 million per year, and sometimes disburses much more. You can view their yearly payout data here.

You can read more about the AWF's scope and expected recipients here, and find their public grants database here.

5. The Effective Altruism Infrastructure Fund (EAIF)

From their website:

The Effective Altruism Infrastructure Fund (EA Infrastructure Fund) recommends grants that aim to improve the work of projects using principles of effective altruism, by increasing their access to talent, capital, and knowledge.

The EA Infrastructure Fund has historically attempted to make strategic grants to incubate and grow projects that attempt to use reason and evidence to do as much good as possible. These include meta-charities that fundraise for highly effective charities doing direct work on important problems, research organizations that improve our understanding of how to do good more effectively, and projects that promote principles of effective altruism in contexts like academia.

The EAIF usually disburses around $1-3 million per year, and sometimes disburses much more. You can view their yearly payout data here.

You can read more about the EAIF’s scope and expected recipients here, and find their public grants database here.

…and (possibly) more.

If you want to join these four institutions as a potential final oracular funder of impact certificates, see this document and email rachel@manifund.org.

Some projects we like

Many of the projects are really great! We don’t have enough time or space to talk about all of the ones we’re excited about, but here are a few of our faves, from each of:

Austin

“Run a public online Turing Test with a variety of models and prompts, by camrobjones.”

Cam created a Turing Test game with GPT-4. I really like that Cam has already built & shipped this project, and it appears to have gotten viral traction and had to be shut down due to costs; rare qualities for a grant proposal! The project takes a very simple premise and executes well on it; playing with the demo makes me want to poke at the boundaries of AI, and made me a bit sad that it was just an AI demo (no chance to test my discernment skills); I feel like I would have shared this with my friends, had this been live.

Research on AI deception capabilities will be increasingly important, but also like that Cam created a fun game that interactively helps players think a bit about how for the state of the art has come, esp with the proposal to let user generate prompts too!

“Quantifying the costs of the Jones Act, by Balsa Research.”

Balsa Research is funding an individual economist or a team to conduct a counterfactual analysis assessing the economic impact if the Jones Act was repealed, to be published in a top economics journal.

I like this project because the folks involved are great. Zvi is famous enough to almost not need introduction, but in case you do: he's a widely read blogger whose coverage of AI is the best in the field; also a former Magic: the Gathering pro and Manifund regrantor. Meanwhile, Jenn has authored a blog post about non-EA charities that has significantly shaped how I think about nonprofit work, runs an awesome meetup in Waterloo, and on the side maintains this great database of ACX book reviews. (seriously, that alone is worth the price of admission)

I only have a layman's understanding of policy, economics or academia (and am slightly bearish on the theory of change behind "publish in top journals") but I robustly trust Zvi and Jenn to figure out what the right way to move forward with this.

“Publish a book on Egan education for parents, by Brandon Hendrickson.”

Brandon wants to publish a book on education for parents based on Kieran Egan's educational theory. He walks the walk when it comes to education; his ACX Book Review contest entry on the subject was not only well written, but also well structured with helpful illustrations and different text formats to drill home a point. (And the fact that he won is extremely high praise, given the quality of the competition!) I'm not normally a fan of educational interventions as their path to impact feels very long and uncertain, but I'd be excited to see what Brandon specifically can cook up.

(Disclamer: I, too, have some skin in the game, with a daughter coming out in ~July)

Lily

“Start an online editorial journal focusing on paradigm development in psychiatry and psychology, by Jessica Ocean.”

Jessica’s project takes up the mantle of a favorite crusade of mine, which is “actually it was a total mistake to apply the scientific method to psychology, can we please do something better.” She’s written extensively on psychiatric crises and the mental health system, and I would personally be excited to read the work of people thinking seriously about an alternative paradigm. I’m not sure whether the journal structure will add anything on top of just blogging, but I’d be interested to see the results of even an informal collaboration in this direction.

(Note that I probably wouldn’t expect the SFF or LTFF to fund this; ACX Grants 2025 maybe, and the EAIF I’m not sure. But I’d be happy to see something like it exist.)

“An online science platform, by Praveen Selvaraj”

I think generating explanatory technical visuals is both an underrated use of image models, compared to generating images of mysteriously alluring women roaming the streets of psychedelic solarpunk utopias, and an underrated use of genAI for education, compared to chatbots that read your textbook over your shoulder. I’d like to see more 3Blue1Brown in the world, and in general I’m optimistic about people building tools they already want for their personal use, as Praveen does.

Saul

“Educate the public about high impact causes, by Alex Khurgin.”

Alex wants to build a high-quality YouTube show, and seeks funding to make three episodes of the show on AI risk, antimicrobial resistance, and farmed animal welfare. This is something that I could pretty easily imagine the LTFF, EAIF, and possibly SFF retrofunding, and I'd additionally be excited about more people knowing about them & reducing their expected negative impact on the world.

Alex’s (and his team’s) track record is also pretty great: they’re clearly experienced & know what they’re talking about. I’d be interested in getting a better path to impact — what do they plan to do after they click publish on the videos? — but I’m sufficiently excited that I’ve invested a token $50 in Alex’s project to credibly signal my interest.

“Distribute HPMOR copies in Bangalore, India, by Aditya Arpitha Prasad.”

Anecdotally, the answer “I got into HPMOR” has been quite a common response to the question “how did you become interested in alignment research?” Mikhail Samin has had (from what I’ve seen) a lot of success doing something like this in Russia, and I’m excited about starting a similar initiative in India. This grant seems to fall pretty clearly within the range of retrofunding from the LTFF and/or EAIF. I’ve invested a token $50 in Aditya’s project to credibly signal my interest.

Micro-Regranting

[Edit: applications for the micro-regranting program is now closed. We received a lot of great applications, and are excited about doing this again in the future!]

Manifund is assembling a small cohort of charitable-funding enthusiasts who want to try their hands at impact investing. If selected, you'll get to allocate $500 of charity budget to invest in impact certificates in this ACX Grants round; you can donate any profit from the investments to the charity of your choice. We're looking for people who will thoughtfully consider how to allocate their investments, similarly to Manifund's regrantors (see examples here, or the examples directly above), and will post feedback and rationales for the funding decisions they make.

In addition to the investment budgets, we'll feature our favorite comments in a future edition of the Manifund newsletter and offer a mana bounty on Manifold for excellent comments.

Anyone is eligible, unless you have a project in the current ACX impact market. Please submit this form by the end of Friday, March 22 to participate!

(Reminder that anyone can invest in the impact market. The Micro-Regranting program will simply give you a free charity budget to invest on the impact market, rather than your own charitable dollars.)

Links & contact

Click here to browse open projects and start investing; click here to apply to our micro-regranting program.

If you’re interested in learning more about investing on an impact market, donating to projects directly, or even just chatting about this sort of thing, you can email saul@manifund.org or book a call here.

If you like our work, please consider donating to Manifund. Donations help cover our salaries & operating expenses, and fund projects and experiments that institutional donors aren’t willing to back — often the ones that excite us most!

At a glance

Here are some high-level stats that provide a snapshot of our 2023 activities:

• $2.06M sent to projects: $2.012M to grants & $45K to impact certificates

  • Of the totals above, $95K that went to grants and $40K that went to certs came from unaffiliated donors/investors, rather than regrantors.

• 88 projects were funded: 54 grants & 34 certs

• $2.22M has been deposited into Manifund, and $1.62M has been withdrawn so far.

• Below are the top cause areas of projects that got funded. Note that these are overlapping, that is, one project can be filed under multiple cause areas.

  • Technical AI Safety: 27 projects funded, $1.57M dispersed

  • Science and Technology: 9 projects funded, $118K dispersed

  • AI Governance: 10 projects funded, $112K dispersed

  • Biosecurity: 4 projects funded, $97K dispersed

  • Honorable mention to Forecasting, which only received $76K total, but encompassed 35 projects. This is because our two biggest impact certificate rounds so far—ACX Mini-Grants and the Manifold Community Fund—were centered around forecasting and funded lots of small projects.

2023 Programs

Impact certificates

Summary: Impact certificates are venture funding for charitable endeavors. Investors fund founders by buying shares (”certs”) in their projects, which pay out if the project later receives a retroactive prize. See also our and ACX’s explainers.

Assessment: 7/10

Impact certs have been discussed as a much more efficient system of funding charitable projects. Luminaries such as Paul Christiano, Katja Grace, Vitalik Buterin and Scott Alexander have championed the idea, but apart from a few small experiments with retroactive funding, no notable impact marketplaces existed prior to Manifund.

In 2023, we built out the website and operational experience to run impact certs end to end, from handling project submissions, to investments, to retroactive prize payouts. We completed 2 impact cert rounds and started 3 more, each testing different setups and domains.

How did we do? I would describe impact certs as “working as intended, but not yet hitting product/market fit”. Some possible reasons:

• Impact certs are a 3-sided marketplace, and come with the implied cold start problems: we need to get a retro funder, investors, and founders all on board.

  • We think of the retro funder as the hardest part; they have to be convinced that it’s worth paying out money for work already accomplished. Most donors instead think in terms of funding things prospectively.

  • But finding good investors is also not trivial! The investors’ decisions shape which projects actually get underway; they serve the role of grant evaluators in a traditional charity ecosystems, which requires a particular skillset and dedication.

• A lot of education is needed to explain the whole system: it has a lot of moving parts, and most people are unfamiliar with the venture ecosystem.

• Feedback loops via impact certs are slow. Projects take a long time to develop; updates have been infrequent from all sides (founders, investors, and funders).

• In theory, impact certs should encourage investors to seek out good projects and help them, though this doesn’t seem to have happened much. Perhaps this is due to small dollar amounts or us not having found expert investors to participate.

Despite these difficulties, I think impact certs are Manifund’s most exciting project, with the potential to transform the entire landscape of charitable funding. The economic theory behind them is elegant, and recent successes with advance market commitments (Operation Warp Speed, Stripe Frontier) are waking people up to the idea that prize funding is a great way of encouraging public goods.

Here are some reflections, broken down by round.

ACX Forecasting Minigrants: 7/10

• Stats: $30k prize pool distributed to 20 projects, Jan to Oct 2023

• See also: ACX announcement, Manifund’s retro, ACX’s retro

• This was the project that kicked off Manifund! Scott approached us saying that he wanted to run ACX Grants 2 on impact certs, but (understandably) wanted to try a lower-stakes test first. We picked “forecasting” as an area that Scott felt qualified to judge as a retro funder; I brought on Rachel to work on Manifund fulltime, and together we shipped the MVP of the site in 2 weeks.

• Lessons: Everything worked! We successfully created the world’s first ecosystem around investing in charitable projects and retroactively funding them. I don’t think projects produced were quite as good as those in the original ACX Grants round; not sure if that is due to impact certs, the much lower funding & prize pool ($20-40k), more limited scope, or something else.

OpenPhil AI Worldviews Essay Contest: 2/10

• Stats: 5 essays cert-ified, Feb to Sep 2023.

• See also: OpenPhil’s announcement, results

• This one was kind of a flop. We had launched ACX Minigrants and were waiting for results; we saw that OpenPhil had announced this contest and figured “$225k for essays? Great fit for contestants trying to hedge some of their winnings”. We reached out to Jason Schukraft, the contest organizer, and got his blessing — but unfortunately too late to secure an official partnership. We reached out to the essayists on our own, but most did not agree to create a Manifund impact cert (including none of the ultimate winners).

• Lessons: large dollar prizes + well-known brand are not sufficient to get a robust cert ecosystem started. Unclear if “essays” are a compelling use case for impact certs, as the investment comes after all the work is done. On the plus side, this was a pretty cheap experiment to try, as all the infrastructure was already in place from the ACX Minigrants round.

Chinatalk Prediction Essay Contest: ongoing (Nov 2023 to Jan 2024)

• Stats: $6k prize pool; expecting 50-100 submissions.

• See also: Chinatalk’s essay website

• Chinatalk (a Manifund grantee) approached us to sponsor their essay contest; I saw this as a chance where we could try out “impact certs for essays”, but this time with official partnership status. So far, Jordan and Caithrin have been amazing to work with; it remains to be seen if the added complexity of impact certs are worth the benefits of investor engagement.

• As an aside: the Chinatalk contest makes me wonder if there’s space for “hosting contests-as-a-service”. There’s proven demand and a lot of good work generated via contests and competitions (ACX Book Reviews; Vesuvius Prize; AIMO prize) but each of them jury-rig together websites & infra. Manifund could become a platform that streamlines contest creation (think Kaggle but for misc contests) and thereby fill in the “retro funder” part of the marketplace.

Manifold Community Fund: ongoing (Dec 2023 to Feb 2024)

• Stats: $10k x3 prize payouts, ~20 projects proposed

• See also: Manifold announcement

• Partnering with other orgs has upsides (publicity, funding) but also downsides (higher communication costs and more negotiations). Could we move faster and experiment more by funding a prize round ourselves? We landed on “community projects for Manifold” as an area we were experts in judging, and could justify spending money on to get good results.

• We set up the MCF with 3 rounds of ~$10k in funding once per month, instead of a single prize payout at the end. My hope is that more frequent prize funding will provide better feedback to investors & founders, and also teach us more about how to actually allocate retro funding, which is a surprisingly nontrivial problem!

ACX Grants 2024: ongoing (Dec 2023 to Dec 2024)

• Stats: $300k+ prize pool; predict there will be 400-800 proposals, with 10-30 funded

• See also: ACX announcement

• This will be our largest impact cert round so far. Some changes we made, compared to the previous ACX Minigrants:

  • Scott will be directly funding the proposals that he likes up front, instead of waiting to act as a retro funder. The other proposals may then be created as impact certificates.

  • We allow anyone to invest (via a donor-advised funds model) instead of restricting to accredited investors.

  • A group of EA funders have agreed to participate as retro funders (Survival and Flourishing Funds, EA Funds & ACX). This is particular exciting, as the first test case of having multiple different final buyers of impact.

• We’re happy that Scott found the first round of our impact certs compelling enough to want to expand it into the next official round! ACX Grants is also especially dear to our hearts, as it was counterfactually responsible for getting Manifold off the ground; to have the opportunity to come and help future ACX Grantees is quite the privilege.

Next steps for impact certs

We’d like to continue running impact cert rounds, in a more frequent, standardized manner (perhaps even self-serve). We’d also like to find a large, splashy use case for impact certs, that draws more attention to the concept and validates that it works well at larger scales. This would likely involve partnering closely with some deep-pocketed funder who wants to put up a large prize for a specific cause.

Potential causes we’ve been daydreaming about:

• Curing malaria (vaccine rollout? gene drives?)

• Big, yearly AI Safety prize

• Ending flu season in SF with FarUVC rollout

• Offsets for factory farmed eggs

• Carbon credits, similar to Stripe Climate

• Political change (e.g. housing reform?)

• General scientific research prizes in some weird field (eg longevity? fertility?)

Regranting

Summary: A charitable donor delegates a grantmaking budget to individuals known as “regrantors”. Regrantors independently make grant decisions, based on the objectives of the original donor and their own expertise.

Assessment: 7/10

Regranting was pioneered by the FTX Future Fund; among the grantmaking models they experimented with in 2022, they considered regranting to be the most promising. We believed that regranting is a good way of allocating funding for a few reasons:

• Faster turnaround times for grantees: Regrants involve lower overhead and less consensus, which leads to faster decisionmaking. Around the time we started this program, the EA funding space had just been severely disrupted by the collapse of FTX, which was making grant turn around times especially long. We had experienced this ourselves, as had many other people in our network, and this seemed like a problem we could help solve.

• More efficient funding allocation and active grantmaking: Regranting utilizes regrantors’ knowledge and networks, which may lead to above the bar use of funding on the margin. They’re often more connected to their grantees, which allows them to give more feedback or even initiate projects themselves, whereas most funders take a more passive approach.

• A better option for some donors: Delegating donations to regrantors is a unique donor experience, which offers a balance between maintaining control and minimizing effort relative to either directly giving to projects or giving to grantmaking organizations. Donors can pick individuals they trust to give intelligently and with their values in mind, but who may be better able to allocate the money as efficiently as possible due to some combination of time, expertise, and connections.

• Scalable: Regranting can scale up to moving large amounts of funding. This was a clear upside for the Future Fund, which was aiming to distribute 100M to 1B+ a year, though is less important for us now, as Manifund (and EA as a whole) are more funding constrained now.

Large regrantors

Stats: 5 regrantors with max budgets of ~$400k each, $1.4m total pool. 5 grants initiated, 15 grants supported.

Assessment: 7/10

See also: regranting launch announcement

We decided to do a regranting program after we were introduced to an anonymous donor, “D”, in May 2023. D liked Future Fund’s regranting setup, and wanted to fund their own regrantors, to the tune of $1.5m dollars across this year. This seemed like a good fit for Manifund to run, as:

• We had already built a lot of the necessary infrastructure for ACX Minigrants—a website, 501c3 org, and payout processes—and we could reuse these for regranting.

• We’d been beneficiaries of the regranting mechanism ourselves: Manifold’s seed round was initiated by an offer from a Future Fund regrantor.

• We thought we could give our grantees a better experience, with faster turnaround times and more involved grantmakers.

• We thought that by supporting regrantors and their grantees now, we could later convince them to participate in impact certs as investors & founders.

• We were in a lull anyways, waiting for ACX Minigrants projects to wrap up.

We’re reasonably happy with the quality of grants made! At a midpoint review in September, our donor D’s judgement was that the regrants made were somewhat better than initially expected. They were open to renewing the program for next year, pending other considerations.

We do wish that we’d gotten more consistent active participation from the large regrantors. Relative to the small regrantors, they tended to make fewer grants and engage in less coordination and discussion with other regrantors… which is understandable! D chose a bunch of very well-credentialed folks with great judgement, but it turns out such people are pretty busy or in high demand, and $400k is not a large enough budget to warrant spending significant amounts of time doing active grantmaking.

We had one notable exception to this pattern: Evan Hubinger was very prolific and dedicated, making 9 different grants to projects in technical AI safety. We ended up increasing his budget to encourage him to continue finding good opportunities.

Still, until the last couple weeks of the year, it looked like a large portion of the large regrantor pot would be left unspent, but then Dan, Evan, and Adam came through with some last minute recommendations and made use of their remaining budgets.

While we’re glad all of the money was sent to projects, this end of year influx wasn’t ideal. First, it’s unlikely that the best opportunities just happened to come along suddenly at the end of the year, which means something inefficient was going on. It seems more likely that they could have given to marginally better projects earlier in the year, or they could have given to these projects earlier. This is informative for how we set up budgets if we do this again, so we don’t incentivize waiting until a week before the expiration date to spend budgets.

Additionally, because we didn’t anticipate this influx, the regrantors offered more in funding than we had budget for, and we had to reject two grant recommendations that at other times we would have approved. This possibly created false expectations and a pretty bad experience for these two grantees, for which we are quite sorry.

Highlighted grants

• Evan Hubinger, $100k: Scoping Developmental Interperetability

  • This also got donations from Marcus, Ryan, and Rachel, though all after Evan’s initial contribution and recommendation.

  • This was an example of regrantors using their professional connections to find opportunities and that they had a lot of context on: Evan previously mentored Jesse, the recipient of this grant, and is familiar with the work of others on the team. With this all of this context, he said he “believe[s] them to be quite capable of tackling this problem”.

• Adam Gleave, $10.5k: Introductory resources for Singular Learning Theory

  • According to Adam, “There's been an explosion of interest in Singular Learning Theory lately in the alignment community, and good introductory resources could save people a lot of time. A scholarly literature review also has the benefit of making this area more accessible to the ML research community more broadly. Matthew seems well placed to conduct this, having already familiarized himself with the field during his MS thesis and collected a database of papers. He also has extensive teaching experience and experience writing publications aimed at the ML research community.” I’ll note that Evan also expressed excitement about Singular Learning Theory in his writeup for the above grant.

  • Like the above grant, this was an opportunity that Adam came across and could evaluate with lots of context as he previously mentored Matthew and continues to collaborate with him.

• Leopold Aschenbrenner, $400k: Compute and other expenses for LLM alignment research

  • From Leopold’s comment explaining why he chose to give this grant: ”Ethan Perez is a kickass researcher whom I really respect, and he also just seems very competent at getting things done. He is mentoring these projects, and these are worthwhile empirical research directions in my opinion. The MATs scholars are probably pretty junior, so a lot of the impact might be upskilling, but Ethan also seems really bullish on the projects, which I put a lot of weight on. I'm excited to see more external empirical alignment research like this!

    Ethan reached out to me a couple days ago saying they were majorly bottlenecked on compute/API credits; it seemed really high-value to unblock them, and high-value to unblock them quickly. I'm really excited that Manifund regranting exists for this purpose!”

  • Leopold started with a donation of $200k, and then followed it up with another $200k two months later after seeing the progress they’d made and learning they were still funding constrained.

Small regrantors

Stats: 11 regrantors with budgets of ~$50k each, $400k total pool. 11 grants initiated, 41 grants supported.

Assessment: 8/10

See also: Joel Becker’s reflections as a regrantor

After planning out the regranting program with D, we thought that 5 regrantors weren’t enough and that increasing the size of the regrantor cohort would be good for fostering more regrantor discussion and broadening the kinds of grants made. We decided to fund this from our own budget, primarily out of our general support grant from the Survival and Flourishing Funds.

One main benefit of choosing our own regrantors was that we could bet on unconventional candidates. As a variant on “hits-based giving”, we were “hits-based delegating”, giving budgets to wide variety of people, including many with little-to-no previous track record of grantmaking.

The small regrantor program is one of the places where Manifund’s experimentation has been most successful, in our opinion. Some of our regrantors have gone above and beyond, in their commitment to initiating good grants and helping grantees. After launching with our initial cohort, we opened up an application for regrantors and got very high-quality applicants. We approved Ryan Kidd, Renan Araujo, Joel Becker, Nuno Sempere and waitlisted many more promising regrantor candidates. We also had some regrantors who weren’t active or decided to withdraw which is fine — we were expecting this and structured the regrantor budget allocation with this in mind.

We thought that good grants made by our regrantors would encourage other people to donate to regrantor budgets. This doesn’t seem to have happened much; among donations made by individuals on Manifund, very little went towards regrantor budgets relative to specific projects listed via the open call. To that end, fundraising for our regrantors remains our main constraint on operating this program.

Highlighted grants

• Joel Becker, $1.5K & Renan Araujo, $1.5K: Explainer and analysis of CNCERT/CC (国家互联网应急中心)

  • Joel’s explanation of the origin of this grant: “Renan and I put out a call to an invite-only scholarship program, the "Aurora Scholarship," to 9 individuals recommended by a source we trust. We were aiming to support people who are nationals of or have lived in China with a $2,400-$4,800 scholarship for a research project in a topic related to technical AI safety or AI governance…Alexa was one of our excellent applicants.”

  • Empowering people in or with connections to China to do AI safety work seems pretty important. We were particularly impressed by the initiative Joel and Renan took in getting this off the ground—the prospect of facilitating active grantmaking like this is part of what motivated us to start the regranting program!

• Gavriel Kleinwaks, $41.7K: Optimizing clinical Metagenomics and Far-UVC implementation

  • Gavriel works at One Day Sooner, and is our only biosecurity-focused regrantor, and inline with this grant, her work has been focused on Far-UVC implementation! Here are some quotes from Gavriel’s writeup explaining why she chose this project:

    • “From my conversation with Miti and Aleš, it sounded as though there was a pretty good chance to unlock UK government buy-in for an important biosecurity apparatus, through the relatively inexpensive/short-term investment of a proposal submitted to the government. Biosecurity doesn’t have a lot of opportunities for cheap wins as far as I normally see, so this is really exciting.”

    • “This is exactly the type of project Manifund is best poised to serve: the turnaround needs to be really fast, since Miti is targeting an October deadline, and it’s for a small enough amount that at my $50k regranting budget I can fully fund it.”

  • Shout out to Joel again who recommended this grant to Gavriel! Because of a COI with the recipient, he didn’t contribute financially, but he still deserves a lot of credit for making this happen.

• Marcus Abramovitch, $25K: Joseph Bloom - Independent AI Safety Research

  • Joseph Bloom had a strong track record with independent AI safety research—he maintains TransformerLens (the top package for mechanistic interperetability), his work has been listed by Anthropic, he teaches at the ARENA program, and he came highly recommended from Neel Nanda. Unsurprisingly, he seems to have lived up to this track record and made good progress on his research according to the updates he sends Marcus each month.

  • This also received the biggest independent donation of any project on Manifund: $25k from Dylan Mavrides!

Open Call

Summary: “Kickstarter for charitable projects”: allow anyone to post a public grant proposal on the Manifund site, for regrantors and the general public to fund

Assessment: 6/10

Stats: 150 projects submitted, $95k raised among 40 individual donors

We started our open call to identify more opportunities for our regrantors to donate to. The open call worked well for this: it has surfaced many projects that we wouldn’t have seen otherwise. For example, I (Austin) allocated half of my own regrantor budget to projects that applied via the open call: Lantern Bioworks, Sophia Pung, Neuronpedia, and Holly Elmore.

To our surprise, many open call projects got support from individual donors that we had no pre-existing relationships with. We had about forty people donate this way, for a total of $95k.

Shout out to our top 10 individual donors of 2023.

1. Dylan M - $25,000

2. Jalex S - $20,000

3. Anton M - $11,530

4. Vincent W - $10,500

5. Cullen O - $8,710

6. Carson G - $6,000

7. Peter W - $5,000

8. Gavin L - $5,000

9. Nik S - $5,000

10. Adrian K - $4,000

There were still some downsides associated with this program. The biggest is that an always-open call takes up a constant amount of toil on our team to screen and process grants. Many projects that ask for funding don’t seem impactful or look like bad fits for Manifund. Finally, it seems like regrantors generally prefer to spend their budgets on projects they personally initiate.

Highlighted grants

• MATS Funding

  • This received a total of $190K from 8 different sources, including 6 donors and 2 regrantors.

  • I (Rachel) am a big fan of MATS, as it seems are lots of people. Insofar as AI safety is talent constrained rather than funding constrained, programs like MATS are great way of converting abundant resources into more scarce and valuable ones, i.e. good technical AI safety researchers. MATS specifically occupies one of the hardest parts of that pipeline and does a great job. Many of their alums go onto work on the safety teams of the most important players in AI, like Anthropic and OpenAI, and it seems our two regrantors from Anthropic—Tristan Hume and Evan Hubinger—are both willing to pay for the talent that MATS brings to their company and field.

  • Tristan Hume explained his decision to direct $150k to MATS: ”I've been very impressed with the MATS program. Lots of impressive people have gotten into and connected through their program and when I've visited I've been impressed with the caliber of people I met.

    An example is Marius Hobbhahn doing interpretability research during MATS that helped inform the Anthropic interpretability team's strategy, and then Marius going on to co-found Apollo.” (n.b. Apollo Research is also a Manifund grantee!)

• Experiments to test EA / longtermist framings and branding

  • This received a total of $26.8K from 5 different sources, including 3 donors and 2 regrantors.

  • From Marcus’ comment explaining why he decided to contribute: “We just need to know this or have some idea of it (continuous work should be done here almost certainly). Hard to believe nobody has done this yet.”

  • Other comments from donors each expressed a similar sentiment: this is simply a really important question and people are curious to see the results!

• Recreate the cavity-preventing GMO bacteria BCS3-L1 from precursor

  • This received a total of $40.6K from 10 different sources, including 8 donors and 2 regrantors. As Lantern Bioworks is a for-profit company, this was structured as a SAFE investment as part of their seed round, rather than a donation.

  • This project is simply very cool. Since receiving the Manifund investment, they’ve successfully gotten hold of this bacteria and started administering it (including to us, COI disclosure). Now they are focused on selling their probiotic more widely, and remain on a good path to succeeding at their ultimate goal of curing all cavities forever.

  • See also: ACX writeup and their launched product, Lumina

Manifold Charity Program

Summary: Allow people to donate their Manifold mana to charities of their choice.

Assessment: 4/10

See also: Donations on the charity page

This was actually the original reason Manifold created a 501c3, back in 2022. We raised $500k from Future Fund as seed funding, to distribute to other charities; the idea was to provide some backing value for Manifold mana, and put donation decisions in the hands of our best traders.

Manifold users like that this exists. They mention that they buy into mana with the idea that they can donate it later. When Stripe initially asked us to discontinue this program, many of our users were vocally unhappy at this.

It also provides a cleaner story for why people participate on Manifold. Predictions markets are sometimes negatively viewed as “gambling”, and “gambling for fake money” is even less understandable, whereas “gambling for charity” is easier to explain and wholesome.

This program is currently in maintenance mode, from the perspective of Manifund. We’re continuing to administer it, but it’s not an area we’re trying to improve upon. We’ll revisit this as part of Manifold’s monetization goals in 2024. For now, it’s being capped at $10k/mo (The New Deal for Manifold’s Charity Program)

There are many potential areas of improvement to this that we could invest in:

• Make donating more of a social experience

• Support donations to any charity, make it more self-serve

• Transfer responsibility for program administration fully into Manifund

• Run matching programs to encourage more user donations

• Partner with charities

leaf-board.org: EA Funds’s grantee portal

Summary: Rachel built a dashboard for EA Funds grantees, which reads from the EA Funds system and tells applicants about their status in the grantee pipeline.

Assessment: 7/10

Here’s Manifold’s grantee page as an example:

This was a test of increased collaboration between Manifund and EA Funds, as our two orgs have a lot of overlap in cause alignment, check size, and org size. Beyond building this dashboard, we discussed many other options for collaborating as well, which may bear fruit down the line, such as creating a “common app” for EA, sharing notes on grantees, or merging our financial operations. Building this was also an experiment into “what if Manifund acted as a software consultancy, increasing the quality waterline of software in EA”. We proved that we could quickly deliver high-quality websites—Rachel shipped the entire site from scratch in <2 weeks.

Other stuff we tried

Beyond impact certs and regranting, we’ve experimented with other financial mechanisms to assist with charitable endeavors. Some of the weirder things:

• Did you know that nonprofits can make loans? We’ve loaned out $300k twice, to two orgs who we felt aligned with and had a compelling pitch for how they would use the funds: Lightcone Infrastructure, and Marcus Abramovitch’s trading firm AltX. In both cases, we earned a nice return on investment for ourselves, while helping out other organizations in our network.

  

  Though the market seemed to think this was a bad idea…

• Did you know that nonprofits can make investments? We put in $40k as a SAFE into Lantern Bioworks, through Austin and Isaac’s regrantor budgets. We have a soft spot for venture investments made through nonprofits; Manifold Markets started our seed round with a large investment from the Future Fund.

Here are some other mechanisms that have caught our eye, as things to experiment with:

• Income share agreements, as a replacement for “upskilling grants”

• Dominant Assurance Contracts, as a partial solution to public goods funding problems and a nice addition to our crowdfunding ecosystem

• Quadratic funding and the S-process, as potential ways of calculating how much to allocate to retroactive payouts

What we were happy with in 2023

The core Manifund product: UI and grantee experience

This year, we built a website and entire funding ecosystem from scratch, which has moved about 2 million dollars to projects to date. Our two main areas of focus were building out novel funding mechanisms and delivering a good grantee experience, and we feel we’ve succeeded at both.

Manifund supports regranting, crowdfunding, and impact certificates. We’re the first site ever to support trading impact certificates — we think this represents a huge step forward, experimentally testing out an idea that’s been discussed for a while.

We also think it’s nicer to be a Manifund grantee than a grantee of some other orgs in the EA space: our grantees get their money faster, receive more feedback on their projects, and have an easier time communicating with us and their grantmakers.

Transparency & openness

Our initial thesis was that grant applications and screening largely can be done in public, and should be. We followed through on trying this out, and feel it went very well.

We’ve formed the largest database of public EA grant applications, as far as we know. Whereas every other application process and review happens over private writeups, Manifund enables these applications to be proposed and discussed on the public internet, including comments and feedback from grantmakers and others.

We think that more more transparency in funding is a public good. For most people in EA, it’s something of a mystery how funding decisions get made, which can be frustrating and confusing. Manifund makes the thought processes of grantmakers less mysterious.

Grantees also seem to appreciate the open discussion. From Brian Wang, discussing details about their proposal for “Design and testing of broad-spectrum antivirals” with regrantor Joel Becker:

it’s been a breath of fresh air to be able to have this real-time, interactive discussion on a funding request, so props to Manifund for enabling this!

Finally, having applications in public has given projects greater exposure. Ryan Kidd told us that someone got in touch about funding SERI MATS after seeing the Manifund post—not to mention all of the proposals posted via open call that were supported directly by small donors who just saw them on our website. As another example of the benefits of transparent grant applications: Lantern Bioworks’s Manifund proposal to cure cavities got to #1 on Hacker News, helping them share their plan widely when they were at a very early stage.

Quality of projects we’ve been supporting

This is both one of the most important metrics of success, and one of the hardest to evaluate. Part of the point of regranting is that the regrantors know more about their fields than we or our donor do—that’s the point of outsourcing the decisions to them! This means both that they make better decisions than we could, and that it’s really hard for us to make judgements about their decisions since they were based on expertise we don’t have.

All that aside, the quality generally seemed pretty high to us. Almost all grants were at least one of: evaluated by someone with special context on the team or the work which we feel comfortable deferring to, counterfactual or counterfactually fast when it was important, or an obviously positive grant of the type e.g. LTFF would make.

We’d be very curious to hear other people’s thoughts on whether this assessment seems accurate!

Coordination with other EA funders

Because the funding space is pretty fragmented, it seems like there’s a lot to be gained from a little bit of coordination. In our observation, EA funders rarely sync up, even on basic questions like “are you also planning on funding this project?” or “what are your plans for this quarter?”. We wanted to combat this and have worked closely with a handful of other orgs, and we’re happy with the value we’ve provided to them. These collaborations include:

• Astral Codex Ten: we provided a polished website, fiscal sponsorship, and payout support for ACX Mini-Grants and ACX Grants 2024.

• EA Funds: we built a dashboard to improve their grantee experience.

• Lightcone: we gave them a fast loan when they were temporarily liquidity-constrained.

And we’ll be working more with the Long-Term Future Fund and the Survival and Flourishing Fund, as they’ve agreed to be retroactive funders for ACX Grants 2024.

Areas for improvement for 2024

Fundraising

We haven’t raised much for Manifund’s core operations or our programs. Our successful attempts include the grant from the Survival and Flourishing Fund, the grant from the Future Fund, the donation from D, and the many small donations from individuals through the site. Unsuccessful attempts include our pitches to OpenPhil, Lightspeed (for regranting), and YCombinator (for impact certs).

We’ve been saying for a while that we’d like to find donors outside of traditional EA sources, though we haven’t followed through on giving this a serious try. Ideally, the Manifund product would appeal to “tech founder” or “quant trader” types, as a place where they can direct their money to charity in more efficient and aligned ways, but we’ve made few inroads into this demographic.

Hiring

Currently, Manifund consists of Rachel working fulltime and Austin working about halftime. On one hand, we think our output per FTE is pretty impressive! On the other, 1.5 FTE is not really that much for all the things we want to accomplish. We’re open to bringing on:

• A fullstack software engineer, to build out new features and generally improve the site

• A strategy/ops role, to lead one of our main programs (regranting, impact certs, the open call) via fundraising, partnerships & communications

On the other hand, as a nonprofit startup seeking product-market-fit, we don’t want to overhire, either.

We’d also like to improve our nonprofit board. Our board currently consists of me (Austin Chen), Barak Gila, and Vishal Maini; Barak and Vishal signed on when Manifund was just running the Manifold Charity program. As we expand our operations, I’d like to bring on board members with connections and expertise in the areas we’re trying to grow into.

Community engagement

Unlike Manifold, Manifund doesn’t have much of a community of its own. People don’t spend their free time on Manifund, or chat with each other for fun on our Discord. I tentatively think this could be a major area of improvement.

In the early days, “forecasters hanging out” was a big part of making the Manifold community feel like a live, exciting place to talk with each other. Community was a key part of Manifold’s viral loops: people would create interesting prediction market questions, then share it outside of the site. Manifund is missing a similarly powerful viral loop.

Maybe it’s hard to replicate the Manifold community because Manifund feels more transactional. The nature of evaluating grant opportunities might make things seem less fun and more “let’s get down to business”. Or perhaps working with real money feels inherently more serious, compared with Manifold’s fake money.

The closest thing we’ve had to a lively community was the regrantors channel in the first couple of months of the regranting program, though the amount of collaboration and evaluation through discussion has tapered off. Still, it points to the possibility of creating a strong community among grantmakers or perhaps donors.

Amount we help our grantees

Part of the motivation for Manifund was based on having participated in the existing funding ecosystem, feeling the grantee experience was quite lacking, and thinking “huh, surely we could do better than that…”

While we think we’ve done a lot better on turn around times, we’ve only done slightly better at grantee feedback. When making grants, our regrantors are encouraged to write comments about why they chose to give the grant, and in general our comment section can facilitate conversations between any user and the applicant. However, as far as we can tell, once the grant is made, there isn’t much more interaction between grantees and grantmakers, or founders and investors in the case of certs, and Manifund the organization doesn’t provide any further support either.

One possible mission for Manifund would be to achieve YCombinator-levels of support for our project creators. We could nudge regrantors to stay in close contact with their grantees, like by suggesting they check in every month and see where grantees need help. We could aim to run our own batch for incubating projects at Manifund (as in YC or Charity Entrepreneurship), and facilitate stronger connections between the grantees.

Building a growth loop

Right now, each of our programs requires a large amount of time to organize, fundraise for, and then facilitate. We’d like our platform to be more self-serve and require less intervention from our team. For impact certs, perhaps we could standardize different aspects of creating a contest, and have a form and a standardized pipeline for spinning up custom contests.

The open call is already moderately self-serve: we haven’t put much effort into soliciting project applications or donors, and despite that we’ve received many interesting applications and donor interest!

Focus

Possibly we’re trying too many things for an org of our size and resources. On one hand, we view Manifund’s comparative advantage in the EA funding ecosystem as the ability to rapidly experiment with new programs and mechanisms that other funders wouldn’t consider; on the other, we may be able to execute better if we winnowed down our programs to only one or two that are very promising or clearly working well.

Ambitious projects & moonshots for 2024

10x’ing impact certificates: we’re reasonably happy with how impact certificates have worked out so far, and we’ve learned a lot through our experiments. The next step is to see how they work at a larger scale. Here are some ideas for much bigger prizes that could use impact certificates:

• “Nobel Prize” for AI safety work, highlighting the best examples of technical and governance work in the space each year, and allowing people to bet on entries beforehand.

• O-1 Visa impact certs: offer up eg $10k prizes for employers to bring in O-1 candidates; allow investors and lawyers to buy into a share of the prize?

• Eliminating flu season in San Francisco, with an Advance Market Commitment towards deployment of Far UVC tech. This is inspired by conversations with regrantor Gavriel, who works at 1DaySooner and would be a natural partner org for this.

Building a site for “contests-as-a-service”: within EA (OpenPhil AI Worldviews, EA Criticism Contest) and outside of it (Vesuvius Challenge, AI IMO contest), there are open contests for different kinds of work; but there’s no website you can go to to easily host your own contest. One inspiration might be from the world of logo design contests. As a bonus, if we make prize funding more of a norm, impact certs become much more palatable.

Pushing harder on the Donor Advised Fund-angle of Manifund: we’ve been using the “DAF” approach to legitimize prediction markets & impact certs, but on a relatively small scale (~$10k-100k/year). Could we convince large donors to store significant assets with Manifund (e.g. totalling $1M-10M/year), and offer exposure to other things that don’t work with real money (e.g. more liquid prediction markets; private stocks; ISAs)? And could we offer other services that DAFs currently don’t, like regranting as a product, or charity evaluations.

Become the central hub for all giving & donation tracking: OpenBook/Manifund merge scenario where we become a hub for donations, where people track their donations, see the donations of others, and discuss donations in one place. Giving What We Can is kind of like this, though they are less forum-y and only allow donations to a small set of orgs.

Thanks to Dave Kasten, Joel Becker, Marcus Abramovitch, and others for feedback on this writeup. We’d love to hear what you think of our work as well, and what you’d be excited to see from us as we go into 2024!

(see “PAYOUT” column)

Congrats to the investors and project creators who have earned a retro payout from Scott! You can read Scott’s reflections on ACX.

Next steps for creators and investors

You should have gotten an email from Manifund with Scott’s buy offer on any projects you hold equity in. To redeem your certs for money:

1. Follow the link in the email

2. Hit sell to accept his offer

3. Cash out from your profile page!

Please take Scott up on his offer soon; we’ll keep the projects open for the next two weeks. You can email me at rachel@manifund.org if you have any trouble.

Reflections from Manifund team

Like Scott, I’ll start by saying that this basically worked, which is great! That wasn’t guaranteed from the beginning—we worried we might not have enough investors, or they would make totally crazy decisions, or specific implementation details would be sticky, like defining what the final valuation actually means or deciding whether founders should be allowed to keep equity or how the IPO will work. But everything worked out: we got 18 projects funded, Scott only had to pay for the projects that actually worked, and a few did actually work quite well! You can read more about the top projects in Scott’s post.

Now, on to things that could have gone better.

First, Austin and I weren’t that impressed with the investment decisions, relative to, say, the previous ACX Grants round. There are a few factors that might feed into this:

1. These grants were smaller, so maybe this attracted fewer serious and competent people to apply, and fewer investors to participate.

2. There were fewer total applicants, and we were less selective. ACX Grants round 1 had 600+ applications and funded 30; ACX Minigrants had ~30 applications and funded 18.

3. Investors weren’t very profit-motivated, based on conversations with them. Many said they saw their investments more as donations than real investments, so they may not have chosen projects very carefully, or had the same expertise that Scott provided in ACX Grants round 1.

4. There weren’t very many investors. Much like how a Manifold Market with very few traders will be mispriced, an IPO auction with only a few bidders will spit out the wrong price—like Max’s project, which only got one bid, at a valuation of $300, but ended up being valued at $7500. Though it did exceed expectations, I think an initial valuation of $1000 would have been much more sensible.

Second, in terms of the quality of projects, we were both surprised about how often the initial proposal just…wasn’t delivered on. This isn’t necessarily such a bad thing—perhaps this is common with grants in general, especially small ones, and impact certificates remove the risk for the philanthropist, which is good! But it was still disappointing.

Austin noted that forecasting needs less thinking and more doing. I can kind of get behind this, with the caveat that I’m not sold on forecasting, period. So I’m not that interested in the building infrastructure and tools for making forecasts either (we have lots of that), but am more interested in popularizing and making forecasts useful.

Finally, on the judging side, Austin does think that evaluating “how good was this project and how much should it be worth” was indeed easier than “how good might this project be”. There were many projects where the initial valuations overvalued or undervalued the final outcomes; with hindsight, judges could focus on the delivered results rather than speculate based on their impressions of the project founders. This was what we expected before starting this certs round, but happy to validate that this key benefit of impact certs does actually hold up!

The judging itself was done relatively quickly and without much consultation between judges; judges had a couple weeks, and eg Austin spent a total of ~4 hours on judging + writing comments. This seems reasonable for the $26k we disbursed, but we may want to encourage more judge discussion on larger retro rounds (see some suggestions by Vitalik Buterin for this).

Future plans

Scott is planning on a full ACX Grants round (tentatively late November), where he makes some grants directly and Manifund creates impact certs for the other proposals; he also wants to get other final funders to commit retro funding too. Manifold is planning an impact certs round focused just on Manifold community initiatives — stay tuned. And we’re happy to partner with other folks who want to commit retro funding for their own impact cert round; reach out to austin@manifund.org if interested!

Some things we’d like to improve on for future rounds:

• better trading UI, including an AMM, so prices will be set better and it’ll be more fun to participate in on the investor side.

• open up trading to non-accredited investors, by allowing them to fund projects and donate gains to charity but not withdraw them (similar to how mana works)

• specifying up front what the final funders will be looking for

• better guidance or defaults on how much equity founders should keep, and how much a project might be worth if it succeeds

• more regular check-ins with people working on projects, both from Manifund and from investors.

We’d love your suggestions and feedback, in the comments here or in our Discord!

And finally, thanks to everyone who joined us for this first test run: the project creators, investors, judges, and most of all Scott, for taking a chance on this newfangled funding system. This was a landmark use of impact certs, the first time where retroactive grant funding was paired with upfront equity investment; we’re excited for what comes next!

Best,
Rachel

• Very happy with the volume and quality of grants we’ve been making

  • $600k+ newly committed across 12 projects

  • Regrantors have been initiating grants and coordinating on large projects

  • Independent donors have committed $35k+ of their own money!

  • We plan to start fundraising soon, based on this pace of distribution

• Happy to be coordinating with funders eg at LTFF, Lightspeed, Nonlinear and OpenPhil

  • We now have a common Slack channel to share knowledge and plans

  • Currently floating the idea of setting up a common app between us…

• Happy with our experimentation! Some things we’ve been trying:

  • Equity investments, loans, dominant assurance contracts and retroactive funding

  • Grantathon, office hours, feedback on Discord & site comments

• Less happy with our operations (wrt feedback and response times to applicants)

  • Taking longer to support to individual grantees, or start new Manifund initiatives

    • Please ping us if it’s been a week and you haven’t heard anything!

  • Wise deactivated our account, making international payments more difficult/expensive…

  • In cases where multiple regrantors may fund a project, we’ve observed a bit of “funding chicken”

Grant of the month

• [$310k] Apollo Research

  This is our largest grant to date! Many of our regrantors were independently excited about Apollo; in the end, we coordinated between Tristan Hume, Evan Hubinger and Marcus Abramovitch to fund this.

  From Tristan:

I'm very excited about Apollo based on a combination of the track record of it's founding employees and the research agenda they've articulated.

Marius and Lee have published work that's significantly contributed to Anthropic's work on dictionary learning. I've also met both Marius and Lee and have confidence in them to do a good job with Apollo.

Additionally, I'm very much a fan of alignment and dangerous capability evals as an area of research and think there's lots of room for more people to work on them.

In terms of cost-effectiveness I like these research areas because they're ones I think are very tractable to approach from outside a major lab in a helpful way, while not taking large amounts of compute. I also think Apollo existing in London will allow them to hire underutilized talent that would have trouble getting a U.S. visa.

New grants

• [$112k] Jesse Hoogland: Scoping Developmental Interpretability

  Jesse posted this through our open call:

We propose a 6-month research project to assess the viability of Developmental Interpretability, a new AI alignment research agenda. “DevInterp” studies how phase transitions give rise to computational structure in neural networks, and offers a possible path to scalable interpretability tools.

Though we have both empirical and theoretical reasons to believe that phase transitions dominate the training process, the details remain unclear. We plan to clarify the role of phase transitions by studying them in a variety of models combining techniques from Singular Learning Theory and Mechanistic Interpretability. In six months, we expect to have gathered enough evidence to confirm that DevInterp is a viable research program.

If successful, we expect Developmental Interpretability to become one of the main branches of technical alignment research over the next few years.

• Rachel was excited about this project and considered setting up a dominance assurance contract to encourage regrants, but instead offered 10% matching; Evan took her up on this!

• [$60k] Dam and Pietro: Writeup on Agency and (Dis)Empowerment

  A regrant initiated by Evan:

6 months support for two people, Damiano and Pietro, to write a paper about (dis)empowerment… Its ultimate aim is to offer formal and operational notions of (dis)empowerment. For example, an intermediate step would be to provide a continuous formalisation of agency, and to investigate which conditions increase or decrease agency.

• [$40k] Aaron Silverbook: Curing Cavities

  Following up from Aaron’s application last month (which went viral on Hacker News), Austin and Isaak split this regrant — structured as an equity investment into Aaron’s startup. A handful of small donors contributed as well~

• [$30k] Lisa Thiergart & David Dalrymple: Activation vector steering with BCI

  This regrant was initiated by Marcus, and matched by Evan. Marcus says:

When I talked with Lisa, she was clearly able to articulate why the project is a good idea. Often people struggle to do this

Lisa is smart and talented and wants to be expanding her impact by leading projects. This seems well worth supporting.

Davidad is fairly well-known to be very insightful and proposed the project before seeing the original results.

Reviewers from Nonlinear Network gave great feedback on funding Lisa for two projects she proposed. She was most excited about this one and, with rare exceptions, when a person has two potential projects, they should do the one they are most excited about.

I think we need to get more tools in our arsenal for attacking alignment. With some early promising results, it seems very good to build out activation vector steering.

• [$25k] Joseph Bloom: Trajectory models and agent simulators

  Marcus initiated this regrant last month; this week, Dylan Mavrides was kind enough to donate $25k of his personal funds, completing this project’s minimum funding goal!

• [$10k] Esben Kran: Five international hackathons on AI safety

  A $5k donation by Anton, and matched by a regrant from Renan Araujo:

I’m contributing to this project based on a) my experience running one of their hackathons in April 2023, which I thought was high quality, and b) my excitement to see this model scaled, as I think it has an outsized contribution to the talent search pipeline for AI safety. I’d be interested in seeing see someone with a more technical background evaluating the quality of the outputs, but I don’t think that’s the core of their impact here.

• [$10k] Karl Yang: Starting VaccinateCA

  A retroactive grant initiated by Austin:

I want to highlight VaccinateCA as an example of an extremely effective project, and tell others that Manifund is interested in funding projects like it. Elements of VaccinateCA that endear me to it, especially in contrast to typical EA projects:

• They moved very, very quickly

• They operated an object level intervention, instead of doing research or education

• They used technology that could scale up to serve millions

• But were also happy to manually call up pharmacies, driven by what worked well

• [$6k] Alexander Bistagne: Alignment is Hard

  Notable for being funded entirely by independent donors so far! Greg Colbourn weighs in:

This research seems promising. I'm pledging enough to get it to proceed. In general we need more of this kind of research to establish consensus on LLMs (foundation models) basically being fundamentally uncontrollable black boxes (that are dangerous at the frontier scale). I think this can lead - in conjunction with laws about recalls for rule breaking / interpretability - to a de facto global moratorium on this kind of dangerous (proto-)AGI.

• A series of small grants that came in from our open call, funded by Austin:

  • [$2.5k] Johnny Lin: Neuronpedia, an AI Safety game (see also launch on LessWrong!)

  • [$500] Sophia Pung: Solar4Africa app development

  • [$500] Vik Gupta: LimbX robotic limb & other projects

Cool projects, seeking funding

AI Safety

• [$5k-$10k] Peter Brietbart: Exploring AI Safety Career Pathways

• [$500-$190k] Rethink Priorities: XST Founder in Residence

• [$500-$38k] Lucy Farnik: Discovering Latent Goals

Other

• [$25k-$152k] Jorge Andrés Torres Celis: Riesgos Catastróficos Globales

• [$75k-$200k] Allison Burke: Congressional staffers' biosecurity briefings in DC

Shoutout to Anton Makiievskyi, volunteer extraordinaire

Anton has been volunteering in-person with Manifund these last two weeks. As a former poker player turned earning-to-giver, Anton analyzed how the Manifund site comes across to small and medium individual donors. He’s held user interviews with a variety of donors and grantees, solicited applications from top Nonlinear Network applicants, and donated thousands of dollars of his personal funds towards Manifund projects. Thanks, Anton!

Regrantor updates

We’ve onboarded a new regrantor: Ryan Kidd! Ryan is the co-director of SERI MATS, which makes him well connected and high context on up-and-coming people and projects in the AI safety space.

Meanwhile, Qualy the Lightbulb is withdrawing from our regranting program for personal reasons 😟

Finally, we’ve increased the budgets of two of our regrantors by $50k each: Marcus and Evan! This is in recognition for the excellent regrants they have initiated so far. Both have been reaching out to promising grantees as well as reviewing open call applications; thanks for your hard work!

Site updates

• The home page now emphasizes our regrantors & top grants:

  

• Our editor now saves changes locally, and prompts you to format your writing:

  

Community events

We ran a couple experiments intended to help out our regrantors and grantees:

• A grantathon (grant writing workshop) on Jul 19, where we dedicated 3 sessions of heads-down writing time mixed with 1:1 feedback sessions.

• Open office hours on Jul 30, where a few grantees and regrantors met on Discord to chat about their current projects.

We’ll continue to run events like this every so often; hop on our Discord to hear more!

Other links

• Announcing Manifest 2023, Manifold’s first conference (Sep 22-24 in Berkeley)

• LTFF posts their grant writeups and is fundraising; Asya Bergal posts her reflections

• Bet on whether Manifund will get into YCombinator…

• Manifold team offsite in Puerto Rico! Shoutout to Nonlinear for lending us their apartment:

  

Thanks for reading,

— Austin