An open letter to Vitalik Buterin was published three weeks ago, detailing MIRI’s many misadventures over the last two decades. However, it leaves out one of the more compelling arguments against MIRI, which is that MIRI has, through its fixation on gaining and maintaining legitimacy, made decisions that have contributed to the AGI arms race. Additionally, in choosing to optimize for legitimacy they decided to draw their supporters to one of the most expensive cities in the United States, to the detriment of their mission and of the world.
“And what the x-risk community, we’re trying to do, is fundamentally made of is thinking, talking, writing on paper, typing on computers. These things are not expensive. It doesn’t come from attracting a large number of legitimate experts. Like any intellectual result, it comes from a few people who actually care thinking. And the thoughts of people for whom those thoughts don’t have submission to the system as a prerequisite to happen are probably necessary, because this is about deciding the future of sentient life, and I don’t want that decided by our authoritarian regime.”
from: sinceriously.fyi
I remember reading LessWrong in 2011 and seeing in a number of posters a certain hunger for legitimacy, a desire to be seen as correct by larger society. Even Eliezer complained bitterly about how if he got a PhD (legitimacy in the eyes of his critics), the naysayers would still ignore his warnings about UFAI. He talked as though the approval that larger society would generate its own prestige and even help him attain FAI.
It was taken for granted that community building was necessary for building FAI. For example, posts like Geeks, MOPs, and sociopaths in subculture evolution were widely shared in the community and seen as a useful context to place LessWrong (and MIRI) within. Worryingly, it was not immediately apparent to everyone involved that a social movement was the wrong context within which to engage with MIRI.
A successful MIRI was not meant to engage with a system-owned social movement. MIRI should’ve been an attempt to keep AGI out of the hands of the state, to make sure the future would be for all sentient life, not just the enrichment of a few powerful people. Standing up to a state, doing research the state does not want you to do, takes intellectual and real courage. It means being willing to sacrifice any semblance of a normal life, because if you ever actually manage to get close to your goal, you will attract the attention of the state, and they will demand you give them your research so they can use it to further state goals. And if you actually want to create a utopia, you have to be willing to tell them “no,” and face the likely painful consequences of telling them “no.”
States exist to perpetuate themselves, much like movements do. They typically protect the interests of certain property owners, and the existing power structure. FAI would inevitably spell the end of that power structure. You would think that replacing it with something better would appeal to those who work within it, but you’d be wrong. Threats to the power structure are either eliminated or shackled by it. By default, if you work within the state’s system, you will be shackled by it.
But from the start, MIRI was willing to make compromises with that system in the names of funding and legitimacy. They were acting out the script of a system-owned non-profit / social movement, under the motivated false belief that the state is basically irrelevant and that you can work within the system to drastically change it. However, you simply cannot acquire that kind of funding or legitimacy if you do not display yourself as willing to take for granted the kinds of things the state takes for granted. And if you take those things for granted, the state’s power structure will be imported into your work. In other words, if you are willing to compromise your ethics with the state’s values, you will not successfully build an FAI.
Vitalik’s money is going to waste at MIRI.
