AI 안전 논의의 뿌리는 사실 '섹스 컬트'라는 주장

I'm back, I missed you all. Since this is an important subject I know too much about, here's a 🧵 explaining that AI Safety Is Mostly A Sex Cult. I don't think these people should make policy. (1/?) (alternative title: Time For Some Cult Theory) There has been a great effort over many years to distance AI Safety discussion and policy from its origins and intellectual center, because its center is a sex cult started by a fan fiction author. If you want to be taken seriously you have to hide that. This is Eliezer Yudkowsky ("Yud"). We see him here meeting with Bernie Sanders to advise him on AI Policy and arguing about AI Policy in The AI Doc, now on Netflix. This is very senior OpenAI employee and unofficial OpenAI spokesman Roon. Roon's opinion of Yud is that 1) he is in and/or running a sex cult and 2) his opinions about AI are very important and should be taken seriously. Yud is the person who connected Shane Legg and Demis Hassabis to Peter Thiel so that they could secure funding to start DeepMind, which is now Google's AI research division. Here's Sam Altman's opinion of Yud. This is the schedule for the Future of Life Institute's conference on AI in Puerto Rico in January, 2015. Elon Musk would found OpenAI that December. Yudkowsky claims that somehow events at this conference caused that, and that he's upset about it, but this seems unlikely. Dario Amodei, current CEO of Anthropic, is here in 2013 identified as a "member of the Effective Altruism community" on the blog of Yud's non-profit. He argues with yud about the "paperclip maximizer". Anthropic papers still cite Yud on occasion, and the entire way we talk about AI uses words his non-profit popularized. "AI Alignment" was popularized by Yudkowsky's non-profit, which was the first venue to use this exact phrasing consistently. It is very hard to discuss AI at all without implicitly invoking Yud's ideas or the rat/ea scene. Coxon, who recently left Anthropic and positioned himself as a whistleblower, screenshotted here. I almost forgot this part: Here's Eliezer Yudkowsky in Time Magazine in 2023, advocating in favor of risking nuclear war over his hobby subject. The guy who tried to firebomb Sam Altman's house cited Yudkowsky as his primary inspiration. I have a recording of him talking about this laying around somewhere if anyone cares, but let's be real, who else is going to ineptly firebomb Sam Altman's house? I promise I will come back to the "sex cult" part, but that's actually the easy part to explain because they post about it constantly. What's difficult is getting past the extensive, complicated and well-funded wall of denial that this is the main event. This picture isn't really a joke, though Yudkowsky's followers call themselves "the rationality community" these days, externally they are called "rationalists", the followers "rationalism". Apparently they think this sounds less cultlike. I personally find the very long "we are not a religion explanation" either funny or insulting. Oh, just so I have cashed this check: The fan fiction that launched a thousand cults is Harry Potter and the Methods of Rationality. It is possibly the highest-impact single modern religious text, and its influence may not yet have peaked. Silly timeline. en.wikipedia.org Harry Potter and the Methods of Rationality - Wikipedia en.wikipedia.org Allegedly, "Effective Altruism" is a different thing and doesn't take marching orders from Yud. This is basically bullshit. The ideas and people heavily overlap, especially in San Francisco. I am also told that Anthropic is not an Effective Altruist company. This is 100% bullshit. That video with Bernie and Yud? Also in the room: Jeffrey Ladish (Palisade Research, Coefficient funded) and Daniel Kokotajlo (AI 2027 author, funded by Survival and Flourishing Fund). All "effective altruists", not "rationalists", but they all got in on that together. Where there is any separation between the two things, "rationalism" is team Yud and "effective altruism" is currently team Dario, and they're competing for attention and funding in that same space. The way they do this is extremely funny. Yudkowsky's most viral idea, the one that infects the most people and spreads the farthest, is that AI, if we continue to develop it, will inevitably 1) achieve superintelligence ("RSI"), 2) escape human control 2) kill everyone ("paperclip them", "doom") All arguments about AI Safety and AI Alignment are primarily about mitigating this exact scenario under his exact terms. Significant modification is relatively rare. In the broad subculture, this is implicitly what "safety" and "alignment" mean. People who think they're Effective Altruist and not Rationalist, but who are deeply concerned with "AI Safety", are generally implicitly or explicitly concerned with Yudkowsky's basic framework for loss of control and doom. This idea seems to be the most infectious because it activates the same neurosis that every apocalyptic or millenarian cult does. The world is going to end, and only I and my special group of people in the know can maybe stop it. (Also, you should give me money and have sex with me.) You can argue with them in any way you like but it barely matters. It's not, mostly, something people believe for either rational or empirical reasons, and they will simply go back to complaining about the apocalypse as soon as you stop paying attention. In person they are gathered mostly in various communes in Berkeley, California. They call their communes "group houses", maybe to make it sound less cultish. Most of the details of this are what you expect from "a commune in Berkeley". Everyone is polyamorous, everyone attends the same events, everyone lives in group homes together, everyone venerates the same few authors, and everyone is very worried about AI doom. They are all very contrarian together. Lots and lots of them work at AI companies and non-profits. The safety, red teaming and alignment teams seem especially afflicted, because that's the area you go on to worry about and try to work against AI doom, which you are sure is at least medium likely. The crown jewel of this whole thing, though, is Lighthaven. It's the former Rose Garden Inn in Berkeley, and it was bought by one of the rationalist charities in late 2022 with SBF's money, which was looted from FTX for them just before it collapsed. There are maybe five people who could have physically pushed through wiring money to a rationalist charity from FTX when it was collapsing, and one of them is Leopold Aschenbrenner. Leopold married one of the others, and she's Chief of Staff at Anthropic. Leopold is, allegedly, an effective altruist and not a rationalist, but you can see how this gets very difficult to keep straight. In what year were you one and not the other? Was your organization, at the time, financially tied directly to Yudkowsky or no? Lighthaven hosts the local Astral Codex Ten reading group with Scott Alexander, a reading group for The Sequences, which is essentially the Rationalist Old Testament, a conference for AI people called The Curve, an EA conference called EAGxBerkeley, MATS, aka "Machine Alignment, Transparency, and Security or ML Alignment & Theory Scholars" and SlutCon, organized by Aella. Gretta, pictured here, is Eliezer Yudkowsky's primary partner, and draws a salary of about 200k from his nonprofit as his assistant while also holding this event at LightHaven. She has no prior qualifications for any job at his nonprofit One hopes no illegal sex work happened at SlutCon, and the presence of Eliezer's partner as an organizer is just fun and quirky. I also don't know that any of the paid orgies, which Gretta also organizes, ever happened at LightHaven. I do note that Lighthaven seems desperate for money. Lighthaven's facilities manager is Ronn