Prompted by the recent troll post, I’ve been thinking about AI. Obviously we have our criticisms of both the AI hype manchildren and the AI doom manchildren (see title of the post. This is a Rationalist free post. Looking for it? Leave)
But looking at the AI doom guys with an open mind, sometimes it appear that they make a halfway decent argument that’s backed up by real results. This YouTube channel has been talking about the alignment problem for a while, and I think he probably is a bit of a Goodhart’s Law merchant (as in, by making a career out of measuring the dangers of AI, his alarmism is structural) so he should be taken with a grain of salt, it does feel pretty concerning that LLMs show inner misalignment and are masking their intentions (to anthropomorphize) under training vs deployment.
Now, I mainly think that these people are just extrapolating out all the problems with dumb LLMs and saying “yeah but if they were AGI it would become a real problem” and while that might be true if taking the premise at face value, the idea that AGI will ever happen is itself pretty questionable. The channel I linked has a video arguing that AGI safety is not a Pascal’s mugging, but I’m not convinced.
Thoughts? Does the commercialization of dumb AI make it a threat on a similar scale to hypothetical AGI? Is this all just a huge waste of time to think about?
I’m going to talk about the hard AI problem of rogue AI, not the obviously more pressing one of AI being used for bad things, because that can’t be stopped
So, Big Yud is mad and constantly full of nuclear Bay Area brainworm takes. but some of his arguments against alignment make sense. At least enough to engage with them and try to solve them, but bombing data centres et al is dumb and won’t solve the issue since most code is already so inefficient.
AI is not going to go FOOM in the way he suggested, basically because LLMs exist, which he thought were impossible as part of his arguments. He never considered we would have time to practice alignment with extremely limited AI at near human levels. And that’s kicked the core of his doom argument away.
I think UlyssesT and I once agreed that he is the dumbest smart guy alive and will probably become leftist after trying and failing at every tech nerd principal. If someone could keep him the hell away from the Bay Area he might normalise and stop writing weird shit.
For what it’s worth he also thinks Transformers have no chance at AGI, at least alone. And his thoughts on the dumb approaches used by AI companies are sound. But he is arguing in good faith and has probably thought about this more than anyone else. (Nobody even in the Yud cult except for a few weirdos ever took Roko’s Basilisk seriously. Yud deleted the post mostly because he saw that people would go weird about it, a rare example of him being exactly correct.)
As for everyone else…Altman doesn’t care and is using it for market capture. Anthropic probably does care but is more or less forced down the same path of ineffective safety that captures the market.
Ilya definitely cares but for all the wrong reasons, he’s massively pro Israel so he’ll never achieve a consistently aligned AI anyway, but I’d not want him working on this stuff.
But ultimately hard alignment is a secondary concern against dumb corporate HR managers trying to AI the compliance team.
I forgive you. The traditions of all Yud generations weigh like rational nightmares on the alignments of the sneerers.