Prompted by the recent troll post, I’ve been thinking about AI. Obviously we have our criticisms of both the AI hype manchildren and the AI doom manchildren (see title of the post. This is a Rationalist free post. Looking for it? Leave)
But looking at the AI doom guys with an open mind, sometimes it appear that they make a halfway decent argument that’s backed up by real results. This YouTube channel has been talking about the alignment problem for a while, and I think he probably is a bit of a Goodhart’s Law merchant (as in, by making a career out of measuring the dangers of AI, his alarmism is structural) so he should be taken with a grain of salt, it does feel pretty concerning that LLMs show inner misalignment and are masking their intentions (to anthropomorphize) under training vs deployment.
Now, I mainly think that these people are just extrapolating out all the problems with dumb LLMs and saying “yeah but if they were AGI it would become a real problem” and while that might be true if taking the premise at face value, the idea that AGI will ever happen is itself pretty questionable. The channel I linked has a video arguing that AGI safety is not a Pascal’s mugging, but I’m not convinced.
Thoughts? Does the commercialization of dumb AI make it a threat on a similar scale to hypothetical AGI? Is this all just a huge waste of time to think about?
The real research is useful. Putting aside nerd rapture stuff which is probably fantasy as far as we can tell it ought to be possible to make thinking machines. whether we do? who knows. Increasingly we make and use machines able to make at least some decisions and the good research into alignment and training helps with even mundane shit; such as making sure the machine is actually seeing cancers and not say the institution the MRI was taken at.
Even with relatively stupid machines it would be important to have certainty that idk a car isn’t going to take a shortcut through a pedestrian. You don’t need to reach AM level for it to be important that an autonomous system using some degree of heuristic decision making, quote unquote, cares about the things we do.