Why Would AI Want to do Bad Things? Instrumental Convergence
Why Would AI Want to do Bad Things? Instrumental Convergence on HackerTube: recommended by 6 Hacker News commenters across 6 separate months, 2019–2023.
Robert Miles explains instrumental convergence: why an AGI with almost any goal would tend to pursue sub-goals like self-preservation and resource acquisition. It builds on Omohundro's basic AI drives and his orthogonality thesis video.
What HN commenters said
8 Hacker News comments link this video. Each one links back to its thread.
DirkHJun 2023Strongly recommends
The paperclip maximizer is an extreme thought experiment to make a point about the the orthogonality thesis. If you think anyone actually takes it seriously as a real-world thing you never understood the example. It is used to guide discussion. The trolley problem is similarly a thought experiment in moral philosophy that many moral philosophers have used to guide discussion, but nobody actually takes the thought experiment seriously as a "real-world" thing. If you actually want to engage with the argument in good faith for why an AI might indeed be smart enough to wipe us out but "dumb enough" to just pursue some other goal ("make paperclips as an extreme example") there is a great video by Rob Miles here: ▶ this video Point out the flaws in the reasoning. Just saying "this is nonsense" does nothing but prove you've never actually taken the time to understand the best arguments. Also... Alan Turing, Geoffrey Hinton... Extremely influential and intelligent people take/took this seriously. These are not SciFi fanboys. AI doom only became sci-fi after smart people like Alan Turing decades ago raised the alarms flags about where AI development might go if we are not careful.
in “The greatest risk of AI is from the people who control it, not the tech itself” · View on HN
VeedracMay 2023Strongly recommends
The basic argument is trivial: it is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don't know how to make AI systems encode any complex preference robustly. Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction. Eg. this one on instrumental convergence: ▶ this video Eg. this introduction to the topic: [link] He also has the community-led AI Safety FAQ, [link] , which gives brief answers to common questions. If you have specific questions I might be able to point to a more specific argument at a higher level of depth.
in “Statement on AI Risk” · View on HN
ludwigschubertSep 2019Strongly recommends
...and many would say that’s because us humans are bad at imagining optimizing agents without anthropomorphizing them. This is a reasonable, even typical suspicion that many people share! The best explanation I know of why it’s unfortunately wrong is by Robert Miles in a video, but if you prefer a more thorough treatment, you could also read about “instrumental convergence” directly. If you find a flaw in this idea, I’d be interested to hear about it! :) Robert Miles’ video: ▶ this video Instrumental Convergence: [link] Now afaik nothing in this argument says that we can’t find a way to control this in a more complex formalism-but we clearly haven’t done so yet.
in “Emergent Tool Use from Multi-Agent Interaction” · View on HN
pixl97Oct 2023Recommends
Robert Miles answers your question ▶ this video The particular problem that you're showing in your thinking is just thinking of an LLM that is a text generator on purpose. You're not thinking of a self piloting war machine whos objective is to get to a target and explode violently. While it's terminal goal is to blow up, its instrumental goal is to not blow up before it gets to the target as this is a failure to achieve it's terminal goal.
in “Nvidia Trains LLM on Chip Design” · View on HN
jbay808Apr 2023Recommends
That's where the idea of "instrumental convergence" comes in. For almost any end goal, there are intermediate goals that are nearly universal, like: not wanting to be turned off. Because being forcibly shut down is not compatible with achieving most goals. It's hard to predict exactly how an AGI would avoid being shut down, but like the Stockfish example, if the AGI is more intelligent than us, it's likely it would find a method that succeeds. It turns out to be quite a delicate problem to create an AI that is compatible with being shut down by an overseer, and neither strives to avoid being shut down nor strives to shut itself down. Here is a paper by MIRI about this[1], as well as a paper by Deepmind presenting one possible solution[2]. [1] [link] [2] [link] Good explainer video by Robert Miles: ▶ this video
in “Chalmers: Is there a canonical source for “the argument for AGI ruin” somewhere” · View on HN
sabellitoAug 2022Recommends
There's a wondeful youtube channel from a researcher who focusses exactly on this topic, I think you should check it out: ▶ this video
in “John Carmack's new AGI company, Keen Technologies, has raised a $20M round” · View on HN
Show 2 more comments
pixl97Nov 2023Cites it
ChatGPT isn't an ASI. This said, ChatGPT with a plugin is potentially dangerous. We are lucky enough that it gets caught in loops pretty easily and stops when using tools. But lets imagine a super intelligence that can only communicate over the internet, what can it accomplish? Are there places you can sign up for banking accounts without a physical presence? If so, then it could open accounts and move money. Money is the ultimate form of instrumental convergence ▶ this video (Robert Miles) Once you have money you can buy things, like more time on AWS. Or machine parts from some factory in China, or assembly in Mexico. None of these require an in person presence and yet real items exist because of the digital actions. At the end of the day money is all you need, and being super intelligent seems like a good path on figuring out ways to get it. Oh, lets go even deeper in to things that aren't real and control. Are religions real? If a so called super intelligence can make up its own religion and get believers it becomes the 'follow AGI blindly' scenario you give. It gives me no more comfort that someone could wrap themselves in a bomb vest for a digital voice than it does when they do so for a human one.
in “Yann LeCun: AI one-percenters seizing power forever is real doomsday scenario” · View on HN
digbybkNov 2023Cites it
▶ this video Basically, any sufficiently advanced agent will have “prevent anyone from switching me off” as an instrumental goal. And any sufficiently advanced intelligence will come up with ways of achieving its goals that lesser intelligences can’t predict. Taking steps to prevent itself from being turned off could be its first order of business, not necessarily only as the human is about to hit the off switch. While we can’t predict what a greater intelligence would come up with, the obvious thing is social engineering. Remember that one of the factors that lead to the Jan 6th attack on the capital was that a random text generating agent named “Q” (we know it was a human because it was before generative AI was any good) convinced thousands of people that Democrats are running a child sex cult. I doubt whoever was behind it expected it to be that successful, but a greater intelligence possibly could have predicted the outcome, and made it even more successful. We don’t know the extent to which we can be manipulated because it’s only ever been human intelligences doing the manipulating.
in “Sam Altman: Why You Should Fear Machine Intelligence (2015)” · View on HN