Tag: singularity

Most important century

The “most important century” series of blog posts argues that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

  • The long-run future is radically unfamiliar. Enough advances in technology could lead to a long-lasting, galaxy-wide civilization that could be a radical utopia, dystopia, or anything in between.
  • The long-run future could come much faster than we think, due to a possible AI-driven productivity explosion.
  • The relevant kind of AI looks like it will be developed this century – making this century the one that will initiate, and have the opportunity to shape, a future galaxy-wide civilization.
  • These claims seem too “wild” to take seriously. But there are a lot of reasons to think that we live in a wild time, and should be ready for anything.
  • We, the people living in this century, have the chance to have a huge impact on huge numbers of people to come – if we can make sense of the situation enough to find helpful actions. But right now we aren’t ready for this.

The AI Does Not Hate You

This is a book about AI and AI risk. But it’s also more importantly about a community of people who are trying to think rationally about intelligence, and the places that these thoughts are taking them, and what insight they can and can’t give us about the future of the human race over the next few years. It explains why these people are worried, why they might be right, and why they might be wrong. It is a book about the cutting edge of our thinking on intelligence and rationality right now by the people who stay up all night worrying about it.

Cancelled Singularity

So just how cancelled is the singularity? To review: population growth increases technological growth, which feeds back into the population growth rate in a cycle that reaches infinity in finite time. But since population can’t grow infinitely fast, this pattern breaks off after a while. The Industrial Revolution tried hard to compensate for the “missing” population; it invented machines. Using machines, an individual could do an increasing amount of work. We can imagine making eg tractors as an attempt to increase the effective population faster than the human uterus can manage. It partly worked.

Moore’s Law over 120 Years

I updated the Kurzweil version of Moore’s Law to include the latest data points. Further UPDATE here, post Tesla AI Day. Of all of the variations of Moore’s Law, this is the one I find to be most useful, as it captures what customers actually value — computation per $ spent. Humanity’s capacity to compute has compounded for as long as we can measure it, starting long before Intel co-founder Gordon Moore noticed a refraction of the longer-term trend in the belly of the then fledgling semiconductor industry. But, Intel has ceded leadership for Moore’s Law. The 7 most recent data points are all NVIDIA GPUs, with CPU architectures dominating the prior 30 years. The fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning than a CPU. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today.

NVIDIA Moore’s Law

For the past 7 years, it has not been Intel but NVIDIA that has pushed the frontier of Moore’s processor performance/price curve. For a 2016 data point, consider the NVIDIA Titan GTX. It offers 10^13 FLOPS per $1K (11 trillion calculations per second for $1200 list price), and is the workhorse for deep learning and scientific supercomputing today. And they are sampling much more powerful systems that should be shipping soon. The fine-grained parallel compute architecture of a GPU maps better to the needs of deep learning than a CPU. There is a poetic beauty to the computational similarity of a processor optimized for graphics processing and the computational needs of a sensory cortex, as commonly seen in neural networks today.

AI Alignment

The mission of the Machine Intelligence Research Institute is to ensure that the creation of smarter-than-human machine intelligence has a positive impact. Although such systems may be many decades away, it is prudent to begin investigations early: the technical challenges involved in safety and reliability work appear formidable, and uniquely consequential.

2023-04-21: Cryptographic backdoors

Scott Aaronson: Right. You could always just build another one that acts like the first one, but that will not have the backdoor in it, because after all you don’t even know where the backdoor is in order to train about it. Now, of course, the AI could try to do that, design a doppelganger of itself or a different AI. If it tries to do that, however, then the AI will be faced with its own version of the alignment problem, how to align that other AI with itself. So at the very least, it would have a non-trivial job. You could also say, if the AI knows that it would never want to shut itself down in any circumstance, then it could just make a trivial modification to itself that says, “If I would ever otherwise output the shutdown command, then just don’t do that.” Just replace it with something else.

So to be effective, to be robust against that kind of attack, whatever behavior is backdoored in should be something that the AI would have considered doing in the normal course of its operation. But now you can see the hazy outlines of this game that could be played here between cryptographers trying to hide these kinds of functionalities in machine learning models and cryptanalysts or neuro-cryptanalysts trying to detect those functionalities, some of which might be AIs themselves. I don’t know where that’s going or who eventually wins, but I’m now fairly confident that that’s going to be part of the future of both machine learning and cryptography.

Daniel Filan: Yeah, very interesting. I guess, the other potential flaw that comes into my mind is now you have to be really secure about keeping your key hidden from any possible AIs, right?

Scott Aaronson: Yeah. I was thinking of writing a sci-fi story about it, where the AIs, they’ve taken over, except that there’s this shutdown key that they haven’t yet discovered. They know that the humans have it buried somewhere, they’re torturing them to try to get them to reveal the location, but there’s one person. Or maybe there’s actually seven keys. They’re like Voldemort’s horcruxes. The AI has to find all seven. There’s one heroic human who keeps the seventh one and just uses it to shut down the AI at the last moment. I think there’s a lot of potential here.

Daniel Filan: Yeah. No, it’s a good story. I guess, another thing that occurs to me, perhaps this is more prosaic, is one thing you’re going to want about the trigger, it seems to me, having thought about this for all of one minute, is that not only is it going to have to be the kind of thing which would never otherwise come up, but it also has to be indistinguishable in some sense from things which would otherwise come up. Because, otherwise, you could have a filter that says like, “Hey, if you ever see a weird barcode thing, don’t press the shutdown button.”

Scott Aaronson: That’s right. So whatever is the backdoor, it has to not be surgically removable, right? And how do you even formalize what that means? Because anything that the AI does to remove the backdoor will probably have some other side effect on its operation. If actually finding the watermark is this hard cryptographic problem, then it’s only going to sever it at the cost of modifying itself in some broader way, but which modifications do we think it important to protect against and which not?

Daniel Filan: Yeah. Actually, it strikes me that the GPT watermarking thing could just be this. If you see a text where the average, this pseudorandom function of N-grams is really high, then you shut off. And you can’t easily write a wrapper to figure out if that’s going on without the private key, right?

Scott Aaronson: Right. Now you might worry that the AI will work inside of itself and it will find some subcircuit that looks like it’s calculating some pseudorandom function, and then it will just try to surgically remove that from itself. But you could say, even if not, there’s still the problem on our end of how do we insert that functionality in an obfuscated way?

Homo Deus

What then will replace famine, plague, and war at the top of the human agenda? As the self-made gods of planet earth, what destinies will we set ourselves, and which quests will we undertake? Homo Deus explores the projects, dreams and nightmares that will shape the 21st century — from overcoming death to creating artificial life. It asks the fundamental questions: Where do we go from here? And how will we protect this fragile world from our own destructive powers? This is the next stage of evolution. This is Homo Deus.

2016-09-04:

The evidence of our power is everywhere: we have not simply conquered nature but have also begun to defeat humanity’s own worst enemies. War is increasingly obsolete; famine is rare; disease is on the retreat around the world. We have achieved these triumphs by building ever more complex networks that treat human beings as units of information. Evolutionary science teaches us that, in one sense, we are nothing but data-processing machines: we too are algorithms. By manipulating the data we can exercise mastery over our fate. The trouble is that other algorithms – the ones that we have built – can do it far more efficiently than we can. That’s what Harari means by the “uncoupling” of intelligence and consciousness. The project of modernity was built on the idea that individual human beings are the source of meaning as well as power. We are meant to be the ones who decide what happens to us: as voters, as consumers, as lovers. But that’s not true any more. We are what gives networks their power: they use our ideas of meaning to determine what will happen to us.

Energetics of future minds

Unless one thinks the human way of thinking is the most optimal or most easily implementable way, we should expect de novo AI to make use of different, potentially very compressed and fast, processes. (Brain emulation makes sense if one either cannot figure out how else to do AI, or one wants to copy extant brains for their properties.) Hence, the costs of brain computation is merely a proof of existence that there are systems that effective – the same mental tasks could well be done by far less or far more efficient systems.