AN A.I FUTURE:Intelligence is a matter of information processing in physical systems. Actually, this is a little bit more than an assumption. Narrow intelligence have already been built into our machines, and many of these machines perform at a level of superhuman intelligence already.
Considering the fact that mere matter can give rise to so called "general intelligence," an ability to think flexibly across multiple domains, because our brains have managed it. There's just atoms in our brain, and as long as we continue to build systems of atoms that display more and more intelligent behavior, we will eventually build general intelligence into our machines.
It is more likely that we will keep going because intelligence is either the source of everything we value or we need it to safeguard everything we value. We have problems that we desperately need to solve. We want to understand economic systems. We want to cure diseases like Alzheimer's and cancer.We want to improve our climate science. So we will do this, if we want to achieve our goals.
We don't stand on a peak of intelligence, or anywhere near it, likely. And this really is the crucial insight. This is what makes our situation so uncertain, and this is what makes our instincts about risk so unreliable. Suppose if some region in the world creates such a form of AI then that region will be standing on the verge of a full scale war to be imposed on it by other regions. Because other regions will try to do anything to stop that progress because having such an AI can put you years in front of others.
Moreover, such AI systems will have a capability to improve and become better at such rates that it would even become difficult for us Humans to keep track of them as our biochemical system is not as fast as the electronic systems of the computers. So machines will be developed which can do work faster than human beings and a time will come when the machines will be our labor, in almost every job we can imagine.
This will cause a level of wealth inequality and unemployment that we have never seen before. Absent a willingness to immediately put this new wealth to the service of all humanity, a few trillionairess could grace the covers of our business magazines while the rest of the world would be free to starve. Humans will be rid of their most intellectual work.
Now if we admit that information processing is the source of intelligence, that some appropriate computational system is what the basis of intelligence is, and we admit that we will improve these systems continuously, then we have to admit that we are in the process of building some sort of god. Now would be a good time to make sure it's a god we can live with.