there’s only a runaway effect if creating intelligences is a linear problem: 2x as intelligent is 2x as hard. it is much more likely it is an exponentially hard problem.
2023-02-11: A similar argument, there may be diminishing returns to intelligence
For most problems in the universe, there are massive diminishing returns to intelligence, either because they are too easy or too hard. We are obsessed with the narrow band of things that some humans can do and others can’t, like graduate from college, or at the extremes what is feasible for a genius of 160 IQ but not a regular smart person at 120, like write a great novel or make a discovery in theoretical physics. But the category of things that either all humans can do or no humans can do is probably larger than the one of things that some humans can do and not others.
