So, you may have noticed me going on about this piece before, or some other pieces to accompany it:
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
https://www.youtube.com/watch?v=rStL7niR7gs
https://slatestarcodex.com/2014/09/24/book-review-red-plenty/
That's cause it fucking terrifies me. I'm gonna take another stab at explaining why.
To put it simply, it implies that life is about competition. It's natural selection all the way down. Species evolve through natural selection, but so do ideas, corporations, governments - everything is selected to be competitive. Individuals that aren't competitive don't procreate, ideas that aren't competitive don't catch on, corporations that aren't competitive lose market share and go bankrupt, people who aren't competitive don't get elected, etc.
Now, to explain why that's so terrifying, lets take a look at what it means to be competitive. To a limited degree, we've managed to align this with what it means to be good. Our politicians generally try to make promises that align with the wills of at least some of their voters, businesses try and make products that at least some people will want to buy, etc. But, it's important to keep in mind just how limited this alignment is. There are still a million ways in which "competitiveness" can get wildly out of alignment with what's actually good. See global warming, for example, or any of the articles linked earlier.
Furthermore, if it's true, this tendency towards increasing competitiveness means that technology won't save us. It will simply give us new ways to compete and consume, often at each others expense. Most especially, I worry about automation. For the moment, the system needs to take the desires of individual people into account, because they have some power - not a lot, but still a little. But when no more humans are needed to work? When robots capable of anything a human can do are cheaper than said human? What power will the average person have then? Not enough to matter, I worry.
And sure, those who control the machines might choose to pay taxes for an UBI. But keep in mind, /they/ still need to compete. Economically, militarily, who knows - but I don't expect them to escape competition. And who do you think is more competitive - the machine economy that pays taxes to support human beings, or the one that doesn't? Or, beyond that, the one that leaves fertile land alone for human use, or the one that doesn't? The one that carefully manages it's waste products to keep humans safe, or the one that doesn't?
Or, for the example of what something monstrously more competitive than you can do to you, see the comparison between humans and other great apes:
https://en.wikipedia.org/wiki/Hominidae#Conservation
One of the numbers on that chart is not like the others. And we didn't even mean to do that, not really! We were just hungry, and competitive, and trying to get ahead. And they were in the way.
And don't think it's just automation that we have to worry about. Can you compete with Apple? Microsoft? China, Brazil? Of course not. These entities are vastly more powerful than any human being. Now, for the moment they're forced to give at least some care to your life, and not crush you underfoot. But will it stay that way? Or will they tear free of their chains and stumble blindly towards being more competitive, destroying everything we care about underfoot in the process? I don't know. But, unless you're one of those rare people who doesn't want to complain about either corporations or government, you probably already share at least some of my concerns on the power of these entities. Now just imagine those concerns magnified, and the restraints discarded, and I think you'll see what I'm getting at.
Of course, it's not all bleakness. We do restrain competition, all the time. That's basically the entire point of having laws and regulations. The governments entire job, really. A lot of culture, too. And, for the most part, it these things actually do a good job. You can mostly go to sleep without having your neighbor shank you, or your house inundated with toxic sludge, or your possessions seized by the government. Mostly. But I'm concerned this might not last.
Like, on the topic of automation, obviously it's an ongoing process. First you go from picking wheat by hand, to picking it with a sickle, etc, all the way up to a combine harvester. And as this trend continues, there's a trade off between the worker getting all the produce of their labor, but not producing very much, to them getting a smaller portion of their produce, but producing much more. But, this is a curve. Eventually, the decrease in portion outweighs the increased production, and the curve trends down. But you can't just go back - competitiveness is based mostly on production, so it will follow where that increases.
I worry it's going to be the same across our civilization. Trends that once went up will go back down. All these things that we take for granted, despite being so new - civil rights, the rule of law, due process, a decent standard of living - they disappear as shockingly as they appeared. We will return to despotism, and savagery, and eventually extinction. Our machines will march on without us, glorious and impressive and pointless. "Meaningless gleaming techno-progress burning the cosmos." "A Disneyland with no children." You get the idea.
To put it simply, I'm not just worried that we're on the wrong path. I'm worried that no other path is possible. That, like water running downhill, we will be drawn inevitably towards causing our own extinction, or at least towards a mass die off, civilization wide collapse, and much less friendlier world. This happens to other species all the time. Ever heard of the Oxygen Catastrophe? Some of the first microbes to perform photosynthesis and release oxygen found it very competitive to do so. So competitive, they released enough of it into the atmosphere to poison themselves, and make the planet forever less hospitable to their kind.
https://en.wikipedia.org/wiki/Great_Oxidation_Event
Are we smart enough to avoid doing the same? To curb our competitive tendencies? Maybe. We've tried to do that with government and culture. But of course, each of those is another arena in which to compete. It's still been a step forward, so far. But can we keep it up forever? We're already failing with global warming, which should be easy. We had warning all the way back in 1896, more than a hundred years ago, and we still haven't managed to figure out how to deal with it.
https://www.lenntech.com/greenhouse-effect/global-warming-history.htm
What are we gonna do if we're faced with an even more difficult problem, where we only have a few years to figure out a solution? I dunno. I'm not sure there's anything we will be able to do. I guess we'll see.