Not many people disagree that we can eventually create artificial general intelligence. But quite a few believe something along the lines of a quote by Edsger Dijkstra, "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."
There seem to be more opinions than there are people arguing :-)
I pretty much exhausted my knowledge of what "those" people believe. I am pretty confident that all important qualities of what it means to be human can be abstracted and emulated by various substrates.
I just assign a small
probability to the possibility that consciousness is a direct interaction with the physical world and that the idea of computing qualia is similar to the idea of trying to extinguish a physical fire with simulated water. In other words, everyone who is certain
that consciousness is computable is overconfident and does not realize that we still don't know what consciousness actually is
The whole problem here reminds me of a cherished idea hold by reductionists. That a perfect copy of something would be of equal value. Which is wrong, since it assumes that an object is fundamentally separable from the environment it is embedded in.
Imagine, for example, I had access to an advanced molecular assembler to create a perfect copy the Mona Lisa and would subsequently destroy the old one in the process. It would still lose a lot of value. That is because many people not only value the molecular setup of things but also their causal history, what transformations things underwent.
Personally I wouldn't care if I was disassembled and reassembled somewhere else. If that was a safe and efficient way of travel then I would do it. But I would care if that happened to some sort of artifact I value. Not only because it might lose some of its value in the eyes of other people but also because I personally value its causal history to be unaffected by certain transformations.
So in what sense would a perfect copy of the Mona Lisa be the same? In every sense except that it was copied. And if you care about that quality then a perfect copy is not the same, it is merely a perfect copy.
Here is another example. Imagine there was a human colony in another star system. After an initial exploration drone set up a communication node and a molecular assembler on a suitable planet, all other equipment and all humans were transmitted digitally an locally reassembled.
Now imagine such a colony would either receive a copy of the Venus figurine digitally transmitted and reassembled or by means of a craft capable of interstellar travel. If you don't perceive there to be a difference then you simply don't share my values. But consider how much resources, including time, it took to accomplish the relocation in the latter case.
Part of the value of the value of an object is the knowledge of its spacetime trajectory. An atomically identical copy of the same object that was digitally transmitted and printed out for me by my molecular assembler is very different. Its spacetime trajectory is different, it is artificial.
Which is similar to drinking Champagne and sparkling wine that tastes exactly the same. The first is valued because while drinking it I am aware of its spacetime trajectory, the resources it took to create it and where it originally came from and how it got here.
The value of something can encompass more than its molecular setup. There might be many sorts of sparkling wines that taste just like Champagne. But if you claim that simply because they taste like Champagne they are Champagne, then you are missing what it is that people actually value.