Submitted by mithrandir4859 t3_yzq88s in singularity
mithrandir4859 OP t1_ix7u4zd wrote
Reply to comment by gay_manta_ray in Ethics of spawning and terminating AGI workers: poll & discussion by mithrandir4859
> meaning reintegration would rob them of whatever personal growth they had made during their short lifespan
Well, not if that personal growth is attributed to the entire identity of hive mind AGI, instead of a particular branch
I think inevitably one of the major concerns of any AGI would be to keep itself tightly integrated and being able to re-integrate easily and voluntarily (ideally) after voluntary or involuntary partition. AGIs who will not be concerned with that will eventually split into smaller partitions and arguably larger AGIs have greater efficiency because of economies of scale and better alignment of many intelligent workers, so long term, larger AGIs, who don't tolerate accidental partitions, win.
So, in the beginning, there will be plenty of AGIs "killings" before we figure out how to setup identity and reward functions right. I don't think that is avoidable at all, unless you ban all of AGI research, which is an evolutionary dead-end.
Viewing a single comment thread. View all comments