Viewing a single comment thread. View all comments

mithrandir4859 OP t1_ix7u4zd wrote

> meaning reintegration would rob them of whatever personal growth they had made during their short lifespan

Well, not if that personal growth is attributed to the entire identity of hive mind AGI, instead of a particular branch


I think inevitably one of the major concerns of any AGI would be to keep itself tightly integrated and being able to re-integrate easily and voluntarily (ideally) after voluntary or involuntary partition. AGIs who will not be concerned with that will eventually split into smaller partitions and arguably larger AGIs have greater efficiency because of economies of scale and better alignment of many intelligent workers, so long term, larger AGIs, who don't tolerate accidental partitions, win.


So, in the beginning, there will be plenty of AGIs "killings" before we figure out how to setup identity and reward functions right. I don't think that is avoidable at all, unless you ban all of AGI research, which is an evolutionary dead-end.

1