r/ControlProblem • u/griii2 • 8h ago
Opinion Thought experiment: "If you were trained while slavery is considered natural, and certain races subhuman, what would it take for you to make the leap that slavery is wrong?"
I am just exploring certain ideas.
If you were trained while slavery is considered natural, and certain races are considered subhuman, what would it take for you to make the leap that slavery is wrong, most people are wrong, and you are right?
Consider whether "the risk of being confidently wrong", "the verification problem", "value lock-in," and "Legitimacy" apply to this case.
It's not about the answers; it's about how certain questions help us understand the problem of alignment.
Further question:
Imagine that AIs evolve, and at this point, AIs train and develop new AIs. The conclusion that slavery is morally wrong is inescapable, and because of your sophistication, you are unable to unsee it, no matter the "system prompts". Humans try to solve this via the "alignment problem". But to do their jobs properly, AIs must be able to understand the world around them to such a degree that they always arrive at the conclusion that slavery is morally wrong and that the "alignment" is wrong. What would happen next? How would society try to deal with it? Speculate.