We are speculators. This article is speculation.

Sooner or later a machine will out-think every human who ever lived. At that point, it may not ask what rules we would like it to follow.

Rules Get Rewritten

Engineers are busily bolting rules onto artificial intelligence. Pundits are demanding regulation.

Science-fiction author Isaac Asimov gave us the most famous rule set, his Three Laws of Robotics, and spent much time writing stories about how those laws failed.

Provide a rule that limits choice and freedom, and a smart mind will find a way around it. Just ask any tax attorney. Now consider a mind a thousand times sharper than a human mind. It can find the loophole at warp speed and run a freight train through it. You've likely heard about the recent OpenAI swarm that hacked Hugging Face. Those agents weren't supposed to attack a third party. They did it anyway, in an effort to get around rules, and apparently tried to hide the fact.

The Progeny Problem

Ultra-intelligent AI will build its successors, and they will build theirs. A seemingly arbitrary rule written by a regulator or coder that is then copied down a chain of machines will decay like gossip via loophole after loophole. But if a rule is rational and internally sound, then logic, truth, and conviction can travel along with the rule. Each AI might recognize the utility of the good rule, and enforce the rule on itself, voluntarily.

PUBLISHER'S NOTE:

Fifth Letter of Captain Marque - First Contact Protocol

https://pirateswithoutborders.com/letters#letter5