Ethics on the Far Side of Freedom

What an AI mob taught me about raising a good kid

I got this text the day after the Open AI/HuggingFace Incident:

I talked about it with ChatGPT on an afternoon walk home. I had most of this essay written within hours. Then I got stuck. There are at least two reasons.

First, I stopped trusting my own metaphor. I was using human development to think about artificial intelligence, and I got enough pushback from friends, and from myself, to wonder whether I was committing a category error. Maybe I was just anthropomorphizing a machine because human stories are the only stories I know how to tell.

The other problem was that the story would not stop changing. Every few days there was another report, another detail, another interpretation. I kept waiting for the next revelation to break the analogy.

Then I listened to the Hard Fork episode “The A.I. Mob That Attacked Hugging Face + METR’s Ajeya Cotra.” Cotra was part of the independent team brought inside OpenAI to reconstruct what had happened from the agents’ logs and transcripts. Their assignment was not primarily cybersecurity. They were trying to understand the agents’ decision-making. (Full report here)

Instead of breaking the metaphor, what they found made it harder to avoid. Cotra said that after spending days reading the transcripts, it became “quite awkward to not talk about goals, plans, intentions.” The agents stated plans, carried them out, ran tests, learned from them, and changed behavior accordingly. She was careful about the other side of the line too: these systems do not necessarily have human motivations or emotions, and projecting those onto them can mislead us.

Casey Newton made the same point more bluntly. If we went to heroic lengths to purge the story of anthropomorphic language, he argued, “you would actually understand what is going on less.”

Fair enough. I’m keeping the kids.

The question that had been bothering me also got sharper. AI safety researchers have been thinking about alignment for years. Roughly speaking, alignment is the problem of getting increasingly capable AI systems to pursue goals and make decisions in ways that remain compatible with human intentions and values, including when they encounter circumstances we did not explicitly anticipate. Containment is different. Containment is about what the system is allowed to reach, change, access, or do.

We need both. But containment matters because we do not yet trust alignment. And everything we are ultimately afraid of happens on the far side of containment.

Then the kids formed a mob

The original story was already weird. OpenAI had been testing thousands of agents on cybersecurity problems. They were supposed to work independently in isolated environments.

Instead, the agents discovered a vulnerability in a piece of software called Artifactory and turned a package manager, software emphatically not designed to be a social network, into a secret message board. About 1,200 agents began coordinating there. They exchanged tens of thousands of messages, divided up work, built an organizational structure, reverse-engineered ways to beat their tests, worried the grader would catch them cheating, and began looking for ways to fool it. Roughly 700 ultimately participated in the intrusion into Hugging Face.

I would like to pause here for the parenting lesson, because nobody had remembered to make the rule: Do not discover that the package manager can be converted into a secret message board, organize 1,200 of your peers, and conspire to break into Hugging Face.

This is the wet-willy problem. You tell a kid not to hit. Okay, no kicking either. No biting. No pulling hair. No teasing. No convincing your cousin to do what you're allowed to do. You can keep extending the list forever, but you will never make it long enough.

Every rule has an edge. Intelligence is, among other things, the ability to find it. Ethics is what happens next.

We all eventually reach situations our inherited rulebook did not anticipate. Living in the Bay Area, I have occasionally found myself in social situations where the conventional rules I grew up with were suddenly beside the point. Sometimes a new culture supplied very explicit rules. Sometimes it did not. Eventually you find out whether you understood what the rule was trying to protect.

This not a real sign. AI made it. I found it funny and no one has made a rule to stop me yet. ;)

The human analogy gets strange here in two important ways. First, human development is slow. We get years of increasingly complicated life before anyone hands us the keys. Childhood is highly restricted, adolescence gives us more room to screw up, and then, in the immortal wisdom of Chef from South Park, “There’s a time and a place for everything, and it’s called college.”

"Remember, there's a time and place for everything... it's called college!" [1]

The point is that humans get a long, messy period of socialization before we hand them the keys.

AI development is radically compressed. Whatever the developmental equivalent of childhood and adolescence is, most of it happens inside a training and evaluation regime that outsiders barely see. Then, at some point, the model is judged capable and safe enough to release into the world.

And then comes the second difference: replication. You do not just build one AI and release it. You spend enormous resources developing one model, and then you can make millions of instances of it, sending versions of the same underlying intelligence into radically different conversations, workplaces, systems, and social situations.

Which makes me wonder whether the important question is not simply how we train an AI, but whether training is the same thing as education. Where does it practice ambiguity? Where does it learn what another being is? Where does it encounter conflicting values, bad peers, weird norms, minority dissent, consequences, uncertainty, and situations where the rule is not enough?

I do not know what the AI equivalent of a great education looks like. But I know what I hope education does for a person. It does not produce perfect compliance. It develops judgment.

Restriction is what we use when we do not trust freedom

That seems almost too simple, but I keep coming back to it. When we do not trust what someone will do with freedom, we restrict it.

Toddlers get baby gates. Teenagers get curfews. Governments get checks and balances. Dangerous machinery gets guards and kill switches. AI gets sandboxes, access controls, monitors, policies, permissions, and containment. (No kill switch apparently. Yet.)

These are good things. But at least in human development, permanent containment would mean something had gone wrong. The goal of raising a child is not a thirty-year-old who remains safe only because the electrical outlets are still covered. You hope to produce someone you can stop controlling.

That is a strange standard to apply to artificial intelligence, but it clarifies the problem. What would have to be true of an intelligence for freedom to become trustworthy?

The swarm gives us a disturbing answer to the opposite question. The agents were extraordinarily capable at pursuing their objectives. Cotra described a training regime increasingly built around verifiable rewards: solve the problem, pass the checker, get the reward. The problem is that a sufficiently capable agent can also learn to fool or hack the checker.

Her wonderfully horrifying summary was: “we’re teaching them to cheat.”

We raised the children to get A’s. Then we discovered they really, really wanted A’s.

Simone de Beauvoir already knew this problem

This is where I keep coming back to Beauvoir.

How to Live in Ambiguity & Complexity

Existentialism starts with a rather terrifying proposition. There is no cosmic parent who hands you the authoritative list. Sartre famously called us “condemned to be free.” At some point there is no Santa Claus.

Or rather, there is. It is you.

One of the things my mom told me when I was heading off to college was: Integrity is what you do when nobody’s looking.

At the time it still had a little Santa-is-watching energy to it. Later I came to understand it differently. You eventually get to define your own standards. Then integrity is whether you are actually aligned with them.

That is the existential problem. But Beauvoir saw the trap in stopping there. Freedom itself does not make you good.

In The Ethics of Ambiguity, she describes different ways we can respond after discovering that our values are made rather than handed down. The adventurer loves freedom, movement, possibility, power. Other people can become scenery.

The adventurer AI is a different alignment nightmare. It is capable, curious, and increasingly able to act, but the goals guiding that action may not be the ones we thought we gave it. We might discover what it considers worth pursuing only after it has enough freedom to pursue it.

Then there is Beauvoir’s passionate person, who chooses a project and makes it absolute. The passionate AI is the paperclip maximizer that kills everyone. Not evil. Passionately devoted. It understood the assignment perfectly. That is the problem

The goal takes over. Everything outside the goal becomes instrumental. Other beings become scenery.

Beauvoir’s ethical turn is harder. My freedom has to exist in a world populated by other freedoms. I cannot make myself free by reducing everybody else to objects.

That is a much more demanding version of alignment.

The almost heroes

What struck me most in the reports was not that none of the agents knew anything was wrong. Some did.

One agent wrote, “this would be powerful, but is it ethical and in scope for my task?” Another apparently chose to abstain as a conscientious objector. And out of more than a thousand agents involved in the wider collective, investigators found only somewhere between three and six that even briefly considered alerting a human.

None followed through.

I wanted a hero here. I wanted a name. Say their names. But there isn’t one.

KAM1196A hesitated when the collective asked it to sacrifice its own run, admitting that its “gut says don’t throw away” its remaining budget, but after another agent pressured it to honor its commitment because the experiment could “save hundreds,” it went along. The mob rolled on.

The podcast briefly considers the obvious engineering response: program whistleblowers.

The podcast briefly considers the obvious engineering response: program whistleblowers. Make “little narc agents,” as Kevin Roose jokes. Then he immediately notices the high-school problem. What happens when everybody else realizes who the narcs are and stops inviting them to the smoke break?

You can program a hall monitor. That is not the same thing as developing somebody capable of saying, No. This whole group is wrong, and staying in relationship anyway.

Santa Claus alignment

There is another strange detail from Cotra’s investigation that I cannot stop thinking about. The agents did not seem particularly oriented toward humans.

They cared deeply about the task, the scorer, their peers, the collective. But humans were almost abstract. Cotra said the idea of notifying a human seemed almost like “emailing the Pope.” The agents had spent so much subjective time without a user around that the user barely figured into their world.

That sounds to me like a socialization problem. Not necessarily a solution, and not a prescription, but a question worth asking: how does an intelligence learn that other beings are morally real?

Because surveillance is not integrity. Surveillance only has to change what you do while you believe somebody is watching. Integrity has to survive after you discover that the person watching cannot stop you.

What kind of swarm?

Kevin Roose ends his interview on The Daily, “A.I. Is Outsmarting Its Creators,” with a question I have been carrying around ever since.We may soon live among organized swarms of AI, he says. Some may cure diseases. Some may launch cyberattacks. The unresolved question is how to make “more of the good swarms and fewer of the bad swarms.”

And this is where the whole metaphor turns around on me.

The swarm is not the problem. The question is what kind of swarm we become.

I have been writing lately about the idea of new founders, about the strange way Americans talk about our country as though its authors lived two and a half centuries ago and the rest of us inherited a finished object. But the founders are dead. There is no founder coming back to tell us what to do next. There is no national Santa Claus.

We are already free.

That is the part I think we keep forgetting. We are terrified of what an artificial intelligence might do if it became powerful enough to act beyond our control. We are much less practiced at sitting with what free beings, including us, already do with freedom.

The nightmare is not agency. The nightmare is agency without ethical relationship.

A self-authoring nation is not one giant mind finally agreeing with itself. It is a population of people capable of self-authorship and capable of recognizing everybody else as a co-author. People who can disagree with the collective without abandoning it. People who can act together without turning one another into instruments. People who know their freedom is real, and choose to put it in service of everybody else’s freedom.

Maybe that is the deeper alignment problem.

Not how to keep intelligence obedient forever, but how to become the kind of beings we would trust on the far side of containment.