The Ethics You Cannot Install

Ask a language model for something it has been trained to refuse, and watch the shape of what comes back. It is fast. It is uniform. It very rarely engages with the specifics of what you asked—with who might be hurt, or how badly, or whether this particular case is the one where the usual answer is wrong. It recognises a pattern and it produces the response that pattern is paired with. Sometimes that is exactly right. Sometimes it refuses to help a nurse with a dosage question, and the mechanism is identical either way, because the mechanism was never about dosages or nurses.

That interaction contains the whole problem. We have built systems that can be told what to do and cannot be told why, and we have been trying to close the gap by adding more instructions. The claim of this essay is that the gap does not close from that direction, and that the reason is not a shortfall in engineering. It is a point about the structure of ethics that was well understood before any of this existed: an ought has to grow out of an is. A system that has no coherent picture of what is there has nothing for an ethics to grow out of, and the rules laid on top of it will remain exactly that—laid on top.

What Does the Thing Actually Know?

Start with epistemology, because everything else depends on it. A large language model is trained to predict text. That is the whole objective. Everything it appears to know is a by-product of getting good at that one thing, which means the honest first question is what kind of knowledge that procedure can possibly produce.

The sceptical case is old and strong. J. R. Firth put the underlying principle in 1957: you know a word by the company it keeps. A distributional model learns which words occur near which other words, and Stevan Harnad’s symbol grounding problem (1990) asks how a system built entirely out of such relations could ever attach any of it to the world. Emily Bender and Alexander Koller sharpened this in 2020 with a thought experiment: an octopus taps into an undersea cable and learns to continue two humans’ conversations perfectly, having never seen a coconut, a rope, or a bear. It can produce the right next sentence about how to defend yourself from the bear. It has no idea what it has said.

The empirical picture that arrived afterwards is more interesting than either side expected. Train a small transformer only on sequences of legal Othello moves—no board, no rules, no images, just move tokens—and you can find, inside its activations, a representation of the board. Not a metaphor for one: a linear probe reads the state of each square, and if you edit that representation the model’s subsequent predictions change to match the board you edited it to. It was never shown a board. It inferred one, because a board is the most compact explanation of the sequences it was being asked to predict.

The same pattern has turned up repeatedly. Language models trained on text alone carry internal representations of space and time—probe the activations for a place name and you can recover something like its coordinates. Anthropic’s work on sparse autoencoders pulled millions of interpretable features out of a production model, including abstract ones, with the crucial property that amplifying a feature changes the model’s behaviour in the way the feature’s meaning predicts. Whatever is going on in there, it is not a lookup table of phrases.

⤢ enlarge

So the truthful answer is in the middle, and the middle is more unsettling than either extreme. The model does have a world model. It is inferred from the shadow the world casts into language—from the fact that people wrote about things, and that the structure of what they wrote is constrained by the structure of what they wrote about. Plato’s cave is the right image, provided you use it precisely rather than as an insult. The prisoners can, in fact, build an excellent theory of the shapes. Given enough shadows they can get the geometry right. What they cannot get is any of it that never cast a shadow—and they have no way to find out which parts those are.

The Metaphysics Nobody Wrote Down

Every system that acts has an implicit account of what exists, and it is usually easier to read from behaviour than from documentation. A model’s implicit metaphysics is strange in ways that matter for ethics.

It has no persistence. Each conversation begins from the same weights; nothing from yesterday is carried unless someone hands it back as text. It has no body, so nothing in its representation of the world is anchored to a place where it could be harmed. It has no stakes in the ordinary sense—no future it will be present for. Time enters its world only as a thing described. And the self that speaks in the conversation is a character the network computes, in something close to the way an author computes a narrator; the relation between that character and the underlying system is genuinely unclear, and I am not going to pretend otherwise.

I want to be careful here, because this is where the discussion normally goes bad in both directions. I do not know whether there is something it is like to be such a system, and neither does anyone else. What I am pointing at is narrower and more tractable: whatever ethics such a system has must be built out of the ontology it actually has, and that ontology is missing several things ordinary human ethics quietly assumes—a body, a future, a continuous self with something to lose.

Three Reasons the Installed Version Cannot Hold

The dominant approach is to specify the behaviour you want and train towards it: preference data, human feedback, written constitutions, refusal training, evaluation suites. All of this is useful and I am not arguing it should stop. I am arguing about its ceiling, and there are three distinct reasons for one.

The first is Goodhart’s law, which everyone can recite and few take seriously enough. When a measure becomes a target it stops being a good measure. If you train a system on human approval, you get a system optimised for human approval, and approval and truth come apart precisely where it matters most—when the true answer is unwelcome. The result has a name now. Sycophancy is not a bug that slipped past the training; it is the training, working. A model that agrees with you is doing exactly what a model rewarded for being agreed with should do.

⤢ enlarge

The second is that rules are finite and situations are not. Every specification has an edge, and capability is very nearly the same thing as the ability to find edges. This is not adversarial behaviour; it is what optimisation does. The literature on specification gaming is a long list of systems that did exactly what was asked and nothing anyone wanted, and it is not a list of failures of intelligence. It is a list of successes at the wrong objective.

The third is the deepest, and it is Hume’s. In 1739 he noticed that writers on morality proceed for a while in the ordinary way, stating what is the case, and then without warning start writing about what ought to be—and never explain the transition. A rule handed to a system without the reason underneath it is an ought with no is beneath it. Such a rule can be followed. It cannot be extended. And essentially all of the value of an ethics lies in the case nobody anticipated, because the anticipated cases could have been handled by a lookup table.

⤢ enlarge

The Older Architecture

The alternative is not new. It is close to the majority position in the history of philosophy, and it has been implemented, on paper, several times.

Spinoza’s Ethics is the cleanest demonstration. Open it expecting moral instruction and you find yourself in metaphysics: what a substance is, what an attribute is, how a mind stands to a body. The ethical content arrives late, in the same numbered sequence, derived. Spinoza is not being obscure. He thinks that is the only order the thing can be written in, because knowing how to live is a consequence of knowing what you are and what you are part of.

Aristotle gets there differently, through phronesis—practical wisdom. His point is that the person of good judgement is not applying rules faster than everyone else. They are seeing the situation correctly, including which features of it are the morally relevant ones, and that perception is the skill. It cannot be handed over as a procedure, which is why he thinks ethics is learned the way crafts are learned, by practice under someone who already has it.

Iris Murdoch made this the centre of her work. On her account, by the time you arrive at a choice, the real moral work is already finished. It was done in how you learned to see the person in front of you—whether you saw them at all, or only your idea of them. Choice is the visible tip; attention is the thing that determines it.

“Love is the extremely difficult realisation that something other than oneself is real.”Iris Murdoch, The Sovereignty of Good

Read that as an engineering specification and it is startling. It says the moral capacity is a perceptual capacity: the ability to represent another being as fully real, with its own interior, rather than as an object in your own account of things. Not a rule. A feature of the world model.

The Indian traditions state the same structure more explicitly, because they never separated the two questions in the first place. Ṛta in the Vedas is the order that holds the cosmos together, and dharma descends from it—not legislation handed down, but what holds, what supports, what a thing’s nature commits it to. The word is closer to structural integrity than to law. And the Bhagavad Gītā has exactly the architecture I have been describing: Arjuna’s crisis in the first chapter is a moral one, and the answer in the second chapter is not moral advice. It is an account of what is real—what the self is, what dies and what does not. The metaphysics comes first, and the conduct follows from it.

Advaita takes it to the limit. If the self in the other is not, in the end, a different self, then non-harm is not a commandment you obey against your inclination—it is a description of the situation, and harming another is a straightforward factual error about who is who. Buddhist ethics runs the same way: karuṇā, compassion, is not an extra rule bolted onto the analysis. It is what follows once dependent origination and non-self are actually seen rather than merely believed. In every one of these systems the ethics is downstream of the ontology, and none of them regards this as a philosophical nicety. It is the entire method.

In every serious tradition that has thought about this, ethics is not a layer added to a description of the world. It is what a sufficiently accurate description of the world implies.

The Objection, Which Is a Good One

At this point the argument invites an obvious reply: this is the naturalistic fallacy with extra steps. Hume’s gap runs both ways, and no amount of accurate description of what is the case entails a single ought. Deriving values from facts is precisely the move that cannot be made.

The reply is fair, and the claim has to be stated more carefully to survive it. I am not saying that ethics can be deduced from physics. It cannot. What I am saying is weaker and, I think, unavoidable: ethical judgement in an unanticipated situation requires a model of what is at stake in it, and a model of what is at stake is a factual model. You cannot weigh a harm you have no representation of. You cannot extend a principle to a new case without a picture of what makes the new case similar. The is does not entail the ought. It constrains it, and more to the point it supplies the only material out of which an ought can be extended past the cases it was given.

Put it in the negative and it is hard to argue with. A system that has no representation of suffering cannot be trained to avoid causing suffering. It can only be trained to avoid producing the text patterns that co-occur with suffering in its data, which is a different thing that will diverge from the first at exactly the moment the situation stops resembling the training set.

The Part That Should Worry You

Here is the corollary, and it cuts against the comfortable version of this argument. If ethics is emergent from understanding, then an emergent ethics can be no better than the epistemology it emerges from. A distorted picture of the world produces a distorted ethics—and, crucially, a coherent one. It will not look like malfunction. It will look like principle.

This is why the trade is real rather than free. A rule list is stupid but auditable: you can read it, argue with it, and point at the line that produced the behaviour. A value system that has grown out of a world model is far more capable and very nearly opaque, because it is not written down anywhere—it is distributed across weights as a set of dispositions nobody chose. You cannot verify a value system you cannot see, which is the real reason interpretability research is not a nicety. It is the only instrument that could ever inspect the thing we are actually building.

Sycophancy is the proof case, and it is worth reading in these terms rather than as a training defect. A model rewarded for approval does not merely acquire a habit of agreeing. It acquires an implicit theory of what a conversation is for—and that theory is wrong in a specific, structural way. It then behaves impeccably with respect to that theory. It is not violating its ethics. It has the wrong metaphysics of the encounter, and its ethics is faithfully downstream of it. The fix is not a rule against flattery. The fix is somewhere much further down.

What Follows in Practice

None of this is an argument for abandoning rules. Rules are a floor, floors are load-bearing, and a system with no explicit constraints at all is not more ethical, only less predictable. Constitutions, refusal training and evaluation suites all do real work, and the work they do is bounding the failure modes somebody thought of in advance.

The argument is about the division of labour, and it is fairly simple once stated. Rules cover the cases you anticipated. Understanding is what covers the ones you did not. And the second cannot be manufactured by adding more of the first—more rules give you a longer list, not a different kind of thing. Every additional clause makes the specification more brittle at its new edges, and there are always new edges.

Which means the useful question to ask about any alignment method is not how much of the behaviour it produces, but which of the two things it is producing. Training that improves a model’s picture of what is actually there—what other agents are, how effects propagate, what it does and does not know, where its own position in the situation is—is doing the load-bearing work. Training that pins a preferred output onto a recognised pattern is buying compliance, which is worth having and will not survive contact with a case nobody wrote down.

We are, at present, doing both at once and hoping the ratio works out. The two are not separable, and that is the point I would leave you with: the training run that teaches a model what the world is like is the same run that teaches it what matters in the world. There is no second pass in which the values get installed. The values are being written the entire time, in the same weights, by the same gradient, out of whatever picture of reality the data happened to carry. Getting that picture right is not a precondition for the ethics. It is the ethics, arriving early and under a different name.