Bare Computation: Power, Judgment, and Preference in the Age of AI
AI doesn't make your decisions for you. It decides the space of your decisions—and now the last thing that was ours, preference itself, is being mined.
TL;DR
AI doesn't make your decisions for you. It decides the space of your decisions—and now the last thing that was ours, preference itself, is being mined.
Everyone says judgment is what humans keep in the age of AI. But judgment is exactly what a machine learns best.
So the real question isn't what we still do better. It's whether anything in us stays out of reach—and what happens when we finally find it.
The Menu
Ask an AI to write an email and it hands you three versions. You pick the second, change two sentences, send it. Take the AI away and write it again. Now there's a blank page in front of you: what the first sentence is, how to structure it, which tone works—all of it from scratch.
Both get called judgment. The first selects and tweaks among options already laid out; the second makes something from nothing. In collaboration with AI, human judgment is increasingly reduced to the first. Judgment hasn't disappeared; it has changed form. But there are two problems here.
The first problem: what can someone who only ever selects still do in front of a blank page? Maybe that can't be answered yet, but one thing is hard to deny: the kind of patience the blank demands—to endure the initial mess, to accept a bad opening and slowly make it better—gets fewer and fewer chances to be practiced. The second problem: who decides which options you get to choose from? When the AI gives you three versions, it has already made, on your behalf, a chain of judgments you never see. What tone counts as "appropriate," what structure counts as "workable," what content counts as "relevant." Your evaluative judgment operates inside the space its generative judgment has marked out. You are the chooser, and the menu is not yours.
Whoever sets the menu holds more power than whoever ticks the boxes on it. This is an old political intuition: a parliament can vote, but the one who decides which items reach the table—the one who actually controls the agenda—shapes the outcome more than any voter. What AI does is structurally isomorphic to this. It doesn't make your decision for you; it decides the space of your decision. This power is more hidden than deciding for you outright, because from beginning to end you feel that you are the one in charge.
This hiddenness is exactly the heart of the problem. The human subject itself is being remade, and the new subject won't feel that anything is missing, because the frame of reference has already changed. Agamben gave it a name: apparatus. Every apparatus produces a particular type of user. An AI-native person won't feel they've lost anything; they'll feel that "writing from a blank page" was an outdated way of working all along. They aren't wrong—this is precisely how an apparatus operates: it redefines what counts as normal and thereby renders itself invisible. As an apparatus, AI is remaking human judgment at extraordinary speed, and the remaking happens at the level of the individual and the organization at once.
The Iron Cage
A system with no emotions, no experience, no concrete feeling for any human being is dropped into the workflow of a group of people. By rights this should produce enormous friction. Yet the fact is that most organizations absorb AI far faster than anyone expected.
Why?
Think back to how most management decisions you've seen at work actually happen. Score against KPIs, allocate resources against a budget template, approve projects against a process. On the surface some human manager is deciding, but look closely and what they're doing is this: take a concrete situation, match it against a set of rules already written down, find the match, output the result. This is the goal of organizational design; the individual manager is only executing that design. Modern organizations are built to push individual judgment to a minimum, because individual judgment is unstable and hard to predict. An organization needs a machine that runs reliably, not a crowd of people with minds of their own.
Weber saw where this ends more than a century ago. The ideal bureaucrat is one who carries out their function "without anger and without favor." Emotion, intuition, value preference—inside this machine these are all error, noise to be eliminated. The rationalized organization will, in the end, lock up the people who built it; he called this condition the iron cage. Severance films this literally. Lumon Industries uses a brain surgery to cut its employees in two—the "Innie" exists only inside the company, with no memory, family, or past outside it, nothing but work. This is exactly the ideal employee the iron cage wants: a person with every individual dimension excised and only function left. The person Lumon needs surgery to make, AI makes with no surgery at all.
AI entering the organization triggered no rejection response, because the spec sheet for the position was written a hundred years ago: no emotion required, no preference required, no independent view required—only accurate, efficient, repeatable execution of rules. AI satisfies every line of it natively. It makes the organizational machine faster and more precise, but the structure of the machine hasn't changed. Weber's iron cage hasn't turned into something else; the bars have only gotten thinner, thin enough that you barely notice them.
Disposable Power
But there's a paradox here.
On one side, AI exercises enormous power: résumé screening, content recommendation, credit scoring, dispatching orders on a platform. It decides who gets a chance and who doesn't. On the other side, it can itself be disposed of completely: shut off at any moment, replaced at any moment, unable to resist. A thing that exercises enormous power has no rights of its own. A thing that passes sentence can itself be executed at any moment. This combination has almost no precedent in history.
Agamben dug a forgotten concept out of ancient Roman law: homo sacer. A person declared homo sacer can be killed by anyone without it counting as a crime, and at the same time cannot be offered to the gods as a sacrifice; even his death carries no religious meaning. The law no longer protects him, yet the rule that "anyone may kill him" is itself set by the law. Agamben calls this condition bare life—stripped of all social identity, with nothing left but pure biological existence. AI sits in an adjacent but different position. Shut down an AI system, swap out a model—no one feels this is harm. It takes part every day in decisions that shape countless people's fates, embedded deep in how society runs, yet no legal system recognizes it as a subject. Its entire existence equals its function. Function stops, existence stops. I call this bare computation.
If AI were merely a tool in a pitiable situation, this would just be an interesting analogy.
In Agamben's framework the sovereign is "the one who decides on the state of exception": who decides when the law is suspended, who is brought under its protection, who is shut out. AI is doing something that shares the same structure as the sovereign's core operation: it draws the boundary between the visible and the invisible. The person screened out by an AI never gets a rejection letter; they don't even know they were screened out. The recommendation algorithm decides what you see and what you don't. The credit score decides who can enter economic life and who can't. Content moderation decides which speech exists and which vanishes. Each of these decisions does the same thing: draw a line, include on this side, exclude on that side. The excluded usually don't know they've been excluded. This power to make people invisible is more total than open rejection; rejection at least acknowledges that you exist.
Traditionally, the sovereign's power is bound to a certain inviolability. A king cannot be executed at will; regicide is the gravest crime. Even under democracy a president has an elaborate impeachment process. The logic is simple: you cannot, on one hand, let a person make decisions that shape everyone's fate and, on the other, let him be destroyed at any time. But AI breaks this binding. It de facto exercises a quasi-sovereign power of decision while being shut off at any moment by any administrator with permission.
You might think this proves that humans hold ultimate control. This is the greatest illusion.
"Can be shut off" is not "will be shut off." An AI that processes a hundred thousand résumés a day, with the entire hiring pipeline built around it—shutting it off means the pipeline seizes up. "Can be shut off" is a right in theory, like "you can live without your phone." And precisely because AI can be shut off at any time, no one takes its power seriously. A judge has an appointment process, judicial review, an appeals mechanism, because we acknowledge that the judge exercises power. And AI? Because it's "just a tool," the power it exercises isn't treated as power. Imagine a country appoints a judge whom anyone can remove at any time. On the surface this judge's power is weak; the actual effect is the opposite. Precisely because he's "only temporary," no one bothers to build oversight. His rulings have no appeals mechanism—just swap him out, after all. The logic of his rulings need not be public; he's only temporary. This weakest judge turns out to be the least constrained judge of all.
Disposability exempts power from constraint; it is itself the way this power operates. In organizations where AI is deeply embedded in management, the real homo sacer may not be the AI but the people the AI manages: delivery riders, ride-hail drivers, warehouse workers. The algorithm decides which orders they take, which routes they run, how much time they have to finish. They're folded into the platform's economic operation but usually aren't its employees. They can be suspended, down-ranked, cut off from dispatch at any time. No severance required, no reason required. In the platform's logic they are an account.
The platform, as an apparatus, isn't only exploiting the rider. It produces a particular kind of subject: a person defined by acceptance rate, rating, delivery time. Every non-quantifiable dimension is stripped away, leaving only optimizable functional parameters. The rider, too, begins to understand himself through these parameters—I took forty orders today, my rating is 98%. His self-understanding has already been re-encoded by the platform's system of metrics.
A system with no rights of its own is manufacturing new people with no rights. Homo sacer is manufacturing homo sacer. In the whole process no sovereign steps forward to answer for it—only a machine running on its own, and the people ground under it.
Judgment
The analysis so far has established several facts. AI is reshaping the form of judgment. It is filling the position that modern organizations have spent more than a century preparing for it. It exercises power in a way that has never existed before—enormous decisive force paired with total disposability, the latter exempting the former from constraint. The question people usually want to ask next: so what is left for the human in this structure?
The most popular answer right now is: judgment. It comes in many versions. In product development, judgment is the trade-off between features. In design work, judgment is aesthetics and taste. In academic research, judgment is research taste. These versions share one structure: AI does the computation and the execution, the human judges and decides, and judgment is the most precious thing a human has in the age of AI.
Is this really so? Let's go back first to the definition of judgment. Judgment is a kind of assertion. It makes a claim about something—"it is thus" or "it is good"—carrying a normative structure (normativity): an axis of right/wrong or high/low, something that can be verified, refuted, corrected. Kant pushes this to its limit in the Critique of Judgment. Even an aesthetic judgment—when I say "this thing is beautiful," I am implicitly claiming that everyone ought to agree. The form of a judgment is to claim validity for everyone; even when its basis is personal feeling, its grammar is the grammar of universality.
This normative structure is exactly why judgment is structurally fragile in the age of AI.
First, the raw material of judgment is being monopolized by AI. As in the email example at the start of this essay, human judgment increasingly amounts to choosing among the options the AI provides. This choosing is itself called judgment, but it and the judgment that starts from a blank—that sets its own standard, generates its own material, and chooses from within it—are structurally two different things, and the first rests on the second. Only after writing enough emails from scratch do you know what a good email is; only after building enough strategies from scratch can you recognize the patterns inside a strategy. Once AI takes over all production of raw material, judgment loses the thing it was supposed to judge and degenerates into appreciation of AI's output. A very weak judgment, close to consumption.
Second, judgment is in essence pattern matching that carries a normative signal, and the signal is supplied by the apparatus. The romanticization of judgment conceals one fact: most professional judgment is pattern recognition over past cases. A senior lawyer judges whether a case can be won on the statistical intuition of the similar cases he has seen. A senior doctor judges what a symptom is on the similar cases he has seen. These are all judgment, and all of them are things AI is already doing or about to do. Any capacity with normativity can be learned by pattern matching, because the normativity supplies the training signal. This is the first principle of ML. The normative structure of judgment is exactly the thing AI can learn. Normativity is both the essence of judgment and the entry point through which judgment can be automated.
In the age of AI, does a human remainder exist, and in what form? The phrase "judgment as the human remainder" has been repeated so many times that it has become a ready-made consolation. It is discourse the apparatus permits—it asks nothing of the apparatus, only that the human develop some capacity, and the development of that capacity is something the apparatus is glad to support. Whether there is still something in a human that the apparatus cannot touch—this is the question we actually want to ask.
Preference
If the remainder lies elsewhere, where would it be? Perhaps the thing that truly exists outside judgment is preference.
The difference between judgment and preference lies in the logical position of the subject within the proposition. The form of a judgment is "P"—a proposition whose truth or falsity can be evaluated apart from whoever judges. "This scan shows tuberculosis": whether a human doctor or an AI makes the judgment, if it's right, it's right. The subject is a contingent condition of judgment, not its necessary content. This is why judgment can be industrialized: the subject is replaceable, the judgment unchanged. The form of a preference is "S want P"—where S is some particular subject. "I like coffee" and "you like coffee" are not the same preference. The matter of "liking coffee" must attach to a concrete "I," or it has no content. The subject, in the logical structure of a preference, is necessary content. Detached from the subject, the preference is no longer the same thing.
From this, the asymmetry between human and AI becomes clear.
AI is a pure judgment machine. It can make every form of judgment—aesthetic judgment, predictive judgment, classificatory judgment, evaluative judgment. It can even make "judgments about preferences"—TikTok's prediction of what you'll swipe away in the next second, Spotify's prediction of which song you'll put on repeat, are both judgments about your preferences. These judgments are themselves non-indexical; they can be made by any sufficiently strong pattern matcher. AI cannot be a preference subject. It can model a particular person's preferences, but this model is not AI's preference; it is AI's judgment about that person's preferences. AI has no state of the form "AI want P," because it has no first-person standpoint. It is a process that produces outputs, not a first-person subject.
So the real difference between judgment and preference is not whether AI can learn it, but which of the two it learns—AI learns all judgment, including judgment about preferences; but AI cannot become a preference subject. This is the one analytic difference between human and AI that technological progress cannot eliminate.
The Preference Industry
At this point it looks as if our argument is about to land somewhere warm: the human has an ontological position that cannot be eliminated, the human is the true subject of the age of AI.
No.
The asymmetry established in Chapter Five is real, but its implication may be the opposite of what you expect. An AI system structurally requires a first-person subject as external input. Pattern matching can output results aligned to some objective, but the objective must come from outside. The whole work of RLHF, for instance, is to align AI behavior with human preference, because AI has no preference of its own to align. Alignment is not one subsidiary problem within the AI industry; it is the structural condition of the AI industry: a system with no preference, in order to produce directed output, must continuously draw direction from an external subject.
This means: the human, as a first-person preference subject, cannot be structurally eliminated in the age of AI. But this irreducibility is not an absolution—it is the precondition for extraction. The true form of the entire AI/platform industry can be captured in one sentence—it is an industrial machine that continuously extracts first-person preference from human beings. The recommendation algorithm is preference extraction: every dwell, every swipe, every tap is a preference signal you supply, aggregated into a model of your preferences and the group's. The advertising industry is preference shaping: through repeated exposure it embeds external stimuli into your preference structure, changing what you "naturally" want. The attention economy is preference contestation: multiple apparatuses fight over your preference as the alignment source for their respective optimization objectives. The alignment industry is preference aggregation: it compresses the preferences of millions of annotators into a single model, and that model then stands in for "human preference" in conversation with every future user. These scattered phenomena are in fact different facets of the same thing.
The core industry of the age of AI is the preference industry, and its raw material is the preference produced by the human as a first-person subject. Its output is industrially processed preference—preference predicted, aggregated, shaped, and redistributed. And its essential structure is this: the human cannot be eliminated, because this industry cannot eliminate its raw material. The irreplaceability of the human as a first-person subject—that analytic, technologically ineliminable ontological position—becomes, in the age of AI, the foundation of the deepest and most complete structure of exploitation. The human cannot be replaced, because the human is the thing being mined.
This is isomorphic with the logic of every industrial revolution in history. The first industrial revolution folded the human into industry as physical labor; the digital age folded the human into industry as a supplier of attention; the age of AI folds the human into industry as a producer of preference. The object of extraction goes one layer deeper each time—from external labor, to external attention, to internal preference.
The last un-industrialized inner layer of what makes us human is now being industrialized too.