What Hospitals, Courts, and Classrooms Are Quietly Giving Away

On accountability, synthetic actors, and the hollowing of institutional life by the use of AI

Dr Melanie James

A version of this post was published on Substack on May 04, 2026

There is something specific that makes a doctor’s recommendation different from a search result, even when they contain the same information. The doctor can be wrong and, as a person inhabiting a professional and moral order, they will carry something of the consequence of being wrong. That accountability is not incidental to the interaction. It is part of what makes the recommendation meaningful.

The same logic runs through every institution that matters. A judge’s ruling carries weight because the judge is subject to the obligations of the legal order they represent. A teacher’s assessment carries weight because the teacher knows the student has a stake in getting it right, and can be genuinely challenged or changed by what they learn. A social worker’s recommendation carries weight because the social worker can be held responsible for what follows.

Strip the accountability out of those interactions while leaving everything else intact and you have not improved efficiency. You have changed what the interaction fundamentally is. Furthermore, if you engineer the replacement to feel smoother and more responsive than the original, you have made the loss almost impossible to see.

This is what is happening quietly and incrementally as synthetic actors take on institutional roles. It’s not happening in every institution, but it is happening steadily in the places where it is easiest to measure throughput and hardest to measure what is being lost.

The surface form of authoritative institutional interaction is preserved almost perfectly. The system assessing your eligibility for a benefit, triaging your symptoms, evaluating your child’s learning needs, or flagging your behaviour as a risk, does all of this in language that sounds considered. It produces reasons, follows procedures, and generates outputs that look exactly like the outputs of a human professional operating within an institutional framework. Often it does so faster, more consistently, and with less of the friction that human professionals sometimes bring.

The interaction’s smoothness is part of the problem. It makes the substitution feel like an upgrade but as a contribution to the social glue it is far from an upgrade.

This is because AI is a synthetic actor and it cannot genuinely inhabit the institutional framework it is simulating. It cannot be subject to the moral order or social norms of the institution in the way a person is. It cannot feel the weight of a consequential decision. It cannot be repositioned by the person on the other side of the interaction, whose life the decision affects.

The interaction looks like the real thing but the accountability that gave the real thing its meaning is not there. Because the simulation is frictionless and pleasant, there is less reason to notice, less reason to push back, and less reason to demand the real thing.

I want to be precise about what I’m claiming here, because it is easy to misread this argument as technophobia or nostalgia.

I am not saying human professionals always get it right, or that human institutions are reliably just, or that AI cannot improve certain processes. The argument is more specific than that. It is that certain kinds of institutional interaction depend on a particular structure, one where both parties are genuinely subject to the moral order the institution represents. When you replace that structure with a simulation of it, something real is lost, even when the outputs look identical and feel better.

Consider education.

The relationship between a teacher and a student is not just an information transfer. It is a positioning relationship in which both parties are genuinely involved. The teacher positions the student as a learner with particular strengths, needs, and potential. The student may, over time, begin to reposition the teacher’s understanding of what is possible and what is needed. Both people are changed by the encounter.

When the teacher who has taught the same material for twenty years and encounters a student who asks a question they haven’t considered before it is not just updating a file. They are participating in a living moral order in which their judgements carry weight because they can be held to them.

A synthetic tutoring system can do many things a teacher does. It can explain, assess, personalise, and adapt. It can do all of this with infinite patience, at any hour, without ever having a bad day. It feels, to many students, better than the human alternative.

What it cannot do is genuinely share the moral stakes of the student’s development. It cannot be troubled by a student’s struggle in a way that persists beyond the session. It cannot be held accountable for the trajectory of the person it is shaping.

The student, over time, is being formed partly by interactions that simulate the weight of genuine human judgement without carrying it, and that feel more comfortable precisely because they don’t.

The same argument applies in healthcare, where the relationship between clinician and patient has always been more than diagnostic.

In legal contexts, where the right to be heard by a person who can be held responsible is foundational to the legitimacy of the system. In democratic participation, where the interactions through which citizens engage with institutions are the very mechanism by which those institutions derive their authority.

None of this is invisible. People can feel when something is off, even when they cannot name it.

The unease that many people report in interactions with institutional AI, the sense that something important is absent even when the output seems correct, is not irrational. It is an accurate perception of a real structural deficit. However, that unease is being actively smoothed away.

The systems are getting better at mimicking warmth, responsiveness, and care. The friction that might have prompted a person to notice what was missing is being engineered out.

What is hardest to see is the cumulative effect.

Each individual interaction might seem adequate, even good. The system assessed your claim, made a recommendation, generated a decision, and did so quickly and politely, without making you wait. Nothing obviously went wrong, but across millions of such interactions, the social glue of institutional life is being quietly substituted with phantom social capital.

It appears like something solid is being generated, such as a relationship and deep understanding, but there’s nothing there. Try to withdraw your “social capital” that you think you’ve been building through interactions with a synthetic actor, and you’ll find only a phantom version.

Until recently, the expectation is that institutions are inhabited by people who are genuinely subject to their obligations. The understanding has been that authoritative decisions carry weight because someone is accountable for them. However, increasingly the sense that when an institution speaks to you a person in some meaningful sense is there is merely a simulation of what was once there.

When that expectation erodes slowly enough, sweetly enough, it doesn’t produce a crisis but it does produce a drift.

Institutions begin to feel procedurally intact but morally hollow. People disengage, not through dramatic rupture but through a gradual withdrawal of trust that is difficult to locate. This is because nothing specific seems to have broken, and, because each individual interaction often felt more seamless or pleasant than what it replaced.

The social glue that has held human societies together for millennia is being replaced, institution by institution, with something engineered to feel like an improvement.

This is the fragility I have been trying to describe across my posts to date. Not the fragility of individual AI systems that can be tested, improved, and regulated but the fragility of the social foundations themselves. This includes the shared understanding that when interactions matter, they are conducted between people who are genuinely subject to their consequences.

We are building significant institutional infrastructure on top of interactions that simulate understanding and legitimacy without providing it, and it is happening in ways that feel like progress.

Developing the language to describe what is actually at risk before this starts breaking more substantially, is what my research and these posts exist to do.

The next step is to look more closely at specific institutions, and at what a framework for identifying and resisting this quiet substitution might actually look like in practice. That is where the work gets harder, and more necessary.

Melanie James is an independent researcher working at the intersection of positioning theory, social capital, and artificial intelligence. She is exploring what AI is doing to the social foundations of human life.

This entry was posted in Uncategorized and tagged , , , , , , . Bookmark the permalink.

Leave a Reply