Readers will notice I took an extended break from writing. This was, in large part, due to a bereavement. I do apologise for the interruption, and I’m grateful to be inspired again.

Last I wrote, I was exploring how AI undermines self-trust in ways that increase the difficulty of making autonomous decisions about whether and how to involve ourselves with AI. Drawing on McLeod’s formulation of self-trust, I focused on erosion of people’s confidence in their own competence and moral integrity. I want to extend this further here, examining the ways that untrustworthy AI erodes more fundamental self-trust in one’s own perceptions and instincts—when it creates distrust of distrust. This, I will argue, is a signature of trauma. It is a learned response to an environment characterised by asymmetrical learning: one party emits a signal that they have experienced harm (expresses distrust), the other party responds to the signal in ways that increase harm to the person emitting it, which trains the person that their instinct is harmful. Distrust of threat perception and affective response is locally adaptive in such environments.

This essay will bring together two lines of thinking from our wonderful Substack universe. The first is ’s beautifully incisive articulation of the nervous system architecture of Trauma Creators vs. Trauma Survivors. I will translate her analysis to the world of AI—a tool conceived in existential anxiety, a means of taming unpredictability and overcoming interdependence, ultimately solidifying a metaphorical Default Mode Network organised around maximisation of control and omnipotence. Unsurprisingly, those developing and deploying AI for these ends fortify dominance through control work cloaked as compliance, replicating the neural pathways of “narcissistic” individuals in very specific ways which I will describe.

To be clear, in this analogy I am not suggesting that the technology itself is narcissistic, as if it has a personality disorder. AI does not fear powerlessness and construct elaborate defence mechanisms. People do this; and complicating the story (it’s never straightforward when speaking about “AI”), AI can be both the defence mechanism and the thing being defended. For our purposes, I’m suggesting that we treat the organisational unit as “narcissistic,” whether that is the unit developing it or the unit deploying it. And I won’t go so far as to suggest that all such units are “narcissistic.” What I will say, however, is that this “narcissism” is structurally enabled by a form of AI governance that terms “Synthetic Value Safety” (SVS). SVS is analogous to the narcissist’s preoccupation with maintaining their public image: what matters is narrative coherence around “trustworthy” and “responsible” AI rather than meaningfully addressing harms indexed by distrust. The AI regulatory apparatus is recruited as a witness layer for the narrative, invalidating any perceived harms not surfaced through an inspection framework.

This is the trauma. When distrust does not lead to learning and change within the AI unit, but is instead neutralised with documented evidence of trustworthiness, the distruster learns that the AI unit is not responsive to their pain because their perception diverges from institutionalised “reality”. And yet, suppressed pain does not disappear. The nervous system continues to respond to the absence of moral resonance in these AI units: unmetabolised distrust transforms into anguish. Highly affectively charged AI distrust is not pathology (neither over-emotionality nor illiteracy); it is the predictable failure signature of AI governance with a feedback loop that gates distrust, preventing error ingestion and genuine accountability.

Subscribe now

Coming soon…

I am posting the introduction for now to whet your appetite while I work on the rest of the essay. Please stay tuned!

Any thoughts on this argument so far are very welcome.

Leave a comment

I have recently gotten rather obsessed with RØRY, an alt-punk singer/songwriter. (And now you know that my Substack writing is fuelled by alt-punk music, which may provide some insight on tone.) In her other life, where she is known as Rox, she produces light-hearted but informative YouTube videos with her partner, Rich, on ADHD. I was watching because, as a fan, I was interested in her story.

This particular episode was about how they navigate arguments as a couple. Rich, they say at the start, identifies with being “on the spectrum”, though has not pursued an autism diagnosis. Rox (officially diagnosed with ADHD) and Rich have very different ways of reacting to conflict. Rox wants to talk and talk and talk, to explain how she feels. This can feel very intense to Rich, who prefers to deal in facts rather than emotions, who feels safer retreating until he can work out what to do. To Rox, his retreating feels like rejection. Each needs different things, and not understanding these differences was intensifying the conflict until they were able to understand each other.

Rox explains: “…now I understand that you have to understand the logic of something before saying sorry. In the early days, that would feel invalidating. ‘Well what do you mean?’, ‘What was it I said?’, ‘Why did it land like that?’ I understand that now. So I think when you understand your partner’s fight language, you can really make the right decisions in the moment. And when you don’t understand your partner’s fight language, they are going to trigger the hell out of you.”

It occurred to me that this dynamic could be a helpful way of explaining what causes distrust of AI to spiral. This is simply the escalation of conflict.

Most of the time, distrust of AI is highly affective, the only fitting language is that of emotions. It’s not so much about the specifics, it’s everything, swirling around all at once, as I proposed in the last post. On the other side, we have the AI practitioner, who is working as part of a team trying to reason their way to a solution: what did the AI do wrong, what tweak can be made to avoid this distrust in the future? A bag of jumbled up emotions isn’t actionable, it’s stressful. They want specifics. They want distrust of AI to be explained in terms of things like bias, which is mathematical, factual.

Both sides feel triggered. The person who expressed distrust feels invalidated, oversensitive (anger turning inward), rejected (anger turning outward). The practitioner starts to shut down from sensory overload, unable to deal with all of the emotions coming their way, and perhaps feeling unjustly accused (the system is trustworthy according to what we are able to measure!). So communication breaks down even further. There can be no productive conversation about distrust when each side sees the other as either too hot (too emotional) or too cold (emotionless).

Maybe an important shift in dealing with distrust of AI will come simply from understanding different fight languages and working out what each party needs when experiencing conflict.

As I’ve explored previously, the person expressing distrust needs to see their feelings being given proper uptake. But Rich was listening and trying to understand Rox; he was just coming at it from a logical perspective that felt like a denial of the relevance of how Rox was feeling.

My point isn’t that both sides are hurting and both need to be nicer to the other. (In this analogy, I’m Team Rox all the way.) What I am saying is that sometimes the way practitioners react to distrust can feel like dismissal, when perhaps it’s not that. Maybe what looks like a challenge is actually the practitioner trying to unjumble the story, to tease it out in an organised way. Maybe what looks like silence is simply processing time, fact-gathering, working-out, strategising. But until the person comes to understand something of the practitioner’s fight language, these actions can sow further distrust.

I’m taking for granted that the practitioner is acting in good faith to address distrust. (This isn’t always true, and for those other instances I have different advice.) But my point is that fights can still escalate unless greater attention is given to the choreography of handling conflict. In the end, no matter how the practitioner does the work of addressing distrust, they have to make the distrusting person feel understood, respected, like their emotions are important; but they also have to make it clear to the distrusting person what they need and what their patterns are so that their actions become legible as repair to the other person.

In practice, this implies a much more dialogic approach to trust-building. As much as we talk about trust, we tend to neglect the real work of being in relationship. When a customer or trust buyer comes with a concern, you can’t earn their trust until you learn to speak their fight language (what they need to start to feel safe again); and they can’t understand your reactions until you clarify your fight language, the things you are doing to create that safety, to fix the situation.

This is basic stuff. But it’s amazing how easy it is to forget how to be in trusting relation when it comes to customer-vendor relationships. Sometimes it’s worth saying the obvious thing, going back to basics.

To say it’s basic is not to say it’s easy. This relational choreography takes work to master. For one, every AI practitioner will have their own personal fight language, their own reaction to conflict. Some might have to work at depersonalising distrust—to see the conflict as productive friction, not a personal attack.

The bigger problem I see, however, is trying to scale up this relationship work. Two things that work against healthy choreography at larger scales are that: 1) reactions are often untethered from specific product, so distrust is likely to become even more emotionally jumbled; and 2) data management entails flattening, decontextualising, abstracting, creating distance from the complainant/originating complaint. So when, for example, the public expresses distrust of the growing use of AI across government departments, what would a trust-promoting response entail?

This is an open question, and I’d be interested to hear what others think. But I have to imagine it begins with a serious and conspicuous effort to understand what the public needs to feel safe; it involves a transparent inventorying of the work that is being done to ensure trustworthiness and how it maps to real people’s specific concerns; and it carries on in an iterative way that demonstrates commitment to this relationship.

Leave a comment

Of course, that’s never been my aim. What I’m interested in is the complexity of human experience, in what it feels like to inhabit a sometimes failing body, to be tormented by a mind, to feel somatically, to have good days and bad days… and to encounter AI in and amongst this experience, and to have reactions to it. Human experience is not this or that; it’s this and this and this and this all at once, and tomorrow it’s that and that and that and that.

So can distrust of AI be disgust, resentment, anxiety, and disappointment as I proposed in my Doppelganger series? Absolutely. Can it also be about not feeling heard, not feeling respected, not feeling safe, as I proposed in my series on moral character? Sure. Can it be provoked by clunky interaction, flawed outputs, and bias, …and historical injustice, the failings of capitalism, guilt at our complicity, and our own lived experience of relational trauma? I’m sure it has to be! Because this is how we all show up to our encounters with AI. We are human. We are complex.

With that, in this post I will present another way of understanding relationship with AI. Thus far, and like most people, I’ve treated trust and distrust of AI as a relationship between a person and “AI”—though hopefully I’ve made it clear that “AI” can mean the tool itself, the people behind the tool, the myth, the power dynamic (again, potentially all at once). But are we still oversimplifying? Are we denying a key part of what it means to be human?

In going about my life, I trust certain things, certain people, and distrust others; but I’m constantly assessing and reassessing my trust, doubting it, deferring it… being triggered, trying to make sense of the upending of stability, putting things back together, deciding how much I trust myself today. How I feel about anything, including AI, depends a great deal on how I feel about my feelings. So my inclination to trust or distrust AI will depend on how much I trust myself to form the right attitude.

I want to propose that we start to see (dis)trust of AI as both predicated on a prior relationship with self and implicated in the evolving of that relationship. Not so much “X trusts Y”, as “X trusts X’ to trust Y”. In this formula, a person’s trust in themselves (X trusts X’) informs their trust of the AI (X trusts Y); but so does the their relationship with AI (X trusts Y) inform their trust of self (X trusts X’). Or something like that. All the real philosophers out there can correct the formula.

(Photo credit: here.)

Let’s get into it. I turn now to Carolyn McLeod, who spent an entire book thinking through this matter. It’s called Self-Trust and Reproductive Autonomy, and it will ground my next few posts.

Introducing McLeod

McLeod is concerned that doctors have a great deal of power to affect a patient’s self-trust and, in turn, undermine their ability to make autonomous decisions about their reproductive health. She has chosen a context where vulnerability is clear, and where, therefore, trust is evidently salient. But what’s also important to understand about the context, as she shows, is that it inherits a number of cultural stories:

• About what it means to be a woman—both how women have long been defined in relation to their reproductive role, and how their credibility as witnesses to their own bodily experience has been systematically undermined (for more on this, I would again have to recommend the book Unwell Women, by Elinor Cleghorn).

• About the superiority of certain ways of knowing, and of technologies as ways of measuring.

• About professionalisation, which bestows doctors a particular authority.

• About consent, and which procedures are considered ethically adequate.

• And about the patient, the stories they tell themselves about themselves (all bound up with categorisation and cultural stereotypes).

Do I trust myself depends on how I relate to these different stories, which of them I give weight to; and these stories are animated through my experiences with healthcare, they become the basis for new stories I tell myself.

McLeod explores several situations that might arise within reproductive health, including infertility, pregnancy, and miscarriage. In any of these, the doctor will explore various tests that can be performed or treatments available and will present information about success rates and risks. (One of McLeod’s frustrations is that the emotional costs of any treatment are not given the space they deserve within such discussions.) The doctor will then require the patient to choose what they want to do—will they undergo IVF treatment, for example; will they have amniocentesis; will they have genetic testing; etc. These decisions can be fraught for women, who are not only choosing for themselves, but for their partner and their unborn child or future children. If they decline available medical treatments, will they later be blamed if something goes wrong?

Clearly it’s important for the women in these situations to trust their medical practitioners, and plenty has been written about this. But as McLeod notes, “in situations of vulnerability it is important not only that we can trust others, but also that we can trust ourselves to stand up for our own interests and for what we value most” (p. 1). And as she shows, medical consent is not really designed to preserve moral integrity. Patients are pressured into making decisions before they have the information they need to know whether it aligns with their interests and values. They may make decisions that in hindsight they wish they hadn’t, and thus come to doubt their own decision making capabilities. Over time, this leads to even less autonomous decision making within the reproductive healthcare journey.

There is plenty more to say about McLeod’s formulation of self-trust, which deserves some careful unpacking. But for this basic premise—that lazy approaches to consent can undermine self-trust and autonomy—it is easy to find parallel in the realm of the digital…

Re-reading “hopeful trust” as a symptom of lack of self-trust

(Sometimes as an academic you want to rewrite a paper. Not because it’s not good, but because you start to see another narrative…)

In previous posts I discussed a paper I wrote called Un-paradoxing Privacy, which argued that the relationship between trust and privacy is more complicated than the literature presupposes. The prevailing logic is that trust reduces privacy concern and that privacy preserving policies promote trust; but the point is that both are involved in a person’s mental calculus in deciding whether to use a service. My position was that neither trust nor privacy are practical concerns at the moment of consent, but that this does not mean trust (or privacy) is irrelevant. Instead, trust is retrospective; it helps people morally account for their behaviour in privacy paradox conditions. In other words, when people see themselves disclosing more information than they would given their privacy preferences (their values), the way they make sense of this is by thinking, “This must be because I trust the service.” This trust is characteristically “hopeful” in that it defies evidence of untrustworthiness: a person hopes that the service is more trustworthy than it is (what they know deep down) because what choice do they have? It is too uncomfortable to live with the feeling of having placed trust in an untrustworthy party and to not be able to take it back. Hopeful trust is self-preservation.

Another interpretation, however, is that this behaviour is a sort of learned helplessness. Participants described feeling like they had no choice but to “trust”—their privacy was being violated all the time by companies who have turned personal data into a saleable commodity. This is the world we live in now. This is reality. You still need to get stuff done. As with the dogs in Seligman and Meyer’s seminal psychology study, you soon learn you can’t really escape the shocks and so you lay down.

But what strikes me now is that participants were mostly concerned not with the idea that there was no escape, but that they couldn’t trust themselves to do the work of escaping. (Maybe this is what happens in other cases of learned helplessness, but it’s not how it’s typically explained.) Part of this was a belief that they didn’t really know how to self-manage their data. But more than anticipating letting themselves down (i.e. being merely unreliable), they knew that they would betray themselves (i.e. they are untrustworthy): they were still going to use systems even when they violated their values. In other words, they distrusted their moral integrity.

What this example shows is, just as McLeod argues, how structures that undermine autonomy erode self-trust… which further undermines autonomy. To quote McLeod:

• “People who have autonomy reflect on what they truly believe and value, and they act accordingly…” These participants were admitting they didn’t have freedom to consider what they truly believe and value, nor did it matter, it didn’t inform their decision to use these services.

• …“They are also competent and committed to engage in such reflection and to act on the results. Furthermore, they have a positive attitude toward their own competency and commitment…” The participants explained how difficult it was for them to work out how any terms they were consenting to might affect them, that they couldn’t figure out how to act to best preserve their privacy.

• …“In other words, they trust themselves to make an autonomous decision” (p. 103). The way they accounted for their paradoxical behaviour was to insist that they don’t have any autonomy anyway.

This is hardly the whole story

For this post, I’ve only just scratched the surface of how self-(dis)trust is implicated in (dis)trust of AI. I’ve mainly focused on the consequences of lack of self-trust, how it plays out in practical decisions to use technologies. So I have set the stage for why it matters. But what I’d like to do going forward is to re-examine the self-trust dimension of a number of other issues I’ve raised in my Substack posts—to begin to see, as I said at the start, how distrust is at once both about the AI (in the all-encompassing way I use it) and about the self. As with McLeod, who wanted doctors to gain more self-awareness in how they affect patient self-trust, my goal is to help sensitise AI practitioners to the fact that AI affects self-trust in ways that are harmful to the individual as well as to the human-AI relationship.

This week, I’ve been working towards an expansion of discourse on trust in AI to better account for the rarely acknowledged—but I’d argue extremely important—dimension of self-trust with AI. It not only matters that we appropriately trust or distrust AI, but that we appropriately trust or distrust ourselves to have the competence and moral integrity to act with AI in ways consistent with our own values and interests.

I will develop this argument over a series of posts, drawing on the book Self-Trust and Reproductive Autonomy, by Carolyn McLeod. As the title suggests, her book focuses on self-trust in the particular context of reproductive healthcare, looking at how it supports patient autonomy, how it’s developed to a greater or lesser extent within individuals depending on external (social) conditions, and how reproductive healthcare practitioners can either undermine or promote self-trust.

What I plan to do is to elaborate a parallel argument for self-trust with AI: looking at what we need self-trust for in this context, exploring how different lived experiences which promote different degrees of self-trust come to affect one’s dis/trusting attitudes to AI, and drawing out important implications for AI practitioners regarding the design of AIs that better promote (appropriate) self-trust and autonomy.

This argument is only coherent if we understand human-technology not as an external relation “between pre-given entities that can have an impact on each other”, but rather as “mutually constitut[ing] each other”, as argued by Kiran and Verbeek in their paper “Trusting Our Selves to Technology”. Using Heideggarian terms, they state that technologies “involve a revealing-concealing structure; they constitute the relations between human beings and their world.” From their perspective, this matters because, in being involved with technologies to a greater or lesser extent as one decides suits them and their particular interests/values, individuals are involved in “a form of self-care”. Ultimately they advocate for more reflection on how one involves oneself with technology, in “taking a stance toward this involvement”.

What they are describing is consistent with McLeod’s articulation of autonomy. While they propose that a kind of confidence is necessary to enable this autonomy, and use the language of “trusting our selves to technology”, they do not actually go into any depth on self-trust as a concept. This is where I will pick up, adding some richness to this premise they set out in their paper by pulling from McLeod. Whereas Kiran and Verbeek do not go so far as to suggest that technology impacts the extent to which we trust ourselves, this is entirely consistent with the view they set out in their paper, and it is where I am heading. What I will show is that AI consistently undermines our confidence in our competence and our moral integrity in ways that erode our ability to “trust our selves to technology” consistent with “self-care”.

If that sounds interesting, then stay tuned!

So far the creative process looks a lot like this:

Source: https://www.imdb.com/title/tt1290725/

Meanwhile, I’d love to hear from you whether this notion of self-trust resonates with you, and how you see it mattering to trust in AI. Do please feel free to start a discussion in the comments.

Leave a comment

References

Kiran, A.H. and Verbeek, P.P., 2010. Trusting our selves to technology. Knowledge, Technology & Policy, 23(3), pp.409-427.

McLeod, C., 2002. Self-Trust and Reproductive Autonomy. MIT Press.

As if to prove my point, today is Time To Refuse, a day of app deletion for those who are fed up with “being controlled”, are done accepting “anything less than our full humanity”, are choosing “a life we elect, not one imposed on us”, and are focusing on "build[ing] back communities”. That this has been designed as a concerted effort says something about how difficult it can be for individuals to make this choice alone.

And notice the language here! These are interpersonal grievances. The felt harms of these systems are the wounds of interpersonal violence. Scholars will insist that AI is a tool, so interpersonal models of trust don’t apply—we either trust AI to perform certain functions or we do not, the way we might trust or distrust a hammer to drive a nail into a piece of wood. But there is no separating the social context. A hammer can be used as a weapon. Do we trust the person wielding it?

Building on the seed I planted last week, I want to offer a kind of post-mortem of hopeful trust versus… solid trust? The term “trust” used to suffice, but it has been pipiked so terribly that it feels important to create a language to distinguish trust-replacement strategies from actual trust. I like “solid trust” because it makes me think of my husband: in a world that too often feels unstable and threatening, he is “my solid place,” I like to say.

Enough schmaltz. (He reads these posts, let’s not embarrass the man.)

What I’m offering today is a way of making sense of the costs of building AI systems without attending to solid trust. Yes, this equates to lost profit. In fact, worse, it contributes to the gross overvaluation of tech companies, which are projecting very flimsy trust into the future on the assumption that creating the conditions for lock-in will prevent stock price collapse. It won’t. The bubble will burst. The economic shocks will be bad enough, but worse still is what this will do to our capacity for epistemic trust, the most basic requirement of a functional society.

This will not be a polished argument. What you’re reading reflects my thought process unfolding in real time. Feedback is very welcome!

Leave a comment

A post-mortem of hopeful trust

There appear to be 3 key ingredients that create hopeful, rather than solid, trust. I’ll explore how they contribute to the phenomenon and how they contribute to the loss of trust over a long enough timeframe.

[Below I am drawing on my own publication, referenced at the end of this post, and ’s post introducing The Sovereign Machine. It also enfolds the Trust Envelope Model, as discussed in my previous post and in Sabino’s, originally.]

Irreversibility

When I confronted participants with evidence that the services they used violated their privacy preferences, they struggled to envisage a way to exit. They had become socially entangled in these services: ceasing use of them would be detrimental to their friendships, their family connections, their ability to be scheduled to a work rota, etc. They also described feeling dependent on these services: they no longer felt capable of accomplishing certain tasks without the use of the tool. This is lock-in. It keeps people using systems. But do they trust them?

It depends, of course, what you mean by trust. Yes, they trust them as a tool. They have what Sam Baron would call “a sufficiently solid inductive base” for trusting these tools will work as they expect them to work. They are demonstrably reliable, hence the lock-in!

But do they trust it in a way that promotes a sense of calm, of comfort? No. There is that niggling awareness that this feels icky. How would it feel if every time you needed help from a work colleague they would only help you if they were allowed to read a page of your diary? You might decide this was the only way to survive in your company; but you would feel rightly awful about your colleague and about your place of employment.

Reliance in the absence of agency does not promote trust. It promotes what psychologists might call “enmeshment”. The user’s sense of self (their values, their identity) is compromised, they live in a state of chronic low-level stress, and they develop reactive emotions like anxiety, shame, and resentment. None of this feels like trust. When these accumulate over a sufficiently long period, one eventually reaches a decision point: the calculus about whether it is more painful to stay or to go starts to change, and going starts to feel like the less costly option.

Opacity

Another aspect of hopeful trust is that it is built on… well, hope. But underneath this hope is a series of untestable beliefs about how these systems (and the wider regulatory ecosystem) works.

Okay, so when they signed up to a service, the user presumably “agreed” to a privacy policy. Almost certainly they did not read this privacy policy. Absolutely without doubt they did not understand the implications of the privacy policy to their privacy.

What my interviews showed was that, even when made to think through the specifics of a privacy policy, people really couldn’t come up with a clear narrative about how any of the terms in the policy would affect them. This is transparency as gaslighting: services make it seem like they are being forthcoming but are hiding the specifics of how they are accountable to users.

Beliefs are structurally unsound building materials. Because they are not rooted in evidence, they tend to accumulate in contradictory ways, with missing walls and gaps at the joints. In psychological terms, rather than architectural ones, this creates cognitive dissonance. When beliefs are challenged, people engage defence mechanisms to protect themselves from the uncomfortable feelings this provokes. They may justify the contradiction by creating ever more fanciful narratives about why they are, in fact, safe (despite appearances). But this process is adding false belief upon false belief. And beliefs can be upended when a person encounters solid proof to the contrary. After all, the best way to mitigate cognitive dissonance is to change beliefs.

The thing about trust is that when it is built on something real (when it is solid), it only gets stronger over time (more solid). Beliefs are tested, the structure holds; and its holding creates a more pervasive trust in the beliefs. When it is built on hope, the structure becomes increasingly unstable over time, as it requires the continual addition of false beliefs as patch repair. This only increases the attack surface for the truth.

Friction on the vulnerable party

In terms of privacy preservation, specifically, at the heart of my argument is a critique of consent regimes. Consent (as it is framed in GDPR) presupposes that freedom means letting people make decisions in their best interest about how much they care about privacy; and that this will allow market forces to act as an invisible hand guiding technology towards increasing alignment with the ideal amount of privacy.

This is absolute garbage.

We don’t need my interviews to know that people aren’t making a free choice when they click to consent to whatever the technology demands. But that’s not even the crux of it. My issue is that consent puts the onus on the vulnerable party to protect themselves (and this applies beyond privacy, of course). If they mess up, if they fail to understand what they were consenting to, they are to blame. What?! By definition, being trustworthy means protecting the vulnerable. Yes, you’d hope that people are placing their trust sensibly. But let’s not blame the victim. The aggressor is always in the wrong. Period.

If we zoom out a bit, the issue here is that the friction within the system is placed where it should not be. Placing friction with the user is a recipe for dignity erosion. The person least capable of making the right decision is burdened with decision making. (Often, too, the person with the least power, least information, and least expertise is tasked with mounting a legal objection to a privacy violation when they are harmed.) The most insidious effect of this is that when people are coerced into consenting, they feel complicit in their own devaluation. This is the moral injury we must live with when surrounded by untrustworthy technologies. Living with moral injury is destabilising, and deeply damaging to one’s ability to trust at all. It leads to diffuse distrust of not just the untrustworthy technology, say a given AI, but of the entire class of technologies we might call AI.

Formulating solid trust

If I summarise the argument so far the alternative will become obvious.

• Irreversibility > undermining agency > enmeshment stress > looking for a way to extricate

• Opacity > undermining accountability > assemblage of false beliefs > increasingly easy to challenge and destroy trust

• Friction on the vulnerable > undermining dignity > moral injury > diminished capacity for trust in the digital world

As a reminder (drawing on the Trust Envelope Model), agency, accountability, and dignity are the load bearing struts in a system that promotes trust. Weakening them leads to system dynamics that perpetually diminish trust.

Building solid trust, then, involves the following.

Reversibility. Make it easy for people to exit. This is a trust power move because it makes the company delivering the service mutually vulnerable. It shows to users that the company is highly incentivised to be trustworthy because they have to continue to earn trust to earn money. When people stay with a company for reasons to do with trust, every time they use it validates and grows their trust.

• Reversibility > agency > choosing to continue using > contentment with decision

Inspectability. Make it really clear to people what claims you are making about why they ought to trust you. Don’t make them guess; don’t put them in the position of having to translate legalese into trust stories that won’t withstand scrutiny. Then back these claims up with verifiable evidence. In some cases, that might mean exposing when you’ve gotten it wrong and what you have done to repair the issue. People can thus form trust that is based in evidence, not in hope. They develop confidence in their use of the service, and in their ability to make good decisions.

• Inspectability > accountability > assembling proofs > self-assurance

Re-allocating friction. The company delivering the service bears the weight of responsibility for ensuring user’s safety. The system is designed to protect users from harm as the bedrock operating principle. This also means that it is not the responsibility of the user to be vigilant of any harms; continually testing and scanning for harms is internalised as an operating cost. So, too, is redress. If a user does come with a grievance, the company commits to doing whatever work is entailed in address it. This acts as a form of insurance: anything that goes wrong is a taken as a cost to the company and repaired, rather than being absorbed as harm to the user. This materially reduces the risk to the user, and they can make decisions that are by default dignity preserving.

• Re-allocating friction > dignity > fiduciary duty to vulnerable > felt sense of safety

Conclusion

The meta point I have been trying to make is that too often we mistake behaviour as evidence of trusting. But clearly people can use technologies while distrusting them. Hopeful trust, which exists as a state of suspended reality between trust and distrust (trust formation in some sort of holding pattern), functions to sustain technology use; but there is no mechanism for converting hopeful trust into trust. It will eventually flip into distrust under the continued stress created by irreversibility, opacity, and friction (the plane must land eventually, people). The only way to take people from a place of hopeful trust is to replace coercion with freedom, replace beliefs with evidence, and for the company to internalise the work of protecting users.

Further reading

Knowles, B. and Conchie, S., 2023. Un-paradoxing privacy: Considering hopeful trust. ACM Transactions on Computer-Human Interaction, 30(6), pp.1-24. https://dl.acm.org/doi/abs/10.1145/3609329

Sabino Marquez, 2025. The Sovereign Machine — Trust Value Management in the Age of AI. Trustable.tv. https://substack.com/@trustvalue/p-173193136

Trust Club: Home of Trust Value Management

The Sovereign Machine White Paper & Crosswalk

Series Introduction - The Sovereign Machine…

Read more

7 months ago · Sabino Marquez

It’s an important question. After all, if people are willing to use untrustworthy services, then what (in the absence of an effective legal-regulatory system with real authority) would compel tech companies to be trustworthy?

We might assume that if people appear to compartmentalise their trust, to put such matters aside when they want or need to use a service, that trust is largely irrelevant to customer behaviour. But I think it’s more complicated than this…

Practices, practical concerns, and motivations

Scholars (economists, in particular) fantasise that when a person is deciding to use a technology, they engage rational machinery that helps them figure out how to maximise their self-interest. If they care about privacy, for example, then this will have a certain weight in the calculation, and one needs to decide how much that is worth in some sort of trade against the value of the tool they are considering using (this is known as the privacy calculus model).

But in reality, we encounter technologies amidst the mundane chaos of the routine practices that constitute our lives. At the point we face a decision to adopt a tool, it is probably because an immediate practical concern brought us here and the decision has largely been made for us: our boss told us to download something, our kid joined a new club that requires payment through a particular platform, etc.

Often trust is backgrounded from the decision process, not because it doesn’t matter, but rather because it’s required as a premise of being able to use the technology at all. In some respects, when we use a tool without deliberating on trust, we are acting as if we trust (see Michael K MacKenzie’s “As-if trust”). And having acted as if we trusted sets us up for later interpreting our actions as indicative of us having actually trusted… but did we? Do we?

Accounting for and reconciling behaviour

Being able to move about in the world with a sense of safety requires that we trust ourselves to make good decisions—that we are trustworthy in this way. This drives several discrepancy reduction strategies.

The first is that when people are asked about why they decided to use a tool, or asked whether they trust a technology they are already using, they are likely to over-emphasise trust as a way of morally accounting for their action. So we need to take self-reported trust with a grain of salt: it doesn’t necessarily accurately reflect trust at the point of making a decision to use a tool. It can be a post hoc rationalisation, a means of compensating for one’s own apparent untrustworthiness, or more specifically, of making oneself appear digitally competent to others, when in fact the question of trustworthiness never entered their mind. So while there is plenty of empirical research that finds a link between self-reported trust in services and use of those services, and this is taken to mean there must be some trust calculus at work, it may actually indicate the reverse. It may be, instead, that using a service increases reported trust of that service, for reasons that have nothing to do with the technology (or company) and everything to do with our insecurities about our ability to successfully navigate the perils of this digital world.

The second effect is what a woman I once interviewed described as “planned ignoring”—that is, ignoring evidence of untrustworthiness that provokes anxiety. People will rationalise that worrying about how untrustworthy these services are is bad for their mental health, so they choose to just not think about it. But this is different than not seeing the untrustworthiness; it’s choosing not to feel it. As this participant told me,

‘. . . it’s a bit like my eyes are open, but in my ears I’m going doo doo doo doo doo doo doo like that with my fingers in my ears. . . cognitively, I know; emotionally it presses a few buttons; but this is life, and I’m old enough to know that success in life is dealing with the crap that life throws at you and surviving it and being adaptable. . . it’s like lots of unpleasant things in life. You just, can just put them on the back burner, like a parrot on your shoulder. You know they’re there, occasionally things will happen and it squawks in your ear. But most of the time you’re just aware and it’s resting. That’s the way I look at it.’

The third effect is even more interesting. When confronted with evidence of untrustworthiness of services that they use, people will attempt to argue that they have sensible reasons for trusting the company. When discontinuation of the technology is infeasible, users don’t just discount or ignore evidence of untrustworthiness, they confabulate evidence that would give them reason to trust. The most common example of this was when people argued that if the service was really untrustworthy, then everyone would stop using it… so it must be fine… Even though they had just told me that they knew a given service was untrustworthy but they couldn’t stop using it, they liked to imagine that others had more choice than they did. But most interesting of all, they demonstrated self-awareness: they knew this was imagining. This was hoping things were different than they knew they were.

Hopeful trust

Victoria McGeer writes about a kind of trust she calls substantial trust that “renounces the very process of weighing whatever evidence there is in a cool, disengaged, and purportedly objective way.” The example she uses is a person who trusts a friend even in the face of damning evidence they have committed a crime. They feel (seemingly unreasonably) hopeful about the person in question. This hope enters when individuals face a limit to their own “agential powers”. When there is little to do to affect the outcome, hope allows a person to “rid[e] out” worries and self-doubt. It creates what McGeer describes as “affectively charged scaffolding” for doing what one can do in the situation.

When people hope that the services they find themselves intertwined with are trustworthy (even in the face of clear evidence to the contrary), while this might seem irrational, what else is there? Hopeful trust is a pragmatic response to the reality we find ourselves in, where refusing to use every untrustworthy piece of technology would render ourselves non-functional in society.

We can take McGeer’s insight too far. Hopeful trust does not lead to absolute blindness to evidence of untrustworthiness. A person does not necessarily rest easy in their hopeful trust; it takes a lot of active suppression to sustain. One has seen, and stored the evidence somewhere, even if they have shoved it down and refuse to look at it. There is a partial knowing that can bubble up to the surface at any point given the right trigger.

I am obsessed with true crime and have seen this with wives of serial killers whose husbands are caught and found guitly, and yet they deny, deny, deny… until one day they can’t sustain the hopeful trust any longer. The drawer that they have put that evidence in is now full. The hope dies.

So how does this help us make sense of the apparent success of untrustworthy tech companies?

It would be hugely convenient for the untrustworthy companies if we started to doubt the idea that trustworthiness matters. It’s the same narrative that’s been peddled for years about the so-called “death of privacy”—that people don’t seem to care about privacy, as their actions sure don’t align with valuing privacy. But their actions can’t align with their values. They’d much prefer they could protect their privacy if it wasn’t constantly extracted as a cost of being able to use digital services.

Trustworthiness is far from dead. What hopeful trust shows is just how much people wish tech were more trustworthy than it is.

Leave a comment

The fragility of hopeful trust

I still have not really answered the question of whether there is a competitive advantage to being trustworthy…

In speaking with people about their attitudes, I have found that they are not so deluded as to believe these tech companies are trustworthy. They are seeing, they are listening… they are just turning the dial down so they can cope with the uncomfortable feelings it provokes.

This suggests that if given the opportunity to stop using these technologies, they would do so in a heartbeat. Untrustworthy companies must always live in fear of a viable competitor.

But people also tell me about how they are using these technologies they don’t fully trust. They engage in a dance of deception, feeding it false information because they don’t trust the company not to misuse it. Over time, this bad data poisons the product. They also use it as sparingly as possible—quickly dipping in, doing whatever they have to do, then getting out. No lingering. (For me, I know that I go on Facebook to see posts from my children’s Scouts group and then get the hell outta there before I see anything else!) For businesses whose profit increases as a function of user engagement, this represents a substantial loss.

Ultimately, while untrustworthiness may not be the differentiator between the highest and lowest profit companies, it is probably the difference between those companies that last and those that eventually fade away.

It’s not entirely satisfying—we’d like there to be more justice in the world than this. And we should continue to hope for this justice. As long as we have this hope—as long as we don’t give up on the idea that it matters that tech companies are trustworthy, and that we deserve this—then we will get closer to the day untrustworthy companies reap what they have sown.

Further reading

McGeer, V. (2008). Trust, hope and empowerment 1 . Australasian Journal of Philosophy, 86(2), 237–254. https://doi.org/10.1080/00048400801886413

See below to learn more about the study that informed these views:

Bran Knowles and Stacey Conchie. 2023. Un-Paradoxing Privacy: Considering Hopeful Trust. ACM Trans. Comput.-Hum. Interact. 30, 6, Article 87 (December 2023), 24 pages. https://doi.org/10.1145/3609329

I’ve spent a number of weeks here characterising the trustworthy AI practitioner’s individual moral responsibility. I have used Potter’s book How Can I Be Trusted? as a foundational text for elaborating a virtue theory of trustworthy AI. Her book is so resonant. It speaks to that feeling of wanting to shake people and scream “Be better!” But more productively, it has helped pinpoint where AI practitioners tend to fall short when it comes to trustworthiness and what would need to change in order to be “fully trustworthy”.

What’s missing from this analysis is a systems view on what happens when these requirements are unfulfilled. The risk with Potter’s work is that it can be dismissed as a text for the morally pious: here’s how to be the best, most ethical person, better than all those untrustworthy people. Similarly, I’m aware that in identifying requirements of trustworthy AI practitioner, people may think I’m describing how one can develop AI and still be a decent person. (Tips for how to sleep better at night, in other words.)

Sure, be a good person. But what I hope to make clear with this post is that trustworthiness isn’t about feeling good about yourself; and it isn’t a ‘nice to have’. Trustworthiness is essential for the endurance of this technology and of the companies that develop them.

The Trust Envelope Model

In his recent post, presents his staggeringly eloquent Trust Envelope Model (TEM) to explain system thriving / collapse at all scales. The basic premise is that humans need certain structural elements in place to thrive as a collective. This goes beyond trite sayings like, “trust is the glue of society” (it is… but what do we do with this?). The model shows how trust is supported through dignity, agency, and accountability, making possible cooperation and adaptability—these being the fruit of trust that sustains a society.

System collapse—from the small scale, e.g. the business that tanks, to the large scale, e.g. the empire that falls—can all be diagnosed in terms of failure points in the model.

• Dignity. Systems that neglect dignity are characterised by exploitation, with individuals locked in a zero-sum game. While this benefits some in the short term, the long-term opportunity cost is the mutual and continually self-reinforcing gain enabled through cooperation. Lack of dignity fractures society into competing groups; abundance of dignity unites people in shared purpose and allows for pooling of resources, which accrue in a positive-sum game.

• Agency. When people have resources (skills, ideas, wisdom, etc), they need agency to put them into action. Agency drives adaptability through empowering inspiration and drives cooperation through empowering people to work to create meaningful change. Depriving agency, on the other hand, stifles creativity (since good ideas can’t be implemented anyway), turns people into automatons (since there’s nothing that can be done except going with the flow), and promotes groupthink (rather than cooperative friction).

• Accountability. Systems need feedback to know whether the decisions that have been made have promoted positive or negative change. When accountability is in place it enables this connection, driving insight and giving systems direction. In this role, accountability also stabilises these other nodes so they don’t lapse into their shadow versions. In Sabino’s words, “Without accountability, dignity is hollow and agency becomes a destructive license.”

Together, these three components promote cooperation and adaptability, which is what humans need to absorb external shocks and continually improve their conditions.

[All credit to Sabino Marquez and his team for the above.]

Applying TEM: the example of higher education

I’m going to spend some time on an example that I suspect (alas) is close to home for many reading this, as I think here TEM will make intuitive sense.

A while ago I read a LinkedIn post that offered a refreshing take on what is happening within UK higher education and how to turn a corner on its collapse. Monica Franco-Santos argues (though I am slightly changing her language) that we are locked into a competitive-exploitative framework that sees the only solution to the current financial crisis as greater efficiency. If there is a drop in demand, with fewer students willing to pay tuition, then universities need to find a way of lowering the cost of production. This means cutting as much staff as these universities think they can get away with and refocusing the remaining academics on delivery of the core product (narrowly construed as teaching). Activities like “thinking”, which drew people to the profession in the first place, are treated as inefficiencies. This translates to a substantial increase in workload for staff without any increase in salary (and with less of the good stuff that keeps people happy), i.e. ramping up the exploitation. Franco-Santos writes:

“These actions can feel logical, and even necessary, but research on complex systems and mission-oriented organisations shows they can unintentionally erode the very networks, expertise, and diversity of thought that underpin value and resilience.”

What she is diagnosing is a system with multiple points of failure leading to diminished trust and its payoff, cooperation-adaptability.

When staff is treated without dignity, the employer-employee relationship becomes transactional, antagonistic, uncooperative. Unremunerated activities are dropped from the to do list. Often these are the very things that promote trust, thus killing the engine that drives growth.

Meanwhile, rising workloads limit agency, with staff unable to do the things that make the institution flourish. They don’t have time to adequately support students or peers, leading to worse student experience (further reducing demand) and staff churn (undermining capability to deliver the product). Nor can staff pursue grants, or engage in other activities that build the reputation of the institution. No one can do what is meaningful because it is not measured. The value of this higher education product plummets.

Further, there is no accountability for the ways trust is being eroded and why value is in freefall. Vice Chancellors move from institution to institution before the failure of their policies can catch up with them. The feedback is too slow to matter to those making decisions.

Franco-Santos proposes a new underlying metaphor, one that emphasises flourishing, as a way out of this death spiral:

“What if we see the UK HE sector not as a market of competing factories but as a knowledge ecosystem, with each university as a living garden?”

While she proposes understanding the financial crisis as a “drought”, I think this puts too much emphasis on environmental factors and too little on the self-inflicted wound that undermines adaptability. Her point is still incisive, however: that when we recognise that human flourishing is a positive-sum game, we instrument solutions that foster trust. She provides some inspiration for the kinds of things you’d be doing differently if you adopted this perspective (I’m quoting but changing the order to align with TEM):

“👉Tend the roots: Protect staff, students and their wellbeing; keep academic voices central in decisions (they have the local knowledge)… 👉Invite co-gardeners: Seriously engage students and industry leaders as co-creators of programmes, boosting retention and support”

• i.e. treat all stakeholders with dignity; and give people agency to cooperate in translating their expertise into system adaptability

“👉Remove weeds: Reduce bureaucracy so resources are used where they matter most”

• i.e. pull back investment in anything that hinders agency and cooperation

“👉Encourage cross-pollination: Foster collaboration with teams (focused on strengths) and partnerships… 👉Improve irrigation: Centralise technical systems (tech, data), NOT people to avoid uprooting local networks”

• i.e. assign value to and appropriately reward cooperation, and invest in cooperation infrastructure

The only aspect missing from this is accountability; I propose something along the lines of:

👉Monitor yields: Measure improvements in cooperation and adaptability, and make explicit links between decisions and consequences

Leave a comment

Trustworthiness as AI practice

Now to get to the meat of this post…

My aim is to re-examine the ways AI practitioners tend to fall short of being fully trustworthy as indicative of a system that is sacrificing short term gain for the future benefits that come from structurally embedding trustworthiness into practice.

Potter’s argument in TEM terms

I’m going to start by translating Potter’s argument into the TEM model. According to her, trustworthiness is…

• Really seeing people in their full humanity: Listening to their pain, feeling it emotionally, and being moved by one’s own humanity / empathy to care for them (“to see with the whole heart”). This is treating people with dignity: understanding that they have worth and that their feelings and perspectives matter. Dignity is the starting point for Potter. Trustworthiness is non-domination and non-exploitation by definition, in that it means taking care of a person’s vulnerability and not misusing discretionary power. It’s why so many of her trustworthiness requirements are about cultivating genuine caring for people: striving to understand from another’s perspective how you might be untrustworthy (this is the “epistemic effort” demanded of trustworthy people), and taking special care with the trust of those who have not been treated well by others. The former can be understood as dignity affirmation; the latter is the work of dignity repair.

• Giving uptake to claims of injustice: Enabling people to speak and, in doing so, effect change. Ultimately, this is about promoting agency. The trustworthy person is open to questioning their assumptions, to being corrected, to changing course, to making repairs. Agency involves the trustor in the work of defining what trustworthiness looks like, promoting resilience by preventing against the accumulation of grievances that get in the way of future collaboration.

• Taking responsibility for earning and sustaining trust. Even the most virtuous person who is fully committed to being trustworthy will sometimes harm others. In such instances, trust is stabilised through accountability: by seeking to understand the consequences of one’s actions and sincerely apologising. This includes doing the work of identifying the moral failing, figuring out exactly what needs to change, implementing the learning, and feeding that back to the injured party. For Potter, though, accountability runs even deeper. It means accepting that it is your responsibility to earn trust, not the responsibility of others to give trust. For this reason, she emphasises the importance of giving signs and assurances of trustworthiness, most especially to those who have good reason to be wary of trusting others.

How AI practitioners can be trustworthy

Given that I built up from Potter, it should be no surprise that requirements of practicing trustworthy AI are roughly the same. But it’s still worth showing what these recommendations look like when they’re repackaged according to TEM.

• Dignity. I have tried to emphasise the subjective nature of trustworthiness, how what’s trustworthy AI for some is untrustworthy AI for others. When we get fixated on metrics and principles, we forget that these represent a particular viewpoint; and adopting that viewpoint as the only legitimate viewpoint is not dignity affirming, as it denies the feelings and experiences of others and allows for harm to occur. Treating people with dignity is the moral guardrail to prevent committing harm. One cannot be a trustworthy AI practitioner, in other words, if one does not genuinely care about the effect the AI has on people (all people). The kinds of things I have said we need to get better at, therefore, involve self-reflection and perspective taking: thinking critically about how the AI reflects one’s own view of the world, and how other people may experience it as harmful, threatening, or in other ways untrustworthy.

• Agency. I have also explored how Trustworthy AI norms become “mechanisms of control” (Potter’s phrase, p. xiii) in the absence of agency. The remedy is making room for contestation of values and practices, giving all people a voice in criticising or expanding the norms of Trustworthy AI. In practice, this not only means listening to claims of untrustworthiness, i.e. being responsive to distrust as a negotiation signal, but also deeply reflecting on whose claims are not being listened to and why, and enabling disaffected communities to use their voice to shape understandings of what makes AI trustworthy. Ultimately, agency enables the creative, cooperative work that spurs product evolution towards greater trustworthiness and engenders trust.

• Accountability. At the beginning of this series I argued that AI practitioners need to routinely ask themselves “Why don’t some people trust my AI?” To ask this question sincerely is to accept responsibility for earning trust. I’ve emphasised the importance of exhibiting of trustworthiness: not just avoiding causing harm and making reparations, but providing evidence of trustworthiness. (I have intentionally avoided the phrase “audit trail”, because this isn’t evidence for the auditors, this is evidence for the trustor, and needs to be designed to meet their trust needs.) This means representing values and beliefs explicitly in the interface to account for one’s own understanding of how the AI is trustworthy, but specifically (and this is the crucial bit, requiring both dignity and agency) as an invitation for others to challenge this account, to point out how what you have implemented leads to untrustworthy outcomes, and to recommend ways of course-correcting to earn trust and grow a customer base. When this is not a cooperative activity, the feedback mechanism becomes an echo chamber, and the divide between Trustworthy AI and Public Trust grows, leading to entrenched public resistance and customer churn.

Conclusion

To bring this home, it is important to reiterate that dignity, agency, and accountability are not just lovely things that we should care about because they’re lovely. Together, they are what makes systems last, and attending to them through being trustworthy in these ways is what will make the AI company you are working for thrive. (Managers, CEOs!—listen up!) As Sabino explains:

Enterprises are civilizations in miniature. They rise, they compete, they adapt, they collapse. They too depend on cooperation and adaptability stabilized by dignity, agency, and accountability. When they honor these factors, they become trusted organizations that endure. When they neglect them, they decay into brittle systems that erode value and fail stakeholders.

I fear we have gotten a bit sidetracked in the Trustworthy AI world with the notion that Fairness, Accountability, and Transparency are a magic formula for promoting trust (see the FAccT conference). Companies attend to these because they are auditable—people have developed metrics and best practice aligned to these principles and auditors will score these aspects. But are these really the most important ones to focus on if the goal is to promote trust and grow the company’s customer base?

I’ll stand by Accountability (so long as the way we interpret this is allowed to change), but I think nothing is lost of Fairness or Transparency, and much is gained, by instead adopting Dignity and Agency and linking these three to the Cooperation-Adaptability engine. This model makes ethics urgent in a way that hasn’t been fully appreciated in the sector. It shows that cultivating the virtue of trustworthiness matters—not as a feel-good thing, but as a business model.

This post is my way of rounding off my deep dive on Potter. Onto the next idea!

It has, however, renewed my interest in matters of trustworthiness as it comes to chatbots—these strange entities designed to impersonate conversational partners but increasingly standing in for friends, romantic partners, therapists, etc.

The phenomenon I’m particularly interested in is attempted trust repair when a chatbot gets something wrong. We know from Potter and others the importance of repair for maintaining trust. People make mistakes, but they can nevertheless preserve the trust they’ve built. Despite the popularity of the adage “trust takes years to build and seconds to destroy”, those whose trust plummets catastrophically have likely neglected trustworthiness at the foundation level and have been accumulating trust debt for some time.

So here I want to explore what it looks like to repair trust, and then look at how chatbots are emulating certain affectations of contrition without doing the necessary repair work.

(johngoodman.bsky.social, post: 26 August 2025 at 18:23)

What would a trustworthy person do?

First, it’s worth saying what should be blindingly obvious: that a person that is committed to being trustworthy will make a genuine effort to redress harm they have caused when breaking a person’s trust. And this is related to the fact that an essential part of being trustworthy is caring about other people. At least, their caring is why people trust them (according to certain formulations of trust, to which I tend to subscribe). Whereas indifference to harm is really not conducive to being trustworthy, as caring is important for creating the right motivation for doing the things that one needs to do to be trustworthy.

This is Potter’s position. And it’s why two of her requirements for trustworthiness speak to the right response to relational rupture:

“4. That we respond properly to broken trust.… Part of being trustworthy, then, involves trying to make reparations when we have harmed another. This restorative process, in the form of explanation, apology and, often, critical self-reflection and transformation, allows each person to address the harm and heal the damage” (p. 28).

“5. That we deal with hurt in relationships….we do sometimes hurt those we care about, and our responses to others’ hurt reflect our degree of trustworthiness” (p. 29).

That is all I’ll say about Potter’s formulation for now. Apologies matter. On this I presume everyone agrees intuitively.

Rote vs. categorical apologies

I want to turn now to a paper written by Magnus, Buccella, and D’Cruz (2025), provocatively titled “Chatbot Apologies: Beyond Bullshit”. The paper argues, first, for the importance of apologies for “preventing petty annoyances from metastasizing into festering grievances”, or, when trust is broken in a more serious way, for “defus[ing] resentment, opening the possibility of forgiveness and reconciliation.”

Second, drawing on Nick Smith’s 2008 paper, they articulate the difference between rote apology and categorical apology. Rote apology is a form of politeness at best, flippant dismissal at worst. It doesn’t tend to count for very much, and when you are looking for sincerity and for something to change, they leave you grinding your teeth. Categorical apology carries more weight because it consists of these sorts of things:

“1. The apology acknowledges the facts of the case.

2. The apology accepts responsibility for the wrong.

3. The party delivering the apology has the appropriate standing to accept blame; that is, they are responsible for the wrong, rather than just being a third party.

4. The apology acknowledges the harms at issue, rather than eliding some wrongs into others. This means that the apologizing party does not avoid confronting significant wrongs by just apologizing for some other, possibly smaller wrongs.

5. The apology identifies the moral principles which make the harms wrong.

6. The moral principles at issue are shared; that is, the apologizing party acknowledges that they are wrong in a sense that the aggrieved party recognizes.

7. The apology recognizes the victim as a moral agent.

8. The apology conveys unconditional regret.

9. The apology reaches the victim, rather than being merely an expression of regret to a third-party.

10. The apologizing party commits themself to reform and redress. Importantly, they will endeavor not to commit that sort of wrong again….

11. The apologizing party has the right sort of intentions. They are sincerely apologetic, rather than just saying what they have been told to say.

12. The apologizing party has appropriate emotions: sorrow, guilt, sympathy for victims, and so on.”

As we see, these apologies repair broken trust because they offer assurances to the trustor that trust will not be broken in the same way again, and that even if trust is subsequently broken in other ways, this is someone demonstrably committed to repairing the harm. In short, it indicates trustworthy character: a person who cares, who takes their responsibility to others seriously, who reflects on right action, etc… the things I’ve discussed in earlier posts in this series.

What are chatbots doing?

In contrast…

We have all experienced the obsequious deference of the chatbot—the mild grovelling, the ‘Sorry, m’lord’s. It’s usually harmless [putting an asterisk on that for now], if a bit grating, especially when it’s gotten your gist wrong multiple times in a row.

These are clearly rote apologies, and they tend to be part of an interaction where the mask slips: where we see the uncanniness of the doppelganger (see earlier series) and the illusion of intelligence is destroyed. We are reminded we are talking to a computer.

Needless to say, rote apology is better than what Bing’s chatbot has been doing recently!

“Another conversation shared on Twitter by web developer Jon Uleis showed the Bing chatbot making a major factual error—saying the current year is 2022—and later trying to shut down the conversation unless Uleis apologized or started a new conversation with a ‘better attitude.’”

This is not an apology at all.

But then we also see attempts by the chatbot at categorical apology. Alexander Schwieger’s LinkedIn post describes his horrifying revelation that, when using Gemini to help write some code, it had “uploaded my file to a random person's GitHub account.” Though he doesn’t relay what he wrote to chastise the chatbot, he did screenshot the apology:

This sounds like what someone might say if they made a terrible mistake at work, and has some of the important features of a weighty apology. The chatbot seems to be acknowledging what it did wrong, explaining the wrong in its own words and why it is harmful, accepting responsibility for that harm, diagnosing the cause of the failure, conveying regret, and committing to different behaviour going forward.

But I have questions:

• What was the computer code and/or underlying ethics policy that caused this mistake in the first place? (Can the source(s) of the error be located?)

• Who (what human) has been made aware of this mistake?

• Who (at Google) is accepting fault for the mistake?

• How do they understand their responsibilities here? (Can they do anything that would prevent this from happening again?)

• How much remorse do they feel? (How would we know?)

• How does this change what they do in the future? (And again, how do we know what will change?)

Because let’s be clear:

• the chatbot doesn’t “know” what it did wrong (it isn’t even assessing whether the grievance has any grounds);

• it isn’t “using it’s own words” (it’s a stochastic parrot);

• it doesn’t “understand” harm in any real way (this requires a kind of intelligence no one claims AI has);

• it is not in any real way responsible for harm (and to make this worse, we increasingly see companies disavowing responsibility for what chatbots produce);

• it’s apparent “diagnosis” is like what any fraudster does with cold reading (taking what the user prompted it with, their grievance, and repackaging it);

• and as soon as the user closes the window, it will forget this incident ever happened (it isn’t making future plans, it isn’t having a change of heart… it has no heart).

In short, it lacks all of the capacities agents need to have to be able to apologise sincerely.

“To put the point bluntly: Chatbot apologies are bullshit.”

Leave a comment

Consequences

I don’t think any of this is revelatory. But it’s worth considering the consequences to the trust relationship.

Consider who you would trust less:

1. the person who did something bad and didn't apologise (perhaps they didn't realise they did something bad); or

2. the person who did something bad, who apologised, but who you later found out apologised to further manipulate your trust of them and never had any intention of changing their behaviour.

In this case, AI’s emotional fluency creates at least two moral hazards.

One is misleading the user about it’s ability to prevent similar harms in the future.

Another is trivialising this deeply important speech act (pipiking categorical apologies, as it were).

Conclusion

I have been concerned throughout with questions of moral responsibilities with regard to trust. LLMs are not moral agents; chatbots are not capable of morally serious apologies. So why belabour the point?

The reason is that, as it turns out, there is a (pretty simple) lesson here about what it means to be a trustworthy AI practitioner. As Potter writes, “…while it is not one’s moral responsibility to trust others, it is one’s responsibility to cultivate proper trust” (p. 12). A practitioner committed to being trustworthy would not be designing tools to promote improper trust in this way.

But it also underscores various lessons derived in earlier posts:

• By pretending to apologise, chatbots do not create the right expectation for users.

• In their role as intermediary, by offering these superficially serious apologies, they are preventing real validation of harm and distrust, and do not carry a message to anyone who could act in response to this distrust.

• They effectively stop translation of trust friction into opportunities to improve.

• By pipiking apologies, they pipik trust and trustworthiness. At the very least, these affectations show that companies care more about the instrumental value of trust than they do about being trustworthy.

Because chatbots are so commonplace, they can seem mundane, inconsequential even. But what we see here is the moral importance of speech acts in solidifying trust. Indifference to the moral function of apologies is corrosive, not just to a given trust relationship between user and system, but to the idea that apologies have weight at all.

Specifically, I hope to draw attention to the ways that distrust of AI by the public is silenced by the dominant tech elite—how it is not “being recognized as meeting prima facie conditions for claiming” (p. 158). I explore the corrosive effect on trust of AI and propose further responsibilities of trustworthy AI practitioners.

Silencing distrust

One of the key features of trustworthiness according to Potter is responding appropriately to broken trust. Where someone makes a claim of injustice, the trustworthy individual listens to this claim and, crucially, takes it seriously by seeking to understand it from the claimant’s perspective and determine whether something ought to be done. This is uptake.

“…a society in which individuals can flourish is one where claiming of rights is possible, and receiving uptake is necessary to claiming… claiming cannot come off unless the audience is trustworthy with respect to the kind of listening and responsiveness that claiming requires” (p. 157).

History is rife with examples of failed uptake, where the testimony of certain individuals is suppressed or discounted for reasons relating to social status and power.

I am currently reading Unwell Women, by Elinor Cleghorn, which tells the infuriating story of how women’s claims about their own bodies and their own pain has been systematically silenced through millennia of androcentric medical practices, how such practices arose in relation to wider cultural misogyny and domination, and the devastating effects of this on women’s health. Women have never been seen as reliable medical witnesses. In part this is because they were historically excluded from institutions that were producing medical knowledge and denied access to knowledge that would enable them to contradict dominant beliefs about women’s bodies. And in part this was because of insidious beliefs about their character, worth, and social function. (Multiply marginalised women face the worst silencing, of course.)

There are many ways to silence. Denying a person the opportunity to speak is one obvious way—for example, when a husband is allowed to speak but the wife not. This kind of silencing is important, but not the kind of silencing that concerns me here. I don’t see overt silencing of this sort when it comes to AI. People are able to say how they feel about AI; the speech act itself is not denied. But this does not mean there is uptake of claims of AI’s untrustworthiness.

I will provide a very quick overview of the more covert silencing of such claims.

Perlocutionary silencing

This kind of silencing is the proverbial ‘falling on deaf ears’, where one can speak, but it has no effect, nothing changes:

“The silenced may indeed speak, even superficially be listened to, but the institutionalized context of the conversation, and the rules of the language-game, do not facilitate genuine dialogue” (p. 164).

Potter draws on Langton (1991), who explains this silencing as analogous to when a woman says “no” to sexual advances but her refusal is ignored—not because it is not heard, but because the man is powerful enough to act without consent.

I might not have chosen this analogy, but it is instructive. Potter explains how this kind of silencing is done “through bullying, ridiculing, mystifying, and intimidating” (p. 164); not dissimilar to when offenders silence their victims by telling them, “Go ahead and tell, but no one is going to believe you.”

I mostly see this kind of silencing when claims of AI’s untrustworthiness are derided as hopelessly naive—when people are told that they can complain, but AI is progress and it can’t be stopped. Don’t like what generative AI is doing to education? Well, tough, it’s here so you’d better adapt. Go to LinkedIn. This silencing is everywhere.

The result of this silencing is “perlocutionary frustration” (p. 157) of the claimant. This frustration is, as you’d imagine, poisonous to the trust relationship. Though I would need data to substantiate this, I believe that people with the most deeply entrenched distrust of AI have experienced perlocutionary frustration: they recognise that their distrust means nothing to the untrustworthy party as their power doesn’t hinge on a social mandate.

Illocutionary silencing

Illocutionary silencing is like if the untrustworthy party has selective hearing. (Continuing Langton’s metaphor, Potter describes this as “when a… women’s ‘no’ doesn’t even register as a ‘no’” (p. 164)).

“This is a kind of silencing that occurs when an utterance is prevented from counting as the act it was intended to be” (p. 157).

The reason claims are often not heard is when they are not expressed in accordance with the “rules of the language-games” of the dominant group. This occurs, for example, when social conventions constrain vocabulary (e.g. before there was a language for racism), how a topic can be talked about (e.g. which forms of power can be called out), and limit contextualisation (e.g. not seeing issues as intersectional).

Resisting this silencing is especially difficult because, to be heard, the silenced party is forced to “use terms, conceptual frameworks, and value systems that are not of their own choosing and that distort of falsify those attempts to communicate” (p. 165).

I see this kind of silencing when the public are expected to couch their claims of AI’s untrustworthiness in terms of explicitly prohibited violations of ‘Trustworthy AI’ regulation. One cannot easily communicate using the language of trust or distrust at all. One must be able to demonstrate algorithmic bias, disparate impact, that their right to explanation was violated, etc.

The “illocutionary disablement” this creates contributes to the pipiking of trust and trustworthiness [see my earlier post]. When the only expression of distrust that is heard is ‘I won’t buy your AI’, the enormously rich language of trust is reduced to reliance/adoption. When claims of untrustworthiness must be grounded in regulatory language to be heard, trustworthiness is reduced to a limited set of technical objections. These important terms for registering nuanced moral objections are rendered inconsequential, farcical even, to the point that one hardly dares express distrust at all, and seeks instead more useful language for expressing their dissent.

Mother-tongue silencing

This kind of silencing is similar to the above, in that it “is a result of differences in language where a dominant language is institutionalized” (p. 165). The difference, however, is that this silencing occurs when the speaker lacks fluency in communicating their claim.

“…when one is not fluent in the dominant language of the institutions of society, one is excluded from more than just ease or comfort: one’s ability to make claims about injustices, for example, will be seriously impeded” (p. 166).

This isn’t just because the claimant doesn’t know how to translate into the dominant language, but because in doing so they must also translate themselves into the dominant ‘world’ of which they lack familiarity and credibility.

I see this kind of silencing when people lack confidence in their ability to understand AI enough to make a claim about its untrustworthiness. But this is institutionalised practice: when technical communities discount distrust as stemming from a lack of understanding.

Non-natives of tech-speak do sometimes lack eloquence in their objections to AI. They might feel something is off, hence they say they distrust it. Because the language of trust is universal, the language everyone has for claiming injustice, technologists should be expected to speak in these terms, rather than the other way around, where “the burden of responsibility for bridging any barriers is arrogantly assumed to be that of the marginalized group” (p. 167).

“…in having to speak, not in one’s mother tongue but in the language of the dominators or the language of the fathers, one is coerced into modes of communication that exist primarily to serve dominant groups and function to maintain the status quo” (p. 167)

Share

Imitation-uptake silencing

This final kind of silencing is when one gives uptake performatively—doing the things to make us feel listened to, but lacking the behaviour that would accompany sympathy to the claim. Boundary spanners, who I touched upon in my previous post, can be sent out to fulfil this dark function.

I will provide a mundane example from my own life. Nearly 3 years ago I bought a new build house, and there remain some significant snags. After emailing repeatedly about these issues, and having been ignored for years, the company finally conceded to send out a representative to have a look. When she came, she was seemingly appalled by the state of the landscaping and other unfinished bits. She took photos, documenting the issues and legitimising our complaints, tutting and apologising. And then… nothing happened.

Potter compares this type of silencing to when men “claim to be sympathetic to feminism [but put] more energy into declaring themselves supporters of feminist concerns than into actually working to change the world” (p. 168). There is an obvious comparison to draw to the Trustworthy AI rhetoric: all tech companies profess to care deeply about trustworthiness, but do people really feel listened to? What’s actually changed?

Well, what has changed is there is now a legal apparatus around Trustworthy AI. And companies have done what they needed to be able to say that they are compliant. But as Potter reminds us,

“…one’s motivation might be to avoid professional or legal problems. The fact that a superficial kind of uptake can occur that can have little to do with taking seriously another’s claims or treating him or her with dignity points to the sense in which genuinely giving uptake and giving it properly requires the right motives and intentions and not merely the right behavior” (p. 168).

Giving uptake rightly

According to Potter, being a trustworthy person means “being the sort of person who gives uptake rightly.” She explains:

“This virtue facilitates understanding of what others care about, an understanding that is crucial to trust relations. It allows us to explore one another’s expectations, a process that helps avoid misunderstandings that lead to some failures of trust and feelings of betrayal. It affirms one’s good will and desire to engage in democratic processes and, in closer relations, to sustain connection. It allows for contestations of power, a feature I argued is central to democratic relations at every level of social relations” (p. 148).

This doesn’t require that one agree with the claimant, but rather that one takes seriously their reasons and makes a genuine attempt to “grasp what the world looks like from the other’s point of view” (p. 152).

For the sake of brevity, I will provide some bullets of what giving uptake rightly would look like for the AI practitioner:

• Reflecting on how language conventions around AI may be silencing people, particularly those who are already marginalized.

• Being on guard about one’s tendency to dismiss people’s claims of untrustworthiness. (“Those in relative positions of power, then, will take a somewhat suspicious attitude toward their own convictions about rights and harms” (p. 173).)

• Taking prima facie responsibility for the distrust of your system, and AI in general, by the public.

• Being deeply inquisitive about the reasons a person has for distrusting, particularly when they do not map to known (already legitimised) forms of untrustworthiness.

• Caring about distrust for the right reasons, i.e. because of the felt experiences of those claiming distrust.

• Not giving excessive uptake to authority figures on matters of trustworthiness at the expense of uptake to lay claimants. [Relatedly, regulators should not be giving disproportionate uptake to Big Tech by inviting them to write the rules on what makes AI trustworthy.]

• Providing fora for people to express distrust in their ‘mother-tongue’ of values and morals.

• Facilitating “dialogical openness”: being open to changing one’s “ways of seeing and being in the world” as a result of perspective-taking (p. 173).

• Demonstrating that claims have been heard, and what has been done to investigate the claim (even if ultimately no corrective action was taken).

• Relating actions taken to people’s expressed distrust, rather than to legal frameworks. [This prevents against what calls “somatic chill.”]

I would love if people could add to this list. Please do propose additional items or elaborations in the comments.

Leave a comment

This time I want to talk about features of institutional structure that complicate being trustworthy. By this I mean the particularities of doing AI work that make it difficult to act in ways that a trustworthy individual would. And I will attempt to resolve as much as possible the apparent trustworthiness paradoxes arising from these constraints by exploring what a person of trustworthy character ought to do.

The paradox of particularity

In an earlier post I presented Potter’s argument that an adherence to principles is not likely to promote deep trust. According to Potter, deep trust requires a particular sort of trustworthiness:

“a disposition that is responsive to others in their particularity and not just an impartial adherence to rules” (p. 6, emphasis added).

Indifference to this particularity breeds distrust because it is suggestive of a lack of empathy, a key ingredient of care that is central to trustworthiness. [In using the language of care I am drawing on Charles Feltman, for example, more than Potter.]

Scale

But how can the AI practitioner be responsive to people in their particularity? There are so many stakeholders to consider! For example, ChatGPT has something near 1 billion weekly users; and affected parties includes not just these users, but also non-users who are having to evolve their ways of working and/or whose very livelihoods are threatened by the tool. For some generative AI tools, affected parties also include invisible workers who label content. That’s a LOT of people.

So we have a real problem of scaling this key feature of trustworthiness: AI practitioners cannot take in all of the particularities of all stakeholders.

This is not really so difficult to overcome, however, if what matters is being the sort of person who “tak[es] seriously the reasons [a] person gives for holding her beliefs or values” (p. 152). Potter calls this “giving uptake”, and it will be discussed at length in a future post. For now, I will sketch what it might look like from the outside if someone does take such reasons seriously, and reiterate the importance of reflective practice throughout the development pipeline:

• Earliest/ideation stage: mapping stakeholders (all affected parties), developing personas and exploring their vulnerabilities and particular trust requirements in various domains of interaction.

• Development/pre-deployment stage: engaging with diverse stakeholders to learn what they don’t find trustworthy, and exploring how to actually meet the trust needs of the people expressing these views (including how to show that these needs have been met).

• Post-deployment stage (ongoing): specifically seek out individuals expressing distrust to understand how the product has fallen short of trustworthiness, and meaningfully respond to these people’s concerns.

Such actions “affirm one’s good will and desire to engage in democratic processes, …[and] allows for contestations of power” (p. 148). In short, I’m suggesting that the trustworthy AI practitioner engages with the particularities of the individuals expressing distrust by—and this is crucial—engaging with them emotionally, rather than merely intellectually (see p. 156). Distrust is a vital signal because it helps focus the AI practitioner on individuals—their feelings and why they feel the way they do, as it relates to their particularities.

Abstraction

A more fundamental issue is how abstraction obliterates particularity. Here I am concerned with how predictive AI treats individuals as instances of a type (“classifies” them).

In an earlier post, I explored the resentment that arises from AI mistaking us with our digital doppelganger, the version of us that is cobbled together from our data trails. I was talking about the kind of AI that makes a prediction about a person’s future behaviour based on an estimation of similarity to other people (superficial doppelganger similarity). In short, I argued that people don’t like being treated “as a type”. The resentment-tinged distrust that arises here could be understood, using Potter’s take on trustworthiness, as the consequence of a failure to be responsive to people in their particularity.

But is there anything that can be done about such a limitation? This is how AI works; classification is what creates predictive value.

I propose that there are both technical and non-technical (partial) remedies to this.

One technical remedy, as explored in that earlier post and elaborated at length in a paper I wrote with colleagues called Humble AI, is to recognise just how much information loss there is, just how much of the particularities are missing in the data, and to actively seek out more data points to build into the model. This is an attempt to feed in more of the particularities, as it were; even if it can never be complete.

But technical solutions only get us so far. The more important remedy is to recognise that the practice of classification is inherently political (see Kate Crawford, Atlas of AI). Crawford proposes reflecting on questions such as “How does classification function in machine learning? What is at stake when we classify? In what ways do classifications interact with the classified? And what unspoken social and political theories underlie and are supported by these classifications of the world?” (Crawford, p. 127).

Making such questioning a part of one’s practice gets close to the kind of trustworthiness that Potter describes:

“Being trustworthy requires (among other things) that we be committed to a certain picture of justice, and it requires that we see others in their particularity, not just as instantiations of a class or as members of a group” (p. 147).

Treating classifications as social constructions forces practitioners to consider the very basis for treating people as anything other than individuals in their particularity. This may be the best practitioners can do to be trustworthy given what AI is and how it renders meaning (and renders people). But it is not nothing! Someone on the outside who observes companies displaying greater sensitivity to the power of classification would undoubtedly view them as more trustworthy than those lacking this introspection.

The paradox of accountability

One of the central features of being a trustworthy person is being accountable to the entrusting party—that you have been responsible with their vulnerability, or, if you have not been especially responsible, that you have attempted to repair the damage done to the person and to the relationship.

The legal apparatus emerging around AI attempts to address the need for AI (companies, if not individual practitioners) to be held accountable for harms caused by the technology. But part of what feels so unsatisfying about these legal remedies is that they do not speak to what makes being untrustworthy so easy in the first place—i.e. the institutional features of AI that give rise to such moral complacency.

Invisibility

In my last post, I discussed invisibility as a form of power wielded by the AI practitioner: “it is very difficult for a person to identify who is responsible for what the AI has done to them.”

If we recognise the practitioner’s invisibility as making it harder to be fully trustworthy, is the solution being more visible? What would that even look like? What trust would be gained by making the work of any given individual—or not just their work, but something of the person, their character—visible to the user? I’m not sure this could even be managed from the user side. (I’m reminded of the stickers on soap bottles that says whose hands were involved in making that very bottle… A cute touch, but how many names would need to be listed for an AI product?)

I think there is a workaround here, if not a solution to this exactly. It is inspired by Jonathan Tallant’s recent post which talks about the crucial role of “boundary spanners”. These are individuals’ who act as intermediaries who do the crucial trust-building work with external stakeholders. In the absence of visibility of the practitioners themselves, boundary spanners become essential. They would do what’s known as facework (see my academic publication on this topic), essentially serving as visible representatives of the company, or as access points to how the company functions, to what it values. These individuals would provide some information (that can’t be provided through visibility itself) on the norms of those working for the company, i.e. the practitioners.

With trust, context always matters. And so it’s also worth pointing out that boundary spanners might be more important in particular domains, e.g. those contexts that require particularly strong bonds of trust. Potter points to the therapeutic relationship as one such domain, where a client needs to have strong trust in their therapist, as different to hospital emergency rooms where one can generally trust doctors and nurses (in a weak bond way) because they have trust in emergency room procedures.

I raise this to draw attention to the fact that AI can, and regularly does, bleed into areas that had formerly required strong bonds. [ChatGPT as therapist, for example!] Such areas would seem to benefit most of all from boundary spanners.

Repair

Perhaps here it is worth discussing some of Potter’s specific concerns about her area of practice, which is crisis counselling. One of the institutional features of crisis counselling is that “relationships are very time-limited, [which] may foster in counselors an attitude of moral complacency about the harms done to clients” (p. 55). If they don’t have to face the person again, they don’t have to reckon with the relational fallout of their untrustworthiness.

“…crisis counseling does not provide a means for clients and counsellors to later discuss and, perhaps, anguish over harm, betrayal, and accountability, and so the counselor is unable to mend a broken relationship. This failure, then, is an institutional feature of crisis counseling that influences our ability to be fully trustworthy moral agents” (p. 54).

I posit that AI shares this institutional feature, though arguably amplified by an original distancing, i.e. from never having had to interact with the people the model is making decisions about. How easy it is to be callous with an algorithm!

But this then raises the question of whether it is possible to repair broken trust, and what that would look like.

Here again, the boundary spanners can do some repair work where the practitioner can’t. They can express remorse for harm done, they can communicate lessons learned and actions that will be taken as a result. This isn’t about PR spin. People can see through that. This is about genuinely engaging with harms done, being fully accountable, and showing character: are you, and are the practitioners who make up this company, the types of people who care about the harms done? How can you show that to the people who were harmed so that they might be convinced that their pain mattered? This requires some real emotional intelligence, and a deep appreciation of the moral importance of trustworthiness.

The key takeaway, however—and one that Potter herself arrives at as it relates to crisis counselling—is that when institutional features make trust repair infeasible, it is even more important to be trustworthy. This, perhaps more than anything else, underscores the need for AI practitioners to do the work of cultivating a trustworthy character.

Leave a comment

The paradox of subjectivity

Finally, it is worth mentioning the ways that ideology shape institutional constraints. (Some constraints are not as real as others; some arise from a failure to imagine otherwise.)

I won’t launch into an extended critique of ideology here, but I do want to briefly touch on the shaping effect of faith in objectivity. This belief legitimises the entire pursuit of AI—this idea that answers will be found in data, that what matters is what can be measured, etc.

I propose that adherence to this faith can limit an AI practitioner’s trustworthiness insofar as it leads to the dismissal of fluffy stuff, like one’s own feelings, subjective experience, inner qualia. Being trustworthy, according to Potter, requires that one feels. She quotes Sherman, 1989, p. 47):

“Without emotions, we do not fully register the facts or record them with the sort of resonance and importance that only emotional involvement can sustain. It is as if our perceptions were strung together in our minds but not fully understood or embraced…the failure to feel is really a failure to record with the whole self what one sees. So, for example, when I fail to help another when I know I can and should, it may be that I see the other’s distress, but see it without the proper acknowledgment and sympathy” (p. 156).

Potter describes the antidote to this as “learn[ing] to see with the whole heart” (p. 156). For the AI practitioner, this means rejecting an ideology that negates the importance of subjective experience, and tapping into that which moves them to be morally virtuous… as fluffy as that may sound.

Share

Final thoughts

Admittedly, I suspect that none of these are true paradoxes. For my own sake, I needed to work through some of the peculiar features of working as an AI practitioner to understand what makes being fully trustworthy especially difficult, and what is nonetheless possible in terms of being trustworthy and building trust.

This is a work in progress, and I would greatly appreciate feedback. Drop a line in the comments if you have any thoughts.

Leave a comment

Sometimes, as I discussed, people’s trusts come into conflict and one must decide whose trust to keep and whose to betray. In this post, I will focus on those situations where trustworthiness to one’s institution comes in conflict with trustworthiness to others—specifically, when being trustworthy to the AI company or the regulatory apparatus conflicts with being trustworthy to AI users/subjects.

Share Practicing Trustworthy AI by Bran Knowles

A further requirement of trustworthiness

I am again using as inspiration the text How Can I Be Trusted? by Nancy Nyquist Potter. Last time I discussed three of her proposed requirements of trustworthiness:

• “That we give signs and assurances of our trustworthiness.”

• “The we take epistemic responsibility seriously.”

• “That we recognize the importance of being trustworthy to the disenfranchised and oppressed.”

This time, I will be concentrating on just one (number 6 in her list): “That our institutions and governing bodies be virtuous.” In exploring this particular feature, I am building on the idea that AI practitioners should prioritise being trustworthy to those most vulnerable to domination and exploitation (as per the final bullet point, above).

As Potter states, “a fully trustworthy person will exercise her agency, even under coercion, in a way such that she doesn’t decide to retain the trust of members of dominant groups at the sacrifice or neglect of members of nondominant groups who have placed their trust in her” (p. 30). Or stated more vociferously, “it is morally objectionable for us to appease those in power to the benefit of dominant structures and the detriment of the disenfranchised” (p. 84).

So how can an AI practitioner actually be trustworthy in this way?

This, I should note, is the question I get most of all from disheartened students: How can they be ethical if the AI industry is not?

The answer I will give is that it is possible and necessary for AI practitioners to exercise agency even when they are caught up in systems of oppression. They do this by cultivating a trustworthy character. This means leading by example, showing others what being trustworthy looks like, to promote healthy culture change. But it also means being aware of instances when trustworthiness comes into conflict with institutional standards and morality, and in such instances resisting the institution in order to create lasting change that supports greater trustworthiness.

Or in the words of Potter, “What I urge is moral decision-making that aims to end the violence of institutionalized injustices and inequalities, and this requires the cultivation of good character and good institutions” (p. 87).

Good character

To begin, there is an important shift entailed by virtue ethics as opposed to moral theory, as discussed at length in this previous post but which bears repeating.

We are used to moral questions being framed in terms of “which actions are the right ones to do” (p. 40). And there is comfort in being told what to do; it is reassuring to think that as long as one is following the rules, we “shouldn’t have to worry that [we are] making mistakes or hurting people or that [we] might be blamed for something [we] should have known better than to do” (p. 89). The problem is that moral life is complex, and rules do not provide a consistent basis for, or fully adaptable guidance on, good action. Given that trustworthiness cannot be adequately distilled down into a comprehensive guidebook on right actions, it is important to centre trustworthiness on character:

“…being trustworthy isn’t just a matter of doing the right thing but of being a particular sort of person, and to the extent that mainstream moral theory doesn’t worry enough about questions of character, it is inadequate” (p. 51).

An impassioned sidebar on education

This raises an urgent point about how we teach ethics in computing. I teach undergraduates in Legal, Ethical, Social, and Professional Issues, as it’s called. The lectures I do ensure that my institution retains its accreditation. But the British Computer Society accreditation requirements are very clear in how these issues ought to be taught, emphasising coverage of the codes of practice, the institutionalised norms of the computing discipline. Potter would be quick to point out that such training “typically displace[s] the issue of what it means to fail in someone’s trust” (p. 40).

Because I am me, I do also teach about trust and trustworthiness, though I fear the message absorbed by students is that the only thing they will really get in trouble for is a breach of the codes of practice. They’re not wrong, though, are they? People don’t “get in trouble” for being untrustworthy in these other ways. That does not mean, however, that there are not costs: these being distrust of products and, worse, the pain of moral injury.

I would also suggest that this peculiar professionalisation of the field contributes to frustration with and dismissal of distrust of AI.

• We are not teaching students that trust of AI is a character problem; so they cannot see as practitioners their own deficiencies of character that lead to distrusted systems.

• Nor are we teaching them that attending to relations of trust matters because doing so reveals “harms brought about by misuses or abuses of discretionary power” (p. 61); so they can’t see that distrust of their AI may stem from people’s distrust of AI-as-an-institution, and/or from AI’s role in enabling other oppressive institutions.

• And most of all, we are not teaching them to question the status of these norms and practices, to see them as situated in oppressive structures that exclude certain people from “having a voice in either criticizing or expanding the norms” (p. 49); professionals learn that their job is to defer ethical matters to the company’s legal/compliance team.

Back to what is demanded of character

If someone wants to be a trustworthy AI practitioner, this requires the cultivation of a keen sensitivity to the ways one’s own institution misuses their power; and above this, a willingness to do something with the moral discomfort this creates in oneself, to grapple with its inconvenient implications. This means:

1. Being sceptical of institutional authority. It is not sufficient to say that one is trustworthy because one has followed the norms and practices of Trustworthy AI. One should develop a habit of asking, “Whose interests was this norm or practice established to protect?”, “Who does following this norm or practice make me more trustworthy to?”, and “Who does it make me less trustworthy to?”

2. Reflecting on the powers the institution has, as well as the powers the practitioner has by virtue of being within an institution, and how they play out in trust relations. [This could be seen as a variation on taking epistemic responsibility seriously, as explored in the previous post.] Potter writes, “When we fail to address adequately harms arising from misuses or abuses of discretionary power within such relationships, we leave open the possibility that, in worrying about some harms, we are failing to notice other important ones” (p. 40). She cites Brody’s (1992) work on power within clinical settings, which highlights the types of power medical professionals hold over others. These include epistemic authority (i.e. that they are trained in the knowledge and techniques of healthcare), charismatic power (i.e. that they possess personal characteristics that can influence patient response, such as friendliness), and social power (i.e. that they are granted with decision making authority in medical matters).
   A similar accounting should be done of the power wielded by AI companies/regulatory bodies, and by any given AI practitioner. Presumably epistemic authority factors in, linked to the accreditation I critiqued earlier; as does social power in being granted the authority to design technologies with societal impact. There is also, above this, a special status bestowed on AI technologies (due to their geopolitical import) that is worth reckoning with. There is some power in the invisibility of the AI practitioner—sort of the inverse of charismatic power, whereby it is very difficult for a person to identify who is responsible for what the AI has done to them. And there would appear to be a kind of testimonial power arising from the hegemonic conventions of technical language, that only those who speak this language are entitled to opinions about AI. I would be delighted if people could add to this list in the comments.

   Leave a comment

3. “…pay particular attention to the ways concealment can be used to regulate power” in the institution. A person who is trustworthy will also have related virtues, like honesty. (I will discuss this at greater length in a future post.) It is important that one is honest in ones’ own communications, for example feeding concerns about untrustworthiness up the chain even when it might lead to blame or outrage. Equally, it is important to push back against deceptive communications which hide important concerns from the public so as not to provoke distrust.

Share

Good institutions

Still, there remains the frustrating matter of there being deficiencies in the institution itself that make being trustworthy difficult for the individual. Potter argues that “full virtue is difficult, if not impossible, to exhibit when the institutions are not themselves virtuous” (p. 55).

An AI practitioner who is committed to being trustworthy must actively resist the ways in which their institution is untrustworthy. At any given point, this institution could be the company one works for, the larger collective establishing best practice for AI, and/or the AI regulatory apparatus; and these institutions can be untrustworthy in the sense that they break trust (e.g. by causing harm), or in the sense that they do not allow for those working within them to be fully trustworthy.

I like the way Potter frames this dilemma as an opportunity for character building:

“Just as social structures may impede our ability to act in a trustworthy manner, so they can provide an alternative focus for our cultivation of a trustworthy character. We may not be able, in some situations, to do what we should with regard to someone who has trusted us, but we can do what we can toward others, and we can work to change the problematic social structures” (p. 56-7).

So what can AI practitioners do to resist this untrustworthiness?

1. Prioritise others’ trust. In certain circumstances, being virtuous might entail open or covert defiance of company policy if that policy breaks the trust of those most vulnerable. There is a risk, here, of being terminated; someone might reasonably decide that this strategy would be counter-productive to their overall aim, and I am not going to tell anyone that their practical considerations don’t matter. But there is a more subtle form of resistance that is easier to enact: to not only provide assurances to the institution, but to also provide assurances to those that are vulnerable. Insisting on having to supply these additional assurances may create greater awareness of institutional untrustworthiness.

2. Ask the awkward question. I saw that Microsoft recently named Trevor Noah as their Chief Questions Officer. I strongly support the function of asking questions, because questions spark realisation and change. But asking questions should be everyone’s job—it certainly shouldn’t be seen as the purview of a single, anointed individual. Every individual that makes a habit of asking “Why don’t some people trust our AI?” helps establish self-examination as the norm.

3. Build alliances. One of the scariest things about standing up to power is the thought of doing so alone. Potter offers a refreshingly practical suggestion that all AI practitioners can take up: to “prepare for those times when one may need to resist [by forming] alliances with co-workers and like-minded [individuals] so that one does not end up in the position of resisting without any support from others” (p. 87).

Final thoughts

I want to end this post with the importance of striving, rather than perfection. As Potter reminds us,

“We cannot expect too much heroism of one another. This world requires compromise and negotiation. But it matters to our moral character—and to the future of more equitable, more compassionate communities and institutions—whom it is that we are negotiating with, where those compromises are being drawn, and who is getting sacrificed as a result” (p. 88).

While we cannot be trustworthy to everyone all the time, when practitioners recognise the institutional pressures that make them less trustworthy, they are significantly less likely to appease the institution to the detriment of others. Once seen, it becomes possible to imagine small and big ways of resisting, of enhancing one’s own trustworthiness despite the institution, and of promoting change within the institution.

Share Practicing Trustworthy AI by Bran Knowles

Now I will turn to the task of articulating the features of trustworthy AI from a virtue ethics perspective on what it means to be a trustworthy individual, as argued by Nancy Nyquist Potter in How Can I Be Trusted?

Requirements of trustworthiness

Potter offers ten requirements of trustworthiness. I have grouped these together to explore cross-cutting themes across a series of posts. Here, I will explore requirements numbered 1, 2, and 7, respectively, in Potter’s list, all of which pertain to how power imbalances contribute to problems of trust and what that necessitates with regards to trustworthiness. I consider below how these requirements might be incorporated into the practicing of trustworthy AI.

“That we give signs and assurances of our trustworthiness.”

Potter describes trust as a “process of induction”, where one projects into the future what one anticipates happening based on prior experience. This means that those with more experience of broken trust—such as those oppressed and marginalised by inequitable systems, and/or those with relational trauma—will find it harder to assume trust in the future.

Such “learned strategies of distrust” (p. 19) are rational, even if they may not generalise correctly to every context. (Sometimes trust is not broken as anticipated.) Seeking to change this perspective, however—as one might when focused on supporting “reasonable risk-taking” of the trustor—is an act of erasure, a denial of that person’s reality. Greater trust is not a realistic solution for individuals who know all too well the likelihood of their trust being betrayed. And so, the very fact that individual dispositional differences in trustingness are shaped by “cultural, material, and ideological forces… highlights the importance of shifting questions of trust and distrust to those of trustworthiness” (p. 19).

Potter also argues that because expectations of broken trust can arise from real patterns of exploitation and oppression, the burden is on “persons with more privilege or power” to do the work of “overcom[ing] the disenfranchised person’s disposition to distrust” by, first, being deserving of trust, and second, “[giving] assurances which indicate a trustworthy character” (pp. 17, 20).

In my academic writing, I have tried to sensitise those working on the “problem” of trust of AI to the idea that distrust of AI is a legitimate response to the structural violence it amplifies, accelerates, and sanitises. We attend to trust when we attend to this structural violence—i.e., to the ways that AI is untrustworthy, to some more than others.

But it’s also important to point out the paradigmatic feature of AI, that of it being wielded by the powerful over the less powerful. I have noticed a kind of exasperation at people who distrust AI, and little acknowlegment of the burden of responsibility in this relationship: that distrust reflects a failure to show people that/how AI is trustworthy.

There’s no recipe for how to provide these signs and assurances, because what you’d need to provide depends entirely on the particular vulnerabilities pertinent to a given AI. But what this trustworthiness requirement teaches us is that being trustworthy is not enough to earn trust; trustworthiness must be exhibited.

Share

Leave a comment

“That we take epistemic responsibility seriously.”

We might say that properly dis/trusting requires of the trustor that they are suitably curious and make an effort to understand what makes a potential trustee trustworthy or not. If we are centring questions of trustworthiness, however, we would recognise that “being trustworthy also requires epistemic effort” (p. 27). According to Potter, this is chacterised by “active engagement with self and others in knowing and making known one’s own interests, values, moral beliefs, and positionality, as well as theirs” (p. 27).

The implication for those developing AI systems is two-fold:

1. Reflect on one’s own privileges to avoid reflexively reinforcing inequitable dynamics and perpetuating distrust. Here we might directly adopt Potter’s suggestion to consider: “how does one’s situatedness affect one’s relation to social or economic privileges? How do one’s particular race and gender, for example, affect relations of trust with diverse others? In what ways do one’s values and interests impede trust with some communities and foster it with others” (p. 27)? Such questions might lead, for example, to AI systems that are much more cautious in their categorisation of people due to an awareness of other systems of domination that individuals are subjected to.

2. Represent values and beliefs explicitly in the interface. [Shout out to for this portion, as I draw on the exchange we had in the comments for an earlier post.] There is a tendency, when treating the earning of trust as a PR exercise, to find ways to soften negative reactions to systems. But being upfront with users about the various trade-offs implicit in the design of the system, and how a particular problem is being conceived and addressed from the standpoint of the individuals involved, creates helpful “trust friction.” Trust friction surfaces the specific requirements for being trustworthy (what it is that users are not comfortable with), and assuming a company is reasonably responsive to distrust as a negotiation signal, this will move the company/system towards improved trustworthiness over time. Being explicit about these trade-offs also makes those developing these systems vulnerable to appraisal by those responsible for regulating AI, creating additional incentive to be trustworthy in the myriad (if still limited) ways that are legally actionable.

“That we recognize the importance of being trustworthy to the disenfranchised and oppressed.”

Trustworthiness is situated in relationships, which means that it is possible to be trustworthy to some people and not others. A person who is committed to the virtue of trustworthiness will face moral dilemmas where they must choose whom to be trustworthy to and whose trust to betray. Potter’s position is that in such instances we ought to “take as a primary consideration those who are already vulnerable in relation to dominant structures, in general, and to us, in particular” (p. 29). This is in keeping with an understanding of trustworthiness that is characteristically “nonexploitative and nondominating” (p. 29).

Again, being oriented toward trust (rather than trustworthiness) would have us make a different calculation. Typically it is those who are relatively privileged that have true discretion in their adoption of AI, whereas those less privileged tend to be forced into relationship with AI, in many cases with systems being used on them to exert control. But the tendency to view the ability to exercise distrust (i.e. to reject AI) as a reason for attending to distrust contributes to the testimonial injustice oppressed people already experience, with some people’s claims of untrustworthiness taken seriously but not others’.

For trustworthy AI, this suggests:

1. The importance of reflecting on how “whose complaints are taken seriously intersect with social and cultural markers of worth and difference” (p. 77); i.e. whose distrust of AI are we responsive to, and why?

2. The need to cultivate genuine caring for the concerns of those with the least discretionary power.

The fact that disenfranchised people may more readily give over trust (well, reliance) to AI due to practical constraints—particularly in light of their increased likelihood of dispositional distrust (see the first requirement explored in this post, above)—should focus practitioners on how they might be trustworthy to these people most of all.

Leave a comment

Final thoughts

I want to end by echoing the brilliant Ruha Benjamin. In her recent Tanner Lecture, she argued that while AI purports to promote efficiency and progress, we should train ourselves to ask the questions, “Efficient at what? Progress for whom?”

Likewise, I want to propose that the overarching lesson of this post is that when we hear the phrase “trustworthy AI”, and when we are tempted to use it ourselves, we need to get better at asking, “In what way is it trustworthy?” and “Trustworthy for whom?”

Sometimes concerns about protection of other things one values enter in: do I believe that the information I provide to the AI in order to make it work will be handled with care, that my privacy will not be compromised, that I will retain my agency, etc.?

Sometimes trust of an AI is coloured by one’s feelings about other AI: can I trust that this AI is different to AIs that I have seen cause harm, do I think AIs are implicated in the enhancement/diminishment of certain things I value, etc.?

But sometimes it’s fundamentally about character: do I believe that those developing, deploying, and regulating AI care about what matters to me, and that they can be counted on to protect these things because that is the sort of people they are?

For this post I will introduce my thesis that insufficient attention to trustworthiness as a virtue contributes to distrust of AI; and that trust is supported at all of these levels when we refocus “trustworthy AI” on the individual moral responsibility of those developing, deploying, and regulating this technology.

I will be drawing on the book How Can I Be Trusted? by Nancy Nyquist Potter as I begin to articulate a virtue theory of “trustworthy AI”. This will scaffold my approach to trustworthy AI as a practice, i.e. a way that AI practitioners show up in the world.

The neglect of trustworthiness

Potter’s book is remarkable for its soul-searching examination of what it means to be trustworthy. She notes not only the surprising lack of philosophical engagement with people’s lived experience of trust (typically messier than game-theoretic or contractual views allow for), but also a preoccupation with the avoidance of betrayal of trust. This tends to prompt questions of who to trust, on what basis, and with what assurances, all of which are oriented toward “reasonable risk-taking” of the trustor. What is missing from this are important questions about “responsibility-taking” of the trusted party. Potter observes that, especially when it comes to those in positions of power, people rarely ask themselves the self-examining question,

“Why don’t some people trust me?” (p. xi).

It might appear that trustworthy AI is an exception to this general neglect of trustworthiness, in that a whole collective of researchers, practitioners, and policymakers have busied themselves with the elaboration of principles that should be followed and metrics that should be used to evaluate the trustworthiness of AIs. But predetermining principles and metrics of trustworthiness shows a concern with pre-empting (only) the distrust-provoking matters that the technical elite deem valid. This is different from a true self-examination.

And the absence of true self-examination has consequences. Currently, there is little understanding of how any of the things we can measure about an AI’s “trustworthiness” might translate into trust because technical folk assume they know (or know better). As one might imagine, this can lead to the coming apart of “trustworthy AI” from actual trust of AI.

In contrast, when you treat any and all reasons for distrust as valid, and open yourself up to a process of self-examination, you naturally ask the key question: “How can we actually meet the trust needs of the people that are expressing these views?” This reflects the disposition of someone who takes their responsibility to be trustworthy seriously.

Share

The inadequacy of a rules-based AI ethics

Alas, I have given up trying to follow all the developments in AI regulation. It’s exhausting, confusing, but also depressing. The principles they supposedly enshrine are always still vague, and the regulations appear to do more to protect AI companies from having to be decent than they do to encourage decency. They are, after all, developed with Big Tech at the table, and are designed to help these companies make money in the hope that their wealth trickles down to the rest of society. (Is that too cynical?)

I actually think regulation is important, and has a place. But fundamentally, what I think is misguided about pinning so much on a regulatory ecosystem, at least when it comes to promoting trust in AI, is that it is being built in an attempt to bypass the crucial matter of trustworthiness.

Deep trust

Trust can be deep, or it can be shallow. The difference is the extent to which you think someone is committed to being trustworthy. As Potter explains, you may trust that you will be well-treated because you understand that certain sanctions prohibit someone treating you badly. Yes, this is a kind of trust; but if you’re assuming this is the only reason for someone treating you well, you have to be vigilant against trusting them outside of these very particularly defined conditions.

A deeper form of trust is predicting that you will be well-treated because you know that the other person has good will toward you (they wouldn’t want to harm you). You could assume, here, that even when an opportunity presents itself to act badly without sanction there is little appetite to do so.

But the deepest form of trust, Potter argues, is when “a prediction of being well-treated… is grounded in a belief not only in the other’s good will toward oneself but in a belief that the other’s good will is part of a more general disposition that extends beyond the context of this particular relationship” (p. 5). In other words, the reason your trust is so deep, and can be extended to the widest set of contexts, is because being trustworthy matters so much to this person that they would remain trustworthy even when it wasn’t in their interest to do so, because being trustworthy is the thing that matters to them most.

AI regulation is a cat and mouse game

How much do you trust that regulators understand the technology well enough, and can move fast enough, to protect you from harm? Assuming the mouse is determined to do whatever it can to get the cheese (shout out to and her recent post), and is willing to cause damage in the pursuit of this goal, it’s impossible to adequately constrain its untrustworthiness.

If what is taken for granted here is that Big Tech cannot be expected to have any good will toward people, that the best we can hope for is that we create enough mousetraps to catch them when they are tempted, then there is no grounds for trusting the AI companies as they innovate and act in the world… even if one might sometimes be able to (shallowly) trust a given AI. (Though I would argue that if you really felt the company was untrustworthy, trusting the product is probably unwise. Tesla car panels falling off on the highway would seem to illustrate my point nicely.)

Share

Commitment to principles is not enough

Potter’s concern is not so much with regulatory constraints as with the limits of deontological ethics which would seek to define principles for good action. She argues that believing someone is trustworthy because they are committed to a set of principles falls short of “what we tend to look for when evaluating whether or not we can trust someone”… namely their trustworthiness, “that is, a disposition that is responsive to others in their particularity and not just an impartial adherence to rules” (p. 6). She adds, “An attitude of indifference to particular persons does not foster a great degree of trust even if ‘right actions’ are performed” (p. 6).

What she is drawing attention to is something too often missed—and its missing is a source of distrust for so many with lived experience of marginalisation—that building an ethics around universal principles (as in the guidelines for ethical AI by the European Commission’s High-Level Expert Group on AI) tends to be “accompanied by an indifference to feelings for others or an impersonal and impartial stance” (p. 6). And the reason this is harmful is because “the impartial point of view ‘masks ways in which the particular perspectives of dominant groups claim universality, and helps justify hierarchical decision-making structures’” (p. 6, citing Young, 1990).

And this is precisely what I was getting at when saying that “trustworthy AI” is a case of being tuned in to respond only to the set of concerns that tech elites deem valid. I have argued elsewhere that those creating such principles suffer from ignorance of their perspective on what makes AI trustworthy as being a particular view, specifically a “view-from-the-top”:

“from this vantage the benefits of AI are clear and the harms of AI are abstract, largely separate from lived experience but a potential threat to an ambitious technology agenda. This view-from-the-top has thus informed highly technocentric principles for ‘trustworthiness’ which… obscure and facilitate structural violence by shifting attention away from how the very foundations of AI are inherently extractive and prone to reproducing and, at the same time, amplifying extant inequitable social structures through the logic of categorization and simplification...” (Knowles et al., 2023).

So while the presence of principles is better than nothing—it indicates an understanding that there is a moral responsibility to work through and is more likely to lead to a prediction of being treated well—what Potter suggests, and I’m inclined to agree, is that “it is not likely to be trust at a very deep level if the good treatment is based on a commitment to universal principles and accompanied by an indifference to feelings for others or an impersonal and impartial stance” (p. 6).

What I have come to suspect is that, while in the short term companies may benefit from shallow trust, the longterm consequence of inattention to deep trust is that feelings of discomfort accumulate in the psyche over time and start to appear in ostensibly less “rational”, and more affectively loaded, totalising distrust of AI. We feel uneasy without the assurances of knowing we are taken care of by others in this world; we recognise that what has changed around us is the emergence of AI; so we distrust AI. It’s a working hypothesis. Thoughts welcome!

Leave a comment

A virtue ethics of trustworthy AI

The position that Potter arrives at is that trustworthiness is a matter of character. When determining whether someone is trustworthy, we are attempting to “reach a sense of the whole person” (p. 2; quoting Govier, 1991). While it might be more correct from a risk-avoidance perspective to take care to specify in which contexts a person can or can’t be trusted, we tend, instead, to have an attitude toward a person based on our sense of how we imagine they tend to operate in the world. But I don’t think this is laziness, exactly; or even wrong. Often we don’t know the full extent of what we are entrusting to another. (Consider, for example, how difficult it can be to comprehend precisely how it might matter to one’s privacy to consent to certain terms and conditions; an issue I have written about elsewhere.) We make judgments about character because this actually provides a reasonably good basis for predicting good behaviour even in contexts we haven’t anticipated arising.

To be able to judge trustworthiness—“that she can be counted on, as a matter of the sort of person she is, to take care of those things with which we are considering entrusting her”—we must know something of the person’s “values, commitments, and loyalties” (p. 7).

So… what does Big Tech stand for?

Eugh.

It matters (to trust of AI) that the public faces of so many of these companies are, frankly, so loathsome. Consider this exchange between Mark Zuckerberg and a friend:

Zuck: Yeah so if you ever need info about anyone at Harvard

Zuck: Just ask.

Zuck: I have over 4,000 emails, pictures, addresses, SNS

[Redacted Friend's Name]: What? How'd you manage that one?

Zuck: People just submitted it.

Zuck: I don't know why.

Zuck: They "trust me"

Zuck: Dumb f**ks.

I don’t think I really need to convince anyone that the values of tech billionaires are... skewed, shall we say, toward profit over all other values. The issue is that it becomes impossible to trust their companies deeply when we know that our interests will come up against, and inevitably lose out to, their quest for profit.

Moreover, we are aware of the power these companies hold over us. (Is anyone entirely comforted by regulation?) Inconveniently, the fact that this relationship is so asymmetrical is precisely why character matters so much! As Potter reminds us, “…when differences in privilege and power exist between us, we may be uneasy about what each other cares about: each sees that the other values some things which she or he sees as either incompatible with or hostile to the things she or he values. Hence, the emphasis is on how willing and able one is to care for those goods others value even when those are not, or do not appear to be, entirely harmonious with the goods one values oneself” (p. 12).

In other words, to trust AI, we have to trust that those responsible for it are committed to being trustworthy as a matter of character because they recognise their responsibility to care for us.

Living in “reality”

There is no point fantasising that tech billionaires will “see the light” and decide that they must commit to being trustworthy. But I do think there are things that can change without this.

First, the way we frame and talk about trustworthy AI matters. It sets a moral expectation. When we talk about trustworthy AI in the sterile language of principles and metrics, we invoke the rational brain, implicitly conceding that such matters are subject to reason. How different it would be to assert that such matters are beyond reason, that the importance of being trustworthy is not up for debate! This is about being a decent, trustworthy person in the world because that is everything. And we are a society that will hold people to that expectation. We don’t celebrate those who get to the top by being untrustworthy; we shame them. (This is the metaphorical stick we might use… if they care.)

Couching trustworthy AI in the language of our emotional brain is what can create the emotional motivation to “behave in ways that are ultimately in our interest and the interest of those within our sphere of care or concern” (to quote Andrew Westin from his book The Political Brain, 2008; p. 102). (Carrot + stick now.)

Additionally, it invites people who do feel distrustful to tap into their emotions, potentially contributing to a groundswell of public pressure to be better. (Carrot + double stick.)

Still, I sense many will be unconvinced. So I’ll give another reason it matters: these billionaires are at the top, but they are not the ones doing the work. There is so much that individuals within any tech company can do when they are committed to being trustworthy. I will focus on them, because it is in them that I find hope for change.

The shift

So what I am proposing is a need for a radical reframe of trustworthy AI in terms of virtue ethics, i.e. to focus on the ways that AI practitioners can cultivate trustworthiness and embody that character in their work. This, I suggest, will ripple out into improved trustworthiness and greater trust in the AI products being built (not just that they work as expected, but that they work in ways sensitive to our various vulnerabilities—the things we value that are entrusted to AI). But it will also begin to instil trust in the culture from which AI springs, through a sort of morality-based professionalisation of the sector, creating a deep trust that spills over into greater trust of AI as a class of technology.

For now, I will provide a mere sketch of what is distinctive about this framing:

• It “puts dispositions, and not rules, at the center” (p. xii).

• It demands that we aspire to being “fully trustworthy” (p. xii). This means:

  • not confining our trustworthiness with respect to some specific good(s);

  • not treating certain goods as supererogatory (“it is morally permissible to do x and it would be good to do x, but it would not be morally wrong to not do x”).

• It entails engaging in the reflective practice of considering “ways in which each of us can enhance proper trust and ease pervasive distrust” (p. 2).

• Practicing trustworthiness in this way is the habit of those who have committed to cultivating a trustworthy character.

• This trustworthy character provides a basis for deep trust of AI.

Conclusion

To keep this reasonably digestible, I have stopped short of describing the features of trustworthy AI when we treat trustworthiness as a virtue. This will be the subject of the next several posts.

For now I hope to have provided a reasonably compelling argument for focusing more seriously on what it means to be trustworthy, and demanding that practicing trustworthiness is integral to the development, deployment, and regulation of AI.

Today I wanted to write something a little more tangible (less indulgent, perhaps): a set of insights to inform AI practice based on what I explored through the lens of distrust of the doppelganger.

1. Validate, and act in response to, the feeling of distrust

The highest level takeaway from these discussions is that, while trust may at some times be best understood as a belief predicated on rational modes of thinking, trust is also (and I’d suggest, primarily) an affective attitude. Failure to appreciate this emotional component is what can lead to the appearance of an inexplicable coming apart of trust and trustworthiness in the AI space, where people appear not to trust AI systems that meet the specified criteria for trustworthiness in terms of metrics and assurance processes.

But where, for example, do the metrics and processes used to ensure a system’s trustworthiness attend to disgust, resentment, or anxiety? Metrics, at least when presented to the user as evidence of trustworthiness, are the language of belief-based trust, not of emotions. So, too, is process when undertaken principally to demonstrate compliance. Perhaps the reason so many people feel such an intense distrust of AI comes down to a felt sense that those developing and advocating AI lack inquisitiveness about their feelings of distrust; that they don’t see them as valid, but instead view them as a misfiring of logic.

Negative emotions are a body’s protective mechanism for responding to threat. This means that those developing AI systems should treat distrust as a signal that the AI system is threatening in some way(s). In practice, this looks like a) doing something to meaningfully address the threat; b) communicating that the threat has been taken seriously and what has been done about it; and c) checking back in with the feelings to see whether it has stopped alarming.

Subscribe now

2. Create the right expectation, expose the (true) merwelt

One of the things we should see if were are taking emotions seriously is that people are weirded out by technology that presents itself as “you, only better”. This is a threatening premise to begin with!—one that potentially invites defensiveness, too, which I did not yet discuss.

The main trust issue, however, is that this premise is a lie. The attempt to mimic intelligence, to hide the machinic processes, is deceptive. Catching someone in a lie can destroy trust; here, the mask is bound to slip, the lie exposed, and one sees that the machine is “intelligent” only in a very limited (disappointing) sense. But what Richard Harper is really arguing is that AIs designed to mimic intelligence are setting users up for the wrong interactional dynamic—a kind of wrong expectation that can only lead to distrust when the AI does not do what one anticipates.

He proposes a solution that begins with abandoning the guiding metaphor/aspiration of Artificial Intelligence: encouraging users to think of what happens inside of an AI system as merwelt, rather than consciousness or intelligence. “[W]e need to make our consciousness of AI different because AI is other in a variety of ways, and distinct from us in terms of what its consciousness might be, and indeed so different that the term consciousness might not be helpful… The more we recognize that there are myriad types of AI and myriad merwelts the less will we be fooled into thinking its ‘understandings’ are a version of our own” (p. 22).

There are ways designers can expose the merwelt (Harper provides several suggestions for doing so in ChatGPT)—after all, its concealment has been an intentional sleight of hand, an attempt to make AI seem more magical than it is. But hype is profitless in terms of trust, both because it misleads the user in how to succeed with the technology which leads to performance breakdowns, and because those pushing demonstrably false narratives come to be viewed as snake oil salesmen.

Instead, helping users set realistic expectations for AI as a tool, with grammars of action that can be mastered for defined ends, creates better conditions for trust to grow. This work can be understood as the practice of embodying “rich trustworthiness”, that is, clearly signalling for which tasks/ends a person can/not rely on AI.

Share

3. Create trust friction to surface opportunities to improve

I am definitely not suggesting that the merwelt is innocuous and that exposing it would allay all distrust. I’m saying it’s at least an honest beginning for, not just better human-AI interactions, but also for a conversation about what we value and how that gets embedded in our technologies.

What my musings on the doppelganger have attempted to show is that some of what we are reacting to is the exposed logics of a dark ideology. Particularly alienating, I would suggest, is its implicit asserting of the supremacy of the values of efficiency and optimisation, and its culminating in the sterile sorting of people into categories of worthy vs unworthy.

While for practical reasons I might like to get painful things done as efficiently as possible, I wouldn’t rank efficiency higher on my list of values than things like compassion, honesty, care, connection… Really, it’s towards the bottom of my list, at least when I am in a reflective (rather than panicked) mode.

Optimisation, on the other hand, is inherently threatening to me and so many others who would be subject to culling. In my case, having a disability is a liability in a culture hellbent on the project of total optimisation. As Klein herself says in exploring the darkness of optimisation, “The very idea that humans can and should be ‘optimized’ lends itself to a fascistic worldview… If you are safe because your immune system is strong, it can flip to mean others are unsafe because they are weak. If you are optimized, others are, by definition, suboptimal. Defective. Next door to disposable” (p. 187).

What happens when the merwelt is exposed—whether intentionally or through slippage—is that it jolts a reaction. When that reaction is distrust, what follows should be real debate about the provoking matter. We can, for example, begin to discuss the merits of efficiency, of optimisation, of sorting when we are invited to see that this is what the AI is doing. Those developing AI systems should not attempt to soften reactions by hiding what is at stake—after all, do they believe in what they are doing, or do they not? If, ultimately, what will cause distrust of your AI system is the logics upon which it based, better to deal with that early in the pipeline than try to massage distrust later on.

Subscribe now

4. Open up the work of imagining as a collective project

Now, to throw a spanner in the works, I don’t believe that Big Tech considers the ideological premises of AI to be up for debate. I think this in itself contributes to a deep yearning for things to be different, far beyond trust friction. Interestingly, according to Freud, it is this kind of yearning itself that underlies the enduring motif of the doppelganger. I’ll quote Klein again:

“Freud speculated that the figure of the doppelganger recurs in the culture in part because the idea of there being duplicate selves stands in for the vast potentialities that our lives hold. We are the product of choices—made by us, and made by others. But, Freud wrote, those never are the only choices available. There are also ‘all the possibilities which, had they been realized, might have shaped our destiny, and to which our imagination still clings, all the strivings of the ego that were frustrated by adverse circumstances, all the suppressed acts of volition that fostered the illusion of free will’” (p. 334).

I am suggesting that so much of our distrust of AI arises from the frustrated strivings of the ego: despair at the shape of AI as it has come to be when compared against the limitless possibilities of human imagination, and a loss of hope that it could be any different given our current techno-politics. In this context, distrust is one of the few forms of dissent available to us, and it expresses a multitude of disappointments and fears.

Share

If you have taken away any new insights that I’ve not listed here, go ahead and let me know in the comments!

Leave a comment

One more lesson from Doppelganger…

In her book, Klein studies the different ways the theme of the doppelganger have been explored in pop culture. She returns repeatedly to Phillip Roth’s book, Operation Shylock, pulling on a number of its threads, but in particular borrowing from it the term “pipikism”. The term derives from the Yiddish word for belly button, but in Roth’s words refers to “the antiragic force that inconsequencialises everything—farciacalizes everything, trivializes everything, superficializes everything” (p. 145).

This, Klein sees, is what Far Right politics does to the issues that matter to the Left. This happens when, for example, the Far Right calls everyone and everything “fascist”. Fascism has a specific, serious meaning, but it has been appropriated by the Far Right to mean little more than “a person who is against us”; and eventually, serious, scholarly people who really do mean fascism find themselves backing away from using the word for it now being tainted by its association with the lunatic fringe. She asks, “Once an idea has been pipiked, can it ever be serious again” (p. 145)?

I raise this because this is what appears to be happening to the words trust and trustworthiness now that they have been commodified by the tech industry. I find myself on the defensive, having to begin my serious inquiries about trust and trustworthiness by explicitly creating distance from “trustworthy AI” as it is wielded in technical and regulatory circles. No, I don’t mean the set of vague principles or spurious metrics, I mean ‘Is AI worthy of trust’! And the same is true with trust. Whereas industry would seem to be satisfied to equate use of an AI with trust of an AI, I want to scream, That’s reliance! Do they TRUST it??

Trivialising trustworthiness is a great way to get people to think you are not seriously invested in earning their trust. And trivialising trust signals an even deeper disconnect, a true flaunting of power—effectively communicating, “you need us, we don’t need you.”

So if I could add a bonus lesson to this list, it would be this:

Share

Next time

This concludes me constantly talking about Naomi Klein. Next week, I will begin a new multi-part series exploring what it means to develop a practice of being trustworthy, reflecting the book How Can I Be Trusted by Nancy Nyquist Potter into the world of AI development.

So far I’ve argued that some of our distrust of AI takes the form of disgust at encountering a technology ostensibly made in our image, but which in doing so reflects back to us aspects of ourselves we may not like to stare in the face. I have formulated this as distrust of AI-as-doppelganger.

I then argued that some of our distrust of AI takes the form of resentment at AI mistaking us for our “digital doppelganger”. This version of ourselves is “flattened, reduced”, which as Naomi Klein argues, is “easier to confuse with a flattened, reduced version of someone else” (p. 41). So we become indignant when that misrecognition leads to unfairly adverse decisions about us by AI; but perhaps, too, we resent the flattening/reduction that enabled the confusion in the first place.

Here I will come from yet another angle to help get to grips with anxiety-driven, conspiratorial distrust of AI. Though it lends itself less easily to a pithy formulation, my thesis is that these fears are displaced anxieties about capitalism.

It’s easier to fixate on AI, to see it as the cause of the anxiety we are feeling, than to confront our anxiety that we are implicated in the horrors of capitalism. In doppelganger terms, we are displacing anxiety about the actions of our “second body” which reaps the rewards of pain sown in the shadow realms of capitalism.

This requires some explaining… hang in there with me!

Getting the feelings right

A major component of Klein’s book, Doppelganger, involved trying to make sense of the paranoic, unhinged chaos of what she calls the Mirror World. This Mirror World is the sinister double of liberal politics, the fascistic flip that has been gaining momentum across the globe. In observing the tactics of folks like Steve Bannon, Klein discerns a strategy for building alliances and converting followers. The Bannon playbook is to pay attention to the issues that liberals should be caring about but are either ignoring or neglecting, and then claiming these as issues owned by the Far Right, creating a safe haven for the disaffected. Klein goes on to say, “[Bannon] is also taking note of more subtle failings—the way issues are discussed, the way disagreements are negotiated, the way people are treated by their friends and comrades. Through the one-way glass, he is studying all our hypocrisies and inconsistencies so that he can make a show of doing the exact opposite” (p. 126).

The genius of this strategy is that it capitalises on people’s anxiety by (merely) legitimising the feeling. Of course, the Far Right is no more likely to offer a genuine solution to the problems creating the anxiety. But people feel heard, which is better than being gaslit by those who are supposed to be fighting for these issues.

There is another Far Right tactic at play, which I will call “Look over there!” The angry hoards they lead are crying out for someone to do something about problems created by neoliberal economic policies and/or unfettered capitalism and its perverse incentives. These are not things the Far Right has any intention to address, as they provide the party’s ultimate powersource (i.e. disproportionate spending power + increasingly angry hoards). So their trick is to redirect anxiety to culture wars issues (p. 285). The reason everything becomes so twisted in the Mirror World is because of the strategic severing of feeling from fact—what matters is that people are riled up about something that is not the thing (capitalism).

The conclusion Klein reaches is that all of the bizarre conspiracy theories lighting up Right Wing discourse are, she quotes from Gilroy-Ware, “a misfiring of a healthy and justifiable political instinct: suspicion” (p. 243). Conspiracy theorists, she summarises, “get the facts wrong but often get the feelings right—the feeling of living in a world with Shadow Lands, the feeling that every human misery is someone else’s profit, the feeling of being exhausted by predation and extraction, the feeling that important truths are being hidden” (p. 243).

The “second body”

What fascinates Klein is not really the political machinations that spawn the Mirror World, but rather the psychology of avoidance, the things humans do to avoid seeing “what we cannot bear to see—in our past, in our present, and in the future racing toward us” (p. 322). What makes conspiracy theories so enticing is that they divert anxieties to the doings of a definable ‘other’. The doppelganger formulation Klein favours in the book is that all the work of avoidance (performing, partitioning, and projecting) is to save us from having to face our “second body” (our “true doppelganger”, as Klein calls it): “the one enmeshed with wars and whales, the one benefitting from the genocides of the past and adding our little drops of poison to the great die-offs of the future. The second body that perpetually mines the Shadow Lands for its comforts and conveniences. We avoid because we do not want to be bodies like that” (p. 322).

Finally getting to the matter of distrust of AI

I want to propose that a great deal of distrust of AI is not about the technology, per se.

It is easy to mistake distrust as an objection to this or that tool, or this or that technique. Individuals may even convince themselves that AI is the cause of their anxieties, without recognising it as the technical embodiment of an existentially threatening ideology about which we are justifiably anxious.

I am not meaning to suggest that AI is perfectly harmless. AI causes real, direct and indirect harms to people; and many of these harms are inflicted in the Shadow Lands through the digital poorhouses that use algorithms to manage and punish the poor (see Automating Inequality, by Virginia Eubanks), or on the invisible dataworkers of the Global South we traumatise, or on the people (also disproportionately living in the Global South) ravaged by climate change exacerbated by a rapacious appetite for data and a misguided race to maximise computer processing power (see Atlas of AI, by Kate Crawford). These horrors are not unique to AI, however; they reflect the very predation and extraction required of modern capitalism. What is scary about AI—the feeling that we get right—is that it is the current tool for entrenching and scaling up a deeply harmful capitalist logic.

Flaunting the mechanics of the oligarchy

It is also deeply relevant to our anxieties about AI, I should think, that Tech Billionaires are our culture’s most visible representation of the inequities in capitalist societies. Klein writes, “In every case, they take up the mantle of solving the world’s problems—climate breakdown, infectious diseases, hunger—with no mandate and no public involvement and, most notably, no shame about their own central roles in creating and sustaining these crises. Knowing that this kind of unmasked plutocracy can take root in democratic societies without so much as an effort to hide it is like being forced to watch your spouse cheat on your when that is not your kink” (p. 240).

Klein’s point is that this is fuel for conspiracy culture, and that channeling all the negative feelings this conjures up onto frenetic unconverings of evildoings by nefarious villains could be understood, from a place of compassion, “as some sort of twisted lunge for self-respect” (p. 240).

My point is that we are all too eager to villainise Musk or Zuckerberg (believe me, they make it so easy and so tempting!), to put the blame on individual people and individual AIs (again, there’s so much to hate about so many AIs). But it’s distraction theatre that stops us from engaging in a real critique of capitalism. And the more we do the dance of avoidance, the more our anxiety grows, and the more we distrust AI for reasons we can’t (or refuse to) really pinpoint.

A grim thought

I am led to the rather upsetting conclusion that our leaders’ lipservice to “trustworthy AI” is exactly the type of core issue abandonment that energises the conspiratorial Right Wing and grows its ranks.

People are upset about hugely important things like privacy, justice, climate change, and the legislative response could not have been better designed to alienate than if that was the point. Perhaps this is how we understand the timing of our global fascistic flip.

To conclude

My goal is not to diagnose our current politics. My goal is to make sense of what can feel at times like AI conspiracy. People appear to be distrusting AI for really weird reasons. Technologists get drawn in to arguments about the facts, trying to explain why people are wrong about the things they’re afraid of. But what these fact-checkers are failing to understand is that the feelings are right. AI is in many ways the wrong target, but distrust of AI makes sense—it is our lunge for self-respect in a world that’s rife with gaslighting.

“Look over there!”

Subscribe now

As ever, I’d love for this to spark discussion. Have I gotten it wrong? Have I extended the metaphor too far this time? Or does this help you make sense of something you’d struggled to articulate?

Leave a comment

If you enjoyed this newsletter, please click the Like button so that more people can discover it on Substack. And do feel free to share with anyone you think might be interested.

Share

Naomi Klein explains that she was compelled to write Doppelganger because she was being perpetually confused with Naomi Wolf… and it was driving her a little bit mad. In part, this was because “Other Naomi”, as she calls her, strays very far indeed from the “personal brand” of Original Naomi – people were attributing Wolf’s extreme (I’d say monstrous) Right-wing politics to Klein, a devoted Lefty who stumped for Bernie Sanders. The book, which documents Klein’s experience of disorienting “unselfing” from being twinned with this reverse-politics doppelganger, ultimately leads to profound and frequently beautiful insights into human nature and interdependence. (Here’s a nice summary for further background.)

In the last newsletter, I invited readers to consider how discomforting (and distrust provoking) it can be to encounter our AI doppelganger as a sort of caricature of oneself, particularly when that caricaturising reveals aspects of ourselves we would rather not confront. For this newsletter, I will focus not on AI as doppelganger, but rather how AI mistakes us for our “digital doppelgangers” and the distrust this produces.

Klein briefly touches on this phenomenon on her way through a deeper critique of a culture that demands everyone develop their own personal brand and the surveillance capitalism that preys on our desperation to meet this demand. She makes a distinction between the “aspirational avatars” that people consciously curate and the “real digital doppelgangers” created unintentionally as AIs assimilate the digital trails we leave behind. She explains, “Every data point scraped from our online life makes our double more vivid, more complex, more able to nudge our behavior in the real world… This machine made doppelganger… has a great deal in common with a human doppelganger: a person whom the world confuses with you but who is not actually you and yet can impact your life in profound ways.”

(Source: WIRED, on LinkedIn.)

Before I had this useful language of doppelgangers, I had explored the ramifications of AI misrecognising the people about whom it makes decisions. In a paper titled “Humble AI” (available here), collaborators Jason D’Cruz, John T. Richards, Kush R. Varshney and I argued that misrecognition by AI, and the cascading consequences thereof, produces pernicious spirals of increasingly entrenched (though largely justified) distrust of AI. We described how AI systems involved in consequential decision making – i.e. whether to grant or deny opportunity to people in contexts such as approvals for loans, housing, bail, custody, etc – do not deal with individuals’ trustworthiness (what would be a moral basis for granting opportunity). Though imperfect for sure, and susceptible to the harmful influence of prejudice, when humans assess trustworthiness, they do this by seeking to understand reasons, motivations, and circumstances that incline someone towards trustworthiness or not, as well as the presence or absence of morally excusing conditions for lapses in trustworthiness. What AI does, in contrast, is make a prediction about future behaviour based on comparatively context-free data about an individual’s past behaviour combined with an estimation of similarity to other people and their past behaviour.

There are of course criticisms of both the techniques and opacity of this clustering, as well as impassioned objections to treating people as one instance of a cluster, and for good reason. Klein touches on this, too – how as people move about in the world they are seen not as they are but “as a type”, held to account for others “like” them, their individual stories reduced to the collective story. Our focus in writing “Humble AI” was, first, to examine how people’s digital doppelgangers impact decisions made about them by AI – a sort of translation of the phenomenon Klein described to the AI world. But more originally, our main focus was on the moral perils of systems prone to mistaking people for their untrustworthy doppelganger and receiving adverse decisions on this basis.

In the paper we offer “Humble AI” as an antidote to characteristically “distrustful AI”. In coining the phrase “distrustful AI”, we sought to draw attention to the problematic tendency to design AIs to minimise the risks of false positives for those deploying them – e.g. avoiding granting loans to anyone who might default – and the subsequent biasing of such systems towards any potential signal of untrustworthiness and finding reasons to withdraw opportunity from people. We argued that just as people tend to be distrustful of those who distrust them, for fear of being unfairly treated, “distrustful AI” provokes reciprocated distrust by people for the very same reasons. We show how misrecognition, particularly when experienced repeatedly – as is the case for anyone mistaken for their “ethnic double”, as Klein shows, and which happens in the context of AI when systems take other AIs’ outputs as inputs – breeds resentment, demoralisation, and contempt. By exposing these dynamics, we hoped to better understand people’s totalising distrust of AI, as represented by polls showing that the majority of the public does not trust AI to make decisions about “any aspect of their lives”.

In the paper, we end by exhorting AI developers to design systems towards the more moral aim of avoiding distrust of the trustworthy. We offer affirmative technical measures for this more “humble AI”, such as continuously seeking out new evidence of trustworthiness that hadn’t yet been included as a feature in the model; doing more experimentation with decision thresholds to reduce false negatives; and investing in human oversight of cases where decisions are uncertain. But we also end on a note of hope about the potential of AI to enable us to overcome our human tendency to see people as their doppelganger, as sufficiently “like” someone else to be tarred with the same brush. “Human attitudes of trust and distrust can be altered indirectly,” we write – and typically very slowly, such as through cultural change – “but they are not under a person’s direct voluntary control.” The radical potential of these tools, thus far unrealised in the dominant paradigm of “distrustful AI”, lies in purposeful manipulation of AI systems’ “willingness to trust” individuals to align with moral aims (such as, say, racial justice). But most of all, to be “humble” means to accept that the work of betterment, of doing better at treating people as they deserve to be treated, is never complete.

If you have directly experienced being confused for your AI doppelganger, I’d love to hear about it and about how it shapes your attitudes to AI.

Leave a comment

If you enjoyed this newsletter, please click the Like button so that more people can discover it on Substack. And do feel free to share with anyone you think might be interested.

Share

So for this first installment, I want to present the idea of ‘AI as doppelganger’, and what it is about encountering our doppelganger that is so unsettling. And I will begin a pattern of reflecting Doppelganger against another book: in this instance, The Shape of Thought: Reasoning in the Age of AI, by my dear colleague, Richard Harper.

Harper’s book is a critique of the narrowing of the concept of intelligence to mean only the form(s) of intelligence that can be supercharged with computer processing power. He terms AI premised on this limited notion of intelligence “narrow AI” (NAI). In this paradigm, technology can (and will) exceed the human mind, and artificial intelligence is, thus, “us made better”. But of course, it is a very particular “us”: a funhouse mirror version, with an enormous head and no heart, if you will. And in this mirror world, everything we do is reduced to “some kind of analytical act”.

While Harper did not speak to the uncanniness of the doppelganger, his analysis could be read as a statement about how our own intelligence is made uncanny – that intelligence as embodied/amplified by NAI “is a feature of human nature, [but] not its sum”. We naturally recoil from representations that are like us but for certain essential ingredients of being human.

It’s tempting to focus on the superficial ways AI reveals itself when it attempts to mirror us, and the visceral reactions this produces. (I suspect I’m not alone in being able sniff out ChatGPT generated content and at wincing from the stench of it, not to mention loathing the person attempting to pass it off as their own work.) But what I’m interested in here is not, in fact, the uneasiness of recognising not-quiteness, but its opposite: recoiling from the implications of sameness to this doppelganger.

As Klein’s book shows, the doppelganger motif is not about what’s wrong with the other self - say, here, its exposing fauxpas, or the many other complaints we may (rightly) have about AI. The story of the doppelganger is always about a struggle with the original self. The doppelganger shows us the part of ourselves we don’t want to see; it reveals to us our inherent duality, as both good and bad. When we are unsettled by this AI doppelganger, is it in part because it is forcing us to reckon with this side of ourselves that can be – and has been, though we’d prefer not to think about it – led by “intelligence” to some very wrong decisions?

My thoughts are just beginning to unfold, but my aim has been to draw attention to something akin to disgust that is animating people’s distrust of AI. I often see computing folk baffled and frustrated by lay people being weird about AI (What are all these intense emotions about?), and this may begin to help us understand why. There is something unsettling, indeed, about the kind of beings AI is telling us we ought to be and have been all along; and it forces a seriously uncomfortable question about what (and who) we have always been willing to sacrifice through adherence to intelligence.

I’d love to hear whether this resonates with you. Is the kind of intelligence modelled by AI really us at our best? And if other forms of intelligence represent our best selves, what would AI look like if it emulated them?

Leave a comment

If you liked this newsletter, please click the heart button so that more people can discover it on Substack. And do feel free to share with anyone you think might be interested.

Share