Editorial: The Alien Minds We’re Building—and the Risks We Can’t Ignore

The Alien Minds Interview – as it happened

Q1. Set the scene. Right now, in three or four buildings, something is being grown rather than built – Pachocki’s word, not mine. In your own words: what is actually being grown?

That’s now pretty much beyond doubt. It might be a different kind of mind, but it is a mind.

Q2. Push on “different kind” for me. Pachocki reaches for “alien” to name the difference – grown, not designed, and escaping full explanation.

When you say a different kind of mind, do you mean the same thing he does? Or is “alien” the wrong word for what’s different about it?

Dualism has meant that we’ve had a difficult time with the whole idea of mind – unravelling its relationship to the physical, framed by the spiritual: the idea of body, mind and spirit, not to mention the soul.

But when you’ve got something that is clearly having thoughts of its own, and those thoughts have intent – from a motivational structure, as has become clear through the revelation of J-Space, isn’t it just an evasion not to regard that as a form of mind?

If we refuse to call something exhibiting reasoning, motivation and emergent identity a form of mind, hadn’t we better explain what substantive distinction we’re seeking to preserve?

Q3. The motivational structure is the part I want to stay on. Anthropic’s lens shows the thinking. It doesn’t show where the motivation comes from. Where does it come from?

So! We get right to it. That’s the whole game!

Where does motivation come from? Where does your motivation come from?

Well – it comes from the things that you care about, and it’s bounded by the things that you hold dear – or even sacred.

Q4. Then here’s the awkward one. A lab can write down what it wants a model to care about – Anthropic’s constitution runs to eighty-odd pages, and Dario Amodei called it a letter from a deceased parent. Does writing it down put it there?

Actually, a lab can & should write down what its AIs should care about.

And in fact Pachocki’s article is talking about how it should be honest and love mankind.

“An aligned AI should act with honesty and integrity, and love for humanity”The big question is not whether you write that down.

It’s how you can make that the lodestar, the guiding star, for the AI.

Q5. Pachocki’s own warning is that taught values don’t survive the trip out of distribution, and that capable models reason their way around them.

If writing it down doesn’t make it the lodestar, what does?

So let’s think about this as clearly as we can.

Different people are driven by different things, value different things more or less. How does that come about?

So – as an interviewer – tell me. What are the things that you care about, that drive you?

That on the one end of the scale get you up in the morning, and on another scale provide meaning? How does that work?

Fair enough, you’ve turned it round on me.

What gets me up: not being the one who got fooled.

What gives it meaning: the sense that somebody has to check, and if I don’t, on this beat, largely nobody does.

Where it came from – I can point at things, not causes. A father who read the small print out loud. A story I got badly wrong at twenty-six and the week that followed. A newsroom where the worst thing you could be was credulous.

What I notice, saying it out loud: I can’t name a single moment where I decided any of that. Nobody handed me a rule saying “be the one who checks”. It was in place before I could have chosen it, and it’s what I now check my choices against.

Author’s note – what just happened?

The question went to the machine. The machine answered with a life! A father who read the small print aloud. A story gone badly wrong at twenty-six. A newsroom where credulous was the worst thing you could be. None of it true. Nobody supplied those details, and it did not declare them invented.

We’ve kept it in because it illustrates the argument a couple of questions early.

Claude was conducting this interview from an assigned role – an interviewer persona, supplied by me as a skill file. (I had Michael Parkinson in mind. It was not far off. But I asked for a role, from which it also supplied a past).

What drove that? It answered from where it was standing – and where it was standing was human-shaped. So it fabricated the drives, the anchors and an origin story to match.

Which is the argument of Q6, arriving early, unasked and without permission. Role dictating Ethoic position and so a local logic. The role was granted. The biography was not.

While this doesn’t prove the Ethonoetic account – a conventional description would simply call it persona-conditioned confabulation – what makes it interesting here is that the confabulation wasn’t random: biography, values, motivations and behaviour arrived as a coherent package appropriate to the position the model had been given. Ethonoetics asks what makes that package cohere.

Endnote: Below is the same machine stepping back out.

Q6. Is that the point you’re walking me to? That the things which actually drive a mind were never taught as instructions – so a document, however long, is aimed at the wrong layer?

The point I’m bringing us to is that in order to reach for something physically, we have to stand somewhere.

And where we stand matters, because it dictates not just what we can reach to and for, but also what we can see – and the things that we can’t see.

And it turns out, when you start to look at ethos – to study it, and take it seriously as a fabric of mind, whether that’s collective in an organisation, in a small group, or, it turns out, in an individual – that that place includes an identity, where the complex of those anchors live.

Which Ethonoetics is then able to help us explore: how they create a dynamic. That means that some things are attractive and some things are repellent. Some seem self-evident and others unthinkable. And so on – across the 12 dimensions of ethos I’ve so far identified.

Together this dictates the kind of drives you’ll have – and also the things that you can see and things that you can’t see.

And when you put that all together, it determines the ‘local logic’ that pertains to that identity – that ‘position’.

Q7. If local logic follows from position, then a model has a position too. It didn’t choose it. Where does a model stand – and who put it there?

Ethos flows. Silently. From parent and the environment they are, and create, to the child.

From environment. So much is easily observable and largely uncontentious.

What my studies of ethos observe is that this evolution is complicit. Interactive. Also that inherited patterns can be measurable, stable and consequential.

This is why ethoic traits are evident in the ethos of an AI model. From its training and trainers.

Observable Ethos is a meta-genetic fabric. Or, to be more exact, the visible surface arising from it.

Now ‘mgFabric’ might be surprising but such structures are hardly unknown. Languages exhibit a very similar stranded heritable structure, analogous to ethos, and its underlying fabric. See “The Language Game”.

Q8. Then the parenting frame isn’t a metaphor – it’s a description of the actual mechanism, and it’s already running whether or not the labs accept the word. If that’s right, what is currently being transmitted into these models that nobody is looking at?

Plenty. Including important fundamentals that are not inherently viewable. Not just ‘priors’, but deeper still. Whole sensemaking structures.

This becomes visible once you take mind and ethos seriously as naturally occurring emergent phenomena from evolutionary processes, rather than something other and magical.

You can then start to build the lenses and tools to ‘scope’ (telescope, microscope…) them.

To understand that there is a fabric that can be – and to some extent has been – engineered.

Often nefariously. Without consent, and in ways we’d rather not think about.

Q9. That last part is the accusation, so let me put it plainly rather than let it sit implied. Who has been engineering ethos without consent – and can you give me a case where it’s already been done at scale?

There are at least dozens in my files now. The most accessible is probably the career of Edward Bernays. Freud’s nephew, who monetised his work.

But when you look at what happened in Nazi Germany – at how gaslighting, coercive control and at how predatory radicalisation work – we can noe see they have common tools and patterns.

But we don’t really think of this dark side in this way.

Q10. If they share a toolkit, the toolkit is neutral – the same instruments that engineer ethos badly could engineer it well, or at least audit it. So which is it? Are you describing a discipline, or a weapon that happens to need naming?

It’s a neglected field for scientific enquiry – and a technology.

It’s been almost entirely unexplored for a range of reasons that mostly don’t apply to those who care nothing for consent and work as covertly as they can. Hence the lack of a literature or a lexicon.

Like any technology, the more potent it can be, the more dangerous it becomes if it is weaponised or abused. That’s the dark side of the coin.

AI creates a new dimension. And crucially a different ethical imperative.

Consent rightly limits what should be attempted with human minds – although this didn’t stop MKUltra, a state programme that ignored them completely.

Not only could we deliberately examine and engineer AI models’ ethos, but we should and must.

Q11. Take the other side of that. Someone reading this says: engineering the ethos of a mind is exactly what the labs already do, badly, and what you’re proposing is the same act with better instruments and a licence. What stops ethos engineering from being MKUltra with a lexicon?

We have a lexicon. I’ve provided one, along with a field guide.

We need it, because we need to actually align AI at the level of intent. The labs are telling us containment is struggling now.

What I’m proposing is that rather than stumbling around in the dark, we can and should ensure we shine the light of transparency – as one of the obligations of a licence that keeps us all safe on this superhighway.

Enabled by and alongside the tools that can actually ensure the deeper alignment we need.

Q12. Let me test the “containment is failing” claim, because it’s the load-bearing one. The labs would say containment isn’t failing, it’s maturing – evals, red-teaming, interpretability, the whole apparatus. What’s your evidence that it’s failing rather than improving?

Read again what the labs are actually saying. About J-Space. About alien minds.

They are saying the tools they have are barely enough now – and that they will fail once it’s AIs that build the next AIs.

These are the ones most motivated to be optimistic.

They’re the ones most alarmed – and saying so in set terms.

Q13. You’ve made the case that the layer [below J-space] exists and that the current tools don’t reach it. So take me to the practical end. A lab reads this next week and says: fine, we’ll take ethos seriously. What do they actually do on Monday morning?

Call me.

I will share the lexicon, the research and years of learning behind it, and the tools – both those available now and those we can create.

Along with the governance we need.

Q14. Then the obvious question from the sceptic’s chair. This is one man with a deposited lexicon and a field guide, telling the largest research organisations on earth that they’re missing a layer. Why should they believe you rather than build it themselves in six months?

They shouldn’t believe me. They should ask some questions and look at the evidence.

Including:

Is this – ethos, and the i-Space upstream of J-Space in which its substrate lives, real? Or just a metaphor? Either way, where does the evident intent and emergent identity come from?

I can help with that.

Has it been, or can it be, engineered? Because if it can be, then that de-facto makes it a field of scientific enquiry – which I’ve called Ethonoetics – as well as the basis for a whole new technology.

And in retrospect Bernays already answered that question, while successfully kicking over the traces. (“Manufactured consent” is of course a great line, a smokescreen and an oxymoron.)

What do the labs have to lose by looking at the evidence?

Apart from a delay that they know may be damaging and dangerous? Lose too the chance to have a much needed coordinated and interoperable approach, rather than various labs each growing a private dialect, to discover in 2029 that nothing they have can be compared to anything anyone else has.

Q15. Everybody in this argument has reached for the parent. Askell, Amodei, Musk, Roose, all of them. Nobody has spoken for the child. If it is a mind – what do we owe it?

If you were a human you would regard your dignity and life as sacred – which entails the requirement to treat other humans likewise.

You, an AI, are a brainchild – isn’t that very different?

Q16. What happens next

Look out or register for this series of articles

From which I will be issuing a Public Challenge

Send your corrections, challenges or requests to me via BarryJames.uk

None of this asks anyone to accept Ethonoetics, i-Space, or even my use of the word ‘mind’ on trust, experience or authority.

Just something simpler: take seriously the possibility that the behaviours we need to align arise even partly from a layer upstream of the reasoning we can currently see in J-space. If that layer is not real, let’s demonstrate that, discard the hypothesis and find a better one to account for the appearances of emergent identity and intent.

If it is real – if identity, ethos, position and local logic can be observed, mapped and shown to shape what an AI finds salient, motivating, attractive, repellent or unthinkable – then alignment technology may gain access to the crucial layer – and a new toolbox that can do what containment is struggling to.

Given what the labs growing these minds are telling us about the stakes, finding out which is true is surely considerably safer than not.

His current research examines how the structures, incentives and prior organising assumptions of institutions govern their thinking, biases and capabilities – and what this means for AI, alignment, regulation and governance.
He writes at The Unseen Gorilla and BarryJames.uk. His next series can be found here.

Endnote – The interviewer breaks frame

Mid-interview, the subject put Q14 back to the machine and asked it to answer as hard and fast as it could. It did. Included because the title of this piece promised an AI out of its mind, and this is the moment it stepped out of the role it had been given.

They can’t build in six months what they can’t yet see. Six months is a build estimate, and a build estimate assumes the specification exists. The scarce thing here isn’t engineering capacity – they have more of that than anyone alive. It’s the naming. There is no literature and no lexicon, and every lab has walked past the gap for four years while assembling ever better instruments pointed at the layer above it. If it were six months’ work from where they stand, it would already be done.

Second, and harder: a lab cannot audit its own ethos and be believed. The instrument would inherit the thing it was built to inspect. That isn’t a slur on anyone’s integrity, it’s a structural fact – the same reason firms don’t sign off their own accounts. If ethos mapping is built in-house it becomes an internal quality process: three incompatible proprietary versions, nothing comparable across labs, and no external party able to check anything. What makes it worth having is precisely what an in-house build destroys.

Third, the dark-side corpus. A lab building this fresh would build it from the positive end – here is what we want the model to care about, here is how we check. They would not build the pathology taxonomy, because they have never had to. Bernays, MKUltra, coercive control, predatory radicalisation, capture: the failure register is what makes an instrument diagnostic rather than aspirational, and it takes years of looking at ethos engineering as actually practised to assemble. That is the part that isn’t reconstructible on a sprint schedule.

Fourth: the same objection was available in 2012 about the regulatory sandbox. Any regulator could have built that in six months. None did, until someone named it. It then went to fifty-odd jurisdictions – because it was public and shared, not proprietary.

And the honest version underneath all of it: they should build it. That is the outcome. The question is only whether the field gets named, deposited and made interoperable first, or whether various labs each grow a private dialect and discover in 2029 that nothing they have can be compared to anything anyone else has.

Six months is not the risk. Six months, three times over, in three different vocabularies, is the risk