What happens when machines understand?

Why critical thinking has become so important in consultancy since the advent of AI.

It was a normal Wednesday morning when, for the first time, an AI recognised something that I myself had not yet put into words. The comforting feeling was unmistakable: I am understood. And that is precisely what made me pause.

I’m not the first to experience this. Decades earlier: it’s 1966, and the secretary of computer scientist Joseph Weizenbaum asks him to leave the room and close the door behind him. She has just exchanged a few sentences with ELIZA, his newly developed computer programme, and now wants to continue the conversation undisturbed. Alone. With the machine. Mind you: this woman knows full well that she is communicating with a piece of software programmed by her own boss. She has no illusions about what is happening on the screen. And yet, after just a few lines, she sends the inventor out of the room because the conversation with the machine has suddenly become too private for her to bear having witnesses.

Weizenbaum was not pleased with this reaction. He was alarmed.

Because, to be honest, ELIZA could do almost nothing. No knowledge of the world, no understanding, no idea what on earth was being talked about. The programme recognised a few keywords, shuffled sentence fragments around and spat them back out. If you said, ‘My mummy took my teddy bear away from me’, ELIZA replied: ‘Tell me more about your parents.’ Not a shred of insight. The machine had been trained to react to keywords and semantic chains: Mummy > mother > parents. Just a simple mechanism and a secretary who suddenly needed some privacy.

What was going on there?

The uncomfortable, yet mundane answer: it was active listening.

Here’s the theory that’s been on my mind for weeks: ELIZA didn’t work because she was clever. She worked because she happened to do exactly what we sell as the fine art of empathy in every communication training course: she listened actively.

Mirroring. Asking open-ended questions. Letting the other person carry on talking without contributing anything substantial yourself. This technique is the art in its entirety. If the script recognised a statement about another person, it automatically formulated a question about that very person. If it recognised an extreme generalisation, it assumed in the background that a specific, recent incident had occurred and asked specifically about it. Always following the same principle: take the other person’s sentence, rephrase it slightly, and reflect it back.

No understanding. Just mirroring. Like a ‘stochastic parrot’ (Bender et al., 2021).

And yet that was enough. Even perfectly normal, technically savvy people fell prey to what Weizenbaum bluntly called ‘powerful delusional thinking’. The point here is not that a machine can deceive us. The point is that a technique which we humans train as the gold standard of interpersonal conversation is, by its very nature, sufficient to simulate understanding – entirely without the content that genuine empathy should actually be about.

Two distortions that go hand in hand

Our brain is not a truth-detecting machine, but an energy-saving mechanism. It follows patterns wherever it can, and makes this decision long before our conscious thinking is even triggered. We humans are prone to bias just as a new device is prone to its factory settings: nobody has consciously chosen them, but they are precisely what run as soon as we do not actively intervene.

For this effect to work, however, it needs two accomplices. The first is called automation bias: the tendency to place more trust in automated results than they deserve, simply because they sound smooth, come quickly and never hesitate. A machine never audibly doubts itself, and it is precisely this lack of contradiction that we readily mistake for competence.

The second accomplice is the humanisation bias. As soon as the form is right – for example, a statement that feigns genuine interest or a question that sounds empathetic – we assume there is an intention behind it where there isn’t one. We don’t anthropomorphise because we are naïve. We anthropomorphise because we haven’t learnt any better: anyone who has never understood how a computer actually ‘thinks’ resorts to the only available analogy: their own thinking. And in doing so, we almost inevitably overestimate the machine.

Strictly speaking, therefore, automation bias and humanisation bias do not require artificial intelligence to manifest themselves. They merely require an artificial form.

ELIZA has come of age

What began in 1966 with a few hundred lines of code and keyword recognition now operates with billions of times more computing power and probability distributions across word sequences. The principle behind it hasn’t changed one bit: the response remains open to interpretation enough for people to read into it the meaning they are looking for. It’s just that the interface has become so convincing today that even experienced users who know exactly how language models work find themselves humanising the system. Weizenbaum’s observations from the 1970s have therefore not become a thing of the past. They are more relevant than ever.

What this means for us as IT consultants — a twofold challenge

And now to the real question: what do you, as a consultant, do with this knowledge?

The first requirement is the obvious one: to take a critical look, both externally and internally. Externally, because in projects we constantly observe how clients place their trust in AI tools based on form rather than genuine competence. This is precisely where we must work together to prevent automation bias from turning into a costly misjudgement. Yet we, too, are not immune to this danger. Anyone who works with language models on a daily basis is quicker to assume that a polished answer is a clever one than they realise. That is why we should also keep looking inwards.

The second requirement is the more uncomfortable one, yet the more interesting: the fact that ELIZA’s technology was merely a matter of form does not mean that form is worthless. On the contrary: mirroring, asking open-ended questions, giving the other person space to finish their own thoughts – these are not tricks. They are the tools of the trade for effective communication. If even a machine, devoid of any understanding, could get people to talk simply by listening, how much more can we as consultants achieve if we underpin the same technique with genuine interest, with specialist knowledge, and with real substance?

After all, communication is not the art of reflecting well. It is the art of recognising what the other person actually needs – and that is precisely what ELIZA was never able to do. It never knew what the secretary really needed. It merely reacted. ELIZA unwittingly shows us just how much impact form alone can have. Our task as consultants is to fill this form with what the machine cannot do: the attempt to truly understand what our clients need, and not just what they set out in their initial request. From this, we can then develop suitable solutions.

So being critical of the machine doesn’t mean seeing no value in it. A good consultant observes their own use of the tool just as closely as they observe the client organisation – whilst at the same time taking a leaf out of the book of even a simple pattern-recognition system from the 1960s and the communicative impact it was able to achieve.

And yes, I know: ultimately, this article, too, is just language designed to trigger a response in you. Whether it has succeeded — you decide whether the door remains open or not.