Let me tell you where this started.
I read the 1985 Direct Manipulation by Hutchins, Holland and Norman for an HCI book club in London, hosted by Ini & Sofia at Gradient Labs HQ.
On paper, that sentence sounds painfully academic. It also reminded me of first-year computer science, where we learned the history of computers just because and now it's a past memory.
However, the conversation in the room was electric. We were talking about direct manipulation and the gap between what a person wants to do and what a computer lets them do. Somewhere during that conversation, several new neurons started firing in my head. I left with a question I could not put down.
What is the final interface?
Not the next phone. Not the next headset. The final interface. The thing we might eventually look back at and say: of course computers ended up like this.
A very short history of getting computers to do things
Asking what the first computer was starts an argument very quickly. It depends on whether you mean mechanical, electronic, programmable or general purpose. The more interesting story is how we learned to talk to computers.
We moved from switches and punched cards, to command lines, to a mouse and windows, to touchscreens, and now to voice and AI. Every major interface made the distance between thinking of a thing and making the computer do it feel smaller.
The paper gave us two useful ways to judge an interface. First, how quickly can I tell the system what I want? Second, how quickly can I understand what it did?
The proper names are the gulf of execution and the gulf of evaluation.1 I prefer the normal-person version: Can I make it happen, and can I tell what happened?
Then the room made the idea messier, in a good way.
Sophia showed how language and tools shape what we are able to think. We keep treating text as the natural way to use AI, even though people also understand through sound, touch, movement and space. Mo's point also stayed with me: most of us in the room were designers, so of course many of our imagined futures were visual. A lawyer, a coder or someone who cannot speak may find a completely different interface more direct.
Someone used a light switch as an example. Pressing it feels direct. But if a person grows up asking a home assistant to turn on the light, voice may feel just as direct to them. Then put the same voice interface on a train and ask someone to say their banking details out loud. Suddenly it is a terrible interface.
That was probably the biggest thing I took from the room.
If you want the fun version of how quickly our tools changed, Mrwhosetheboss recently put 42 years of phones in one room. Watching every generation side by side makes one thing obvious: the object keeps changing because we keep searching for a more natural way in.
So what happens when the easiest way in is simply to ask?
Let's take a peek through movies
When I think about the future of computers, I do not first think about research papers. I think about the things I watched growing up.
Matrix, IronMan, Harry Potter etc
JARVIS was never trapped inside one screen.
In Iron Man, JARVIS could be present in Tony Stark's home, his workshop and his suit. The suit was powerful, but the suit was not the whole system. It was one way for Tony to see, speak to and use the same intelligence.
Harry Potter had a wand, but the wand was not the magic.
Harry still needed to learn. He needed the right words, the right movement and the right judgement. The wand gave the magic a form he could control. It made something invisible usable. (and the wands have versions holy sh*t!)
The Power Rangers Morphers / Ben 10 Omnitrix.
A small object could unlock something much bigger than itself. The object was not impressive because of its screen size or its menu. It was impressive because of what it connected the person to.
These are stories, of course. But stories let us test an idea before the technology is ready. They make the strange thing feel obvious.
My first thought...
Maybe we should separate the system from the tool.
I looked at how encroached i am in the Apple ecosystem. The MacBook, the iPhone, the Apple Watch & the iPad
Each of these are different devices with a somewhat unified system that makes the whole experience feel seamless.
In the version of the final interface I envison, they seem to be more like doors into the same place.
This Apple Watch experiment made this feel real.
Years ago, Shervin Shares tried travelling from Denver to Chicago for 24 hours with only an Apple Watch. No iPhone. No wallet. The watch handled enough of his normal digital life for the experiment to be possible.7
What I like about the experiment is not that it proves the watch should replace the phone. It proves almost the opposite. Many of the same services and the same operating system can follow you onto a new object, but the object still changes the experience. A tiny screen changes what is easy to read. A wrist changes how long you want to interact. A battery changes how much you can trust it. The medium is not a neutral container.
So maybe an operating system is already becoming an interface that can wear several bodies. The intelligence, accounts and services stay connected. Each piece of hardware decides what part of that world you can comfortably reach.
The phone
Private, portable and always close. Good for quick instructions and capture.
The laptop
Space to think, compare, make and inspect the details.
The watch
A quiet signal on the body. Useful when a screen would be too much.
The glasses
A layer on the world. The system can respond to what I am already seeing.
The home
A shared space where the system has to understand more than one person.
The next object
Something we have not invented yet should still be able to join the same system.
so the hypothesis here is this.
The best tool depends on what I am doing.
This was a theme from the conversation. Someone mentioned how it was so much slower & expensive to use AI to edit a UI in code that could take him a couple seconds.
Another mentioned having stopped using the CLI in claude code & codex and now uses just the canvas. Someone else preferred text because code already lived naturally there and it goes on and on.
I realised everyone had a way to make the most of the same interface to work best for them.
which leads me to my next point.
The final interface has to be fluid.
Which is why i have a hunch it's going to be more about the operating system not the hardware. The intelligenct system should not force every situation into the same shape. It should contextually morph to fit the medium it's needed in.
Real examples of this "future".
01 - AI glasses are becoming normal objects.
Meta Ray-Ban Display puts a small display inside glasses that still look like glasses. The paired Neural Band turns tiny muscle movements into controls. This matters because the interface starts to leave the screen and move onto the body.2
Google is taking another route with Android XR. Its new glasses and lightweight XREAL AURA hardware put Gemini into what you see, while moving some heavy computing into another device. The glasses can become lighter because the whole system does not need to live inside the frame.3
02 - Watches already understand quiet computing.
A watch can tap my wrist instead of demanding my eyes. That feels small, but it may be closer to the future than a giant virtual screen. Good technology should sometimes whisper.
Many iOS apps also have a mini-version for the apple watch.
03 - Multi-device Continuity.
ChatGPT can remember useful context across conversations when memory is enabled. Apple has spent years letting tasks move between its devices. These are incomplete pieces, but both point toward a system that does not make me start from zero every time I change screens.45
I do not think one company has assembled the full thing. Maybe no company should own the full thing. But the parts are arriving.
A powerful tool should still require something from us.
The wand comparison from Harry Potter matters more than ever because Harry could not wave it around and expect wisdom. Such power needed skill.
I do not want an AI system that makes people passive. I want one that helps us do more while still requiring a level of effort from us. A child should not grow up believing that asking is the same as understanding. An adult should not hand over every decision because the system sounds confident.
Perhaps learning to use this final interface, aka "the operating system" will become its own kind of literacy.
- Knowing how to explain what you want.
- Knowing when the answer is weak or wrong.
- Knowing what the system remembers about you.
- Knowing when to do the work yourself.
- Knowing when to switch the whole thing off.
More questions I'm musing on
Should this system live on the cloud or somewhere I own? (and how much difference does it make?)
How should it behave in a shared space, like a home, school library, the workplace? (whose memory does it use?, who has what access?)
When context can travel seamlessly there's ease in that. But how much is enough memory? An unforgiving system can create a world of no second chances. It can also create a world were data is so personalised that'd make Meta ads today look like they're from the stone age.
How do we prevent "Big Tech" from making us all pawns for capital gains through this interface? (i dread a future like Bladerunner 2077)
But I'll leave it at that. Let's enjoy these days a little more before some smart kid figures out the next interface we'll be using in a few years.
Maybe the final interface is not a screen, a headset or a pair of glasses. Maybe it is a relationship with one system that knows how to meet us through all of them.
Notes and sources
- Hutchins, Hollan and Norman, Direct Manipulation Interfaces. The 1985 paper describes the gulfs of execution and evaluation. Ben Shneiderman introduced the direct manipulation idea in 1982, with his main paper published in 1983.
- Meta Ray-Ban Display and Neural Band. Meta's description of the in-lens display, EMG wrist control and intended short interactions.
- XREAL AURA and Android XR. Google's description of lightweight glasses, a separate compute puck and Gemini working across visible apps.
- Memory and new controls for ChatGPT. An example of useful context continuing across conversations, with user controls.
- Apple Continuity. An example of tasks and content moving between devices in one ecosystem.
- Mark Weiser, The Computer for the 21st Century. A much earlier vision of computers fitting into everyday life instead of demanding all our attention.
- Shervin Shares, I Survived 24 Hours with ONLY an Apple Watch. A trip from Denver to Chicago without an iPhone or wallet, used here as a practical example of how the same software world changes when the medium changes.








