Lobsters - 03 Oct 2026

Page 2 of 2

None of this is exotic. Apple's Keynote's presenter view has been shipping for twenty years plus with slides on the projector, notes on your screen. Apple never promoted the pattern to the OS even though it makes obvious sense. I'm asking for presenter view as a desktop primitive.

Browser tabs detach into real windows and join whatever task they belong to. The "browser" stops being a place windows live.

This all makes sense until you get to "how do you organize this stuff around tasks". I think for more expert users they're going to be able to do this stuff, but part of the problem is that we don't want them to ever have to drop back down into a configuration file. A mouse and dragging stuff around is too clunky for how this would work. Ideally I should be able to ask something that has access to what information is contained inside of each one of these browser tabs and window and help me organize it in some logical flow.

That's where the local LLM comes in.

I'm calling it a "quake overlay" because I'm old and the idea of a text input that drops in from the top with a universal shortcut has always been the quake terminal dropdown to me. Younger people know the same pattern from the Discord overlay, which I have decided not to be mad about. However in the previous diagram its shown as a more friendly "Ask" box.

But the basic flow would be that you ask the LLM to assist you with organizing, it would show you what it's thinking by querying the information through a constrained MCP server with a set verb list and then gives you a preview of what the layout is going to look like. The model never touches the window manager directly. It proposes and you press y.

People rarely pick up completely novel tasks: a graphic designer spends their life in "open files from network storage, make changes, save output, paste into chat, repeat." People rarely sit down at novel display contexts: laptop-only, the desk, the conference room. People rarely attach novel displays. Novelty is rare everywhere, so the expensive one-time machinery of classify, propose, preview, confirm only runs rarely, and everything in between is deterministic replay. You organize a limited series of tasks once and reuse them forever. Check the tickets, open Vim and a browser, work the ticket, write the update, open the PR, post it for review, next ticket. You could make the config a nightmare of JSON and stop caring, because the only intended reader is a model that doesn't mind. The config file stops being an interface.

The biggest issue here would be spatial memory. People understand where things are in relationship to each other on their computer and they don't like it if they are changed. Think of it like "if I came and messed up your physical desk by moving stuff around and adjusting your chair". So I suspect you would need to enforce a strict "append, don't replace" model.

There's a name for this actually. Kirsh and Maglio call it epistemic action, where Tetris players rotate pieces more than the game requires, because rotating is how they think about the piece. Link Your window layout is thought in progress. A helper that "optimizes" it mid-thought is interrupting you.

This is a problem I encountered with just trying this idea out though. Modern LLMs are actually not very good at "touch this stuff, never touch that stuff". So unclear if this is a realistic idea or not. One boring fix: the LLM proposes, dumb deterministic code executes, and the executor refuses to touch anything pinned. You don't need the model to have discipline if the door won't open.

You'd still need a high level UI element for users and this is where I think the more inclusive concept of filesystem could come in.

Tasks become roughly comparable to Directories now. But everything flows from the initial high level concept of "Tasks" and then flows out to individual things. One application is never the task. It's "some chat window, a browser, maybe Preview." Can we try to build it?

So my hope was that here I would be able to post a Linux desktop running an example. But as it turns out (unsurprisingly) it is insanely complicated to make something like this run at all. Some of the underlying ideas work reasonably well (global terminal shortcut dropdown from the top, scroll-able tiles), but it's still pretty clunky. I'm gonna keep working on it and see if I can get something as a demo running, so if you are interested either add me to an RSS reader or just....wait I guess.

In the interest of full disclosure I think this is gonna take many weekends of work to even get a functional demo running. So I'll do my best, but be patient.

Reasons this design sucks

So in attempting to get this up and running in a Linux VM, I immediately ran into serious logical issues with my design. I think failures are often more interesting to read about than successes, so let's talk about them.

- The privacy-minded design runs on surveillance-shaped data. There's an irony I sat with for a while. The design promises the OS will finally know which monitor is focal which is the thing Grudin measured twenty-five years ago and the mechanism is input-focus telemetry. Watch which display gets the keystrokes. Watch which windows never receive focus. Store that over time. The privacy-first window manager wants to know more about what you're doing than your current OS does. Keep it local, keep it ephemeral but it's still a lot of behavioral data to drive this system. Just the observation that every mechanism in this post is hungry for data that has historically derailed ideas like this.

- My favorite privacy feature was a joke. The original plan was elegant: private stuff far left, public to the right, and when you go idle the viewport drifts back to public. Like a friend changing the subject for you. Then I ran it, and the first thing I learned is that reading is an idle state. You stop typing to read the thing, and the system starts scrolling the thing away from you. Meanwhile, a person walking by sees everything, because a "private" window on a wide monitor isn't hidden it's just further left. The feature hides content from its owner and shows it to the threat. Privacy by occlusion needs to know when a threat exists, and the threat is a pedestrian, which is undetectable.

- Nothing in this design ever ages out. The canvas is infinite, the rule is append-only, spatial memory is sacred so the mess grows monotonically. I have solved the electronic messy desk by buying it an infinite desk. When I first read WindowScape, I called their evaporating photographs "the gap to fix." I've changed my mind because evaporation was quietly doing the job of a garbage collector. Something needs to go dormant without moving but deciding what goes dormant is a reaping decision, and reaping is exactly what append-only forbids. It didn't take long for the infinite desktop to feel overwhelming when I tried it.

Fun experiment

I don't know if this underlying idea is a good idea, but it is liberating to stop waiting for MacOS and Windows to do something better or interesting in this space and decide to try it yourself. The graveyard up there is full of good ideas that died of distribution. We're overdue for a radical experiment that ships as a default.

The most eye-opening part of this entire experience was how much the original overlapping window design was a hack to get around fundamental hardware limitations and how soon after it was launched did people clearly see the problems. I'm surprised how often that happens in technology, where you see a problem, search for academic papers about the problem and find just a massive wealth of information from people saying "oh yeah this is 100% a problem and one we should solve soon". Maybe this post inspires someone smarter than me to solve it finally.

Previous page | More Lobsters | Headlines

Original: https://matduggan.com/what-does-my-dream-os-ui-look-like/