Open Questions Around LLM Interfaces
Very early note on open questions for how to interface-with-LLMs.
High-level it feels like the industry is basically shipping the API.
If this thing is eventually going to be used by a human, start off with the user interface, because it is really hard to put a good user interface on sheer functionality after the fact.
— Alan Kay in an interview at SPLASH 2024 ↗
- online forms often have this feeling of Computer Science Positivism, but on the other hand, they clearly guide the user through what's possible
I see a common thread about how chat isn't the best interface for LLMs. But it's certainly more flexible than our current UI.
Our current UI paradigm based on forms. The kind you fill out.
— https://x.com/iamwil/status/1816986882808643774 ↗- to which Rasmus Andersson responded:
I love this kind of UI. It shows me what's possible
— https://x.com/rsms/status/1817303041579647467 ↗- which, I think, is a better verbalization of what I sometimes thought of as "I don't know where the edges of the system are"
- this ties into Computer Science Positivism - we used "forms" for a very long time, but in paper ones you can add a comment, cross something out, etc. - it's software that over-constrained them
- but, now, with LLMs, maybe we could have forms again that could be molded to what you're trying to do - maybe you start with the system showing you "what's possible" in some sense, but then can go on modifying that behavior, which seems relevant to Malleable Software
- to which Rasmus Andersson responded:
- Dynamic Documents seem like great possible direction for interacting with LLMs
- Anthropic Artifacts are a step in that direction, but are missing - I think - an important part of the model "seeing" the resulting Artifact
- for example, if I ask for a tool to do some investing simulation with a couple of inputs, I can't then play with these values, and ask the model a question about the specific result that I'm seeing!
- I think some imagined ideal state is that the "conversation" with the LLM is a "document" at the same time, can be branched off, picked apart, re-combined (Malleable Software), and contains both "text" and "computation"
- which all seems related to Chatting With Glue ↗, Converge ↗, and Wikum ↗
- Anthropic Artifacts are a step in that direction, but are missing - I think - an important part of the model "seeing" the resulting Artifact
- another idea that seems interesting is using LLMs for dynamic Semantic Zoom, especially in case of programming
- I'm imagining being able to browse the code by zooming in-and-out (and around?) through different "lenses"
- one of them is what is rendered - on the highest level there's the visual UI, I can drill into what components make it up, and then the actual implementation for the specific component
- another one is user stories - how does "logging in" work like - and from there the specific functions and endpoints that make up that behavior
- yet another one could be data - starting from what the table I see on this specific screen, where do all the values come from, and for a specific value that seems like a bug, being able to drill into the code-paths that make it up
- a lot of these ideas relate to Programming Systems - and I think the approach should be a mix of strong heuristics and LLM "fuzziness" sprinkled on top
- I'm imagining being able to browse the code by zooming in-and-out (and around?) through different "lenses"
- VPLs are a "favorite problem" of mine, so there's a separate VPLs and LLMs
- for LLM-assisted programming specifically:
- Solving Things Visually - LLM could generate ad-hoc debug visualizations, I think there could be interesting ways in a "Moldable Development"-like interface where the debug visualization are created ad-hoc
- figuring out code review interfaces is even more important - LLMs can throw a ton of code at the developer, and the effort shifts from creating, to curating:
This study of Copilot shows that developers spend more time reviewing code (as suggested from Copilot or similar tools) than actually writing code. As AI-powered tools are integrated into more software development tasks, developer roles will shift so that more time is spent assessing suggestions related to the task than doing the task itself (e.g., instead of fixing a bug directly, a developer will assess recommendations of bug fixes). In the context of Copilot, there is a shift from writing code to understanding code. An underlying assumption is that this way of working-looking over recommendations from tools-is more efficient than doing the task directly without help. These initial user studies indicate that this is true, but this assumption may not always hold for varying contexts and tasks. Finding ways to help developers understand and assess code-and the context within which that code executes and interacts-will be important.
— Taking Flight with Copilot, Christian Bird et al. - I'm also imagining interfaces for refactoring where I can drag things around ("this should be in this file"), leave little sticky notes ("make this an enum"), and then click "go" to make the LLM apply the changes all at once
The potential of LLMs goes far beyond a natural language interface.
For example, an application could feed the relevant context to the model behind the scenes and use that to preemptively suggest what I should do next. The toolbar could adapt to my specific task. Dialog boxes wouldn't have to be so static. I could point to a region of the screen and ask for an explanation. It could identify a misunderstanding before I know about it (see my prior work on inquisitive interfaces). The system could show me examples based on what I'm doing. Tutorials could take on a personality that better suits me.
— Natural language is the lazy user interface ↗ - Austin Henley
If I had a personal programmer that sat next to me, how would I work with them? Would I give them a sticky note with the task and wait for them to come back with the code for me to verify?
Or would we ask each other clarifying questions, point out significant details, discuss design tradeoffs, and show incremental progress as we work?
I think that an iterative human-in-the-loop flow is not just the ideal UX for developer tools, but also necessary because of how inconsistent LLMs are.
— How I would redesign Copilot Workspace ↗ - Austin Henley
Try writing a tax code in chat messages. You can't. Even simple tax codes are too complex to keep in your head. That's why we use documents - they let us organize complexity, reference specific points, and track changes systematically. Chat reduces you to memory and hope.
This is the core problem. You can't build real software without being precise about what you want. Every successful programming tool in history reflects this truth. AI briefly fooled us into thinking we could just chat our way to working software.
We can't. You don't program by chatting. You program by writing documents.
— Chat is a bad UI pattern for development tools ↗ - Daniel De Laney
I think the case for visual programming is much stronger now. Visual programming works well at higher layers of abstraction but sucks at expressiveness. LLMs are better at code than higher order thinking. Visual tools that allow us to view and edit our software at varying depths makes a lot of sense right now.
— https://x.com/masadfrost/status/1837524351589273643 ↗
The real breakthrough won't come from smarter chat boxes. It will come when prompting feels like play — when you can sketch, drag, remix, and layer ideas the way you do in Figma or GarageBand. That's when AI stops being a command line and starts being a true creative medium. Just as WYSIWYG unlocked the web, new interfaces will unlock AI.
— Prompting isn't the future. Creating is. ↗ - Elizabeth Laraki