Augmenting UX with LLMs

• Even if your model can answer any question it's less useful if the user still has to think of every single question.
• Lots of typing involved and this can feel unncessary.
• Can even make it harder to run the LLM if you need to get out structured intents.

Generative Interfaces Beyond Chat

• Youtube Link
• Talk by Linus Lee on April 13, 2023 at LLMs in Production Conference
• In an application we have all sorts of context beyond what's provided directly in the chat. Having to type the entire context into the chat box like you do with ChatGPT is inconvenient when the app can provide a lot of that context.
• e.g. a spreadsheet has the spreadsheet data itself but also the selected cell.
  • Using things like the current selection as context into the chat helps
  • "Sum column A into this cell"
• AI's observations
  • Omniscience - AI knows the full state of the app
  • Call by name - Ability to explicitly mention objects by a name in the chat
  • Literal mention - dragging objects (files, contacts, paragraphs, etc.) into the prompt box, which are then turned into the appropriate identifier when sending them to the LLM
  • Model-Generated Context Menu - Similar to the right click context menu. But given the selected thing, have the model figure out the most likely things and present more advanced options than a normal context menu.
  • Dot-driven programming - similar to code/CLI autocomplete where you have some kind of trigger key to popup options
• Can use the model both to do actions, and to suggest possible actions.
• Most of these basically boil down to a "verb the noun" phrase
• Closing the feedback loop
  • Respond with a range of outputs (not always convenient, but useful when you can)
  • "Shopping" over "creation" — Selecting from a list of objects is easier than requiring the user to invent an option
  • Interactive Components — Model responds not just with text but with widgets that are prefilled with the answer but have other traditional UI controls to tweak the options
    • e.g. "What's the weather in NY this weekend" returns a weather widget prefilled with the location and date, but user can use date controls or other stuff to change the location without having to interface solely through the LLM.
• Avoid the "blank page state." Better to suggest options for the user to get them started.
• Full chat as an escape hatch rather than the default.

Demos

• LLM Librarian demo by Eugene Yan
• Potluck + AI Demo by Geoffrey Litt