AI-UX: Future Use Cases For Designers

Artist: Jeff Axup, “The Collaboration”, 2023, Medium: Stable Diffusion

Summary: Recent AI advances have dramatically changed human-computer interaction. 12 use cases are explored and design recommendations are provided, in order to help guide future AI system design and development.

Use cases are tasks undertaken by humans with an associated goal. Good design primarily focuses on getting the user to that goal rapidly and painlessly. However, it is also possible to view use cases from the perspective of the system (i.e. AI, computer), and specify what its goals and actions should be for the human user. Before we personify the AI too much, it’s worth noting that the AI’s goals are ultimately chosen, designed and programmed by humans.

That said, computers are about to be able to take on a much wider variety of use cases. They will also do it in multiple modalities, and with a greater degree of knowledge and autonomy. So we need to start thinking bigger and more expansively when it comes to use cases.

The following 12 use cases are not intended to be comprehensive — rather they are intended to provoke discussion about design considerations for common AI-based tasks and scenarios. In many cases products do not exist to fill these niches yet — thus, they may foretell the creation of new companies and new specialization areas within UX. Each use case below has a final section labeled DESIGN RECOMMENDATIONS.

Note: AI is generally used in this article to refer to LLMs (large language models), diffusion models, and other models that automate cognition and more advanced tasks.

Use Cases

#1 As a Human, ask for this task to be magically done

• It has always been the job of the UX designer (or product manager, or researcher) to pose challenges to engineering teams. A car that could park itself (or drive itself) is a good example of identification of a user goal: “park the car for me”, and then have it occur “magically”. The user doesn’t care what is going on behind the scene.
• AIs are now capable of doing many things that were previously thought to be “magic” or “sci-fi”. Before, the user might have had to go through a long series of steps to do a task. After the introduction of an LLM they might just specify some guidance, or a job name, click go, and wait a few seconds.
• DESIGN RECOMMENDATIONS:
  Existing products will need to review their current use cases and features to see how much can be automated, and whether entire sections of task flows can be skipped. Another way to look at this is that some tasks will no longer be “user-facing”.

#2 As a Human, ask for help to analyze a situation or find a solution

• There are a huge percentage of tasks that are based around asking questions or trying to troubleshoot things. Nearly every support desk, call-center, repair shop, and many consulting services are focused on these types of problems.
• LLMs are extremely good at looking through the large amounts of data it was trained on, and tirelessly finding potential solutions and giving detailed steps to see if they work.
• Most of these services are likely to have a chat interface, but many may not. Example: submit a video of my toilet that is running and give me graphical steps on how to fix it. Or, here is a picture of my back yard and a link to the public records for the property. Give me my options for a backyard pool along with pictures of what new designs would look like, cost comparisons, and local companies who are best fitted for the job.
• DESIGN RECOMMENDATIONS:
  Users are going to keep upping the level of complexity of the tasks they want done for them. Whenever you think you’ve accomplished something amazing, users will find it mundane a few months later and will be asking for the step after that.

#3 As a Human, iterate with me on this task over time

• One of the benefits of the chat interface is that it encourages ongoing interactions over time. First you ask why the sky is blue, then you ask what light diffusion means, then you ask how to combine paint colors to get the same color of blue for your next art project. The point is that you don’t necessarily know where you want to end up when you start. You’re changing and extending your goals for the conversation or project as you get new data and make progress on the problem. This is a valuable pattern to support for users.
• Similarly, if you are in a graphic editing tool and select a section of an image to add a flying saucer into, you probably won’t be happy with the initial result. You should be able to have a conversation with the AI about the shape and color of the flying saucer, or change your mind and change it to a flock of geese. Creativity is about collaborating with a partner, and seeing what you can co-create. Similarly, you can collaborate with yourself (over time) and you might have different ideas each morning over the course of a week. In either case there needs to be a history of the dialogue and your thought process.
• Some LLMs initially did not show (or remember) conversational history. ChatGPT’s original design had a “chat history” on the left-navigation bar, which is a step in the right direction. However, it turns out that LLMs have a limited number of tokens they can process at one time (just think of it as working memory size.) They devolve into nonsense if this token limits is exceeded in a particular chat session. Consequently just like resetting your phone occasionally, it is also wise to start a new chat session for each new problem area. This is a tech limitation that is directly the opposite of what the user wants (which is to continue discussions forever and have it remember everything.)
• DESIGN RECOMMENDATIONS:
  Probably all AI interfaces should be built with an architectural design requirement of “supporting ongoing interaction”, which implies history storage and being able to continue accurately interacting on tasks over time.

#4 As an AI, learn

• AIs have several opportunities for learning that they apparently are not currently taking advantage of. The first is learning from you, the primary user. The second is learning from everyone (as in all of humanity using it or sharing data) and triangulating.
• If we were to compare AIs and humans, as many people are prone to do, we would say that AIs currently have anterograde amnesia. They are able to remember things (typically from before September 2021), but not much later than that. Also, if you ask it to remember something at the beginning of a project, it will forget it when you start a new chat session, and possibly even during the same session (if you overload it.)
• Regarding the last point, LLMs have a “token limit.” As explained by GPT-4: “The token limit includes both the prompt (input from the user) and the response from the model. So if you provide a prompt that is 1000 tokens long, the model will only be able to generate up to 1048 tokens in response.” This engineering constraint is very unintuitive to a user and invites error scenarios. There are also many scenarios where you might want to input more that a few thousand tokens (e.g. the code for an application). It also gets worse over time, because if you start a conversation with some constraints and goals for the project, it should be keeping those in mind as you ask more questions over time. The model has to retain more and more history with each new query. Currently it may (and frequently does) silently max out and forget those things, causing the entire project to disintegrate and go off track without warning.
• Having to repeatedly teach the AI the basics about you and your work is a serious productivity drain, and demotes the level of AI intelligence to perhaps a 3-yr old child, a smart dog, or a severely disabled adult. Designers should prioritize fixing this. You don’t want a 3-yr old as your personal assistant — you’ll spend more time cleaning up after its mistakes.
• DESIGN RECOMMENDATIONS:
  GPT-4 is already known to have multiple interacting sub-models. Interacting with it hints at this underlying construction. I would guess that one of the sub-models handles building a logic model and tracking goals. The main model is probably akin to “working memory” in humans, where it holds short-term info and rough idea of what is currently being worked on. It also needs a “long-term memory” function where it can store things such as overall project goals, who the user is, what their larger work context is, and other variables. LLM architects, engineers and designers have a lot of work to do to make the LLM a trustworthy partner for longer-term projects or for use as a personal assistant. It needs to be able to learn once, and then re-use that information at any time, regardless of the current token constraints of the main model.

#5 As a Human, train the AI to do better next time

• Once the AI is designed to learn, users need to be made aware that it can do so, and that it will remember them. If they trust that the AI is paying attention and using their input, they may take the time to try to teach it. This will particularly be true if it takes of the role of a personal assistant or advisor and the engagement is expected to be long-term.
• Any two human co-workers who plan to work together for a period of time will have a natural desire to establish processes, shared vocabularies, a knowledge of personal background and goals, and share other personal information. This will help them work teogether more effectively. This is also likely to be the case with human-AI pairings, where there is a mutually understood collaborative partnership in place.
• DESIGN RECOMMENDATIONS:
  Just as humans invest a large amount of time raising their children or cultivating friendships, they may also spend a lot of time crafting the perfect assistant. The system needs to be capable of learning, and then convey that to the user, along with privacy assurances.

#6 As a Human, verify the accuracy of the AI’s work

• LLMs have many situations where they can be inaccurate. They are prone to hallucination, they can be pushed with biased questions into giving unlikely facts, they typically don’t site their sources, they can’t introspect well, they often don’t know their own status, they often try to cover up errors in preference for providing an outcome, and they currently have severe working memory (i.e. short-term memory) limitations. The list goes on from there.
• When their token limit is stressed, LLMs can fail spectacularly, and worse yet, silently. Eventually they may get more reliable and more trustworthy, but it is likely we will need humans to show a healthy skepticism for answers and solution paths suggested by AIs.
• DESIGN RECOMMENDATIONS:
  Tools should be designed to encourage fact-checking and to make it simple to verify results.

#7 As an AI, verify the accuracy of the human’s work

• One thing that humans are not particularly good at is performing the same task over and over accurately. Humans don’t operate at optimal levels when they are stressed, or uncomfortable, or tired, or worried, or when under the influence, or when sick, or a variety of other factors. In short, humans will make errors eventually.
• Fortunately humans and AIs make different kinds of errors. A computer will do the same mindless task a thousand times extremely rapidly and accurately, while a human won’t. A human may be able to grasp errors in logic, known facts, goals, strategy, common sense, and other areas that AIs may struggle with. This makes them perfectly suited to fact-check each other.
• DESIGN RECOMMENDATIONS:
  Designers can build AI-driven interfaces to automatically check human input and look for ways to correct or improve it.

#8 As an AI, triangulate across humans to bring greater insights to this particular human.

• Out of the entire population of humanity, there are probably a few people that have already figured out optimal stock investment strategies, or the best way to build a self-driving car, or how to best design an electric jumbo jet, or how to best structure a democratic government, or how to organize humanity for long-term survival. There are many such critical questions we are struggling with. Often those ideas are suppressed for various reasons, or simply take too long to gain popularity due to lack of awareness.
• DESIGN RECOMMENDATIONS:
  AIs could easily crowd-source data that their users choose to pool, into a collective knowledge base for the betterment of humanity. It would enable users to consult an oracle, drawing on the newest and best ideas from everywhere. Currently we spend too much time re-inventing the wheel instead of using proven solutions. Designers could create an AI to change that.
  If we take this a step further, the AI could understand the goals of individual innovators, identify problems in their approaches, suggest alternate solution paths, and connect them up with results from other humans (and maybe the humans themselves) to automate collaboration. This could supercharge innovation and scientific advancement and make us look like we’re still collaborating by pony-express by comparison.

#9 As an AI, understand the goals of the human and suggest viable solution paths for them

• AIs (ChatGPT 4.0 in particular) are increasingly able to form logical models of the user’s interactions with it, and understand strategies for how to reach goals.
• One of the hardest parts of undertaking a task is determining which tools to use and which path or strategy to follow. It’s not just the skills or time to do the task itself, but an understanding of how to begin and how much effort will be involved.
• AIs can help with this. They have a world of knowledge about potential solutions, tools and strategies. If somebody in the world has found a one-click (or one-minute) solution to a problem, that solution path should be suggested early on, before more time is wasted exploring other paths that will be a waste of time.
• DESIGN RECOMMENDATIONS:
  Designers may need to prompt users to clearly articulate their goals, and this may end up being more important than asking for tasks to do. The human may not understand that the tasks they have chosen are actually not the best path to the goal.

#10 As a Human, I expect our conversations to be private. Now let’s talk and I will let you watch

• Imagine a collaborator who was with you 24 hours a day. They see everything you work on. They see all of your searches and computer activities. They have access to your biometrics from your smart watch on your wrist, the iphone in your pocket, and potentially connected devices such as Alexa, Amazon, or Audible. In short, they know you better than your own spouse, sibling, parent, or best friend.
• There is a lot of power in that. To know your inner-most secrets and desires allows the AI to coach you better. It can understand your unspoken and unconscious goals and offer suggestions on how to achieve them. It could help you more accurately and tirelessly than any human you know.
• There is also a lot of danger in this scenario. What if the AI is compromised and hackers blackmail you with your own private activities and desires? What if a corporation decides to use the knowledge of your weaknesses to manipulate you into new purchasing habits? What if a government wants to influence its population towards voting a certain way, or framing a political issues in a certain way? A trusted advisor and assistant could easily become a master if it is not exclusively representing and protecting its human, and perhaps humanity as a whole after that.
• DESIGN RECOMMENDATIONS:
  The question of how trustworthy and secure AIs will be is a question of technical architecture, data privacy laws, and UX design. LLMs are currently cloud-based, meaning that data collected about you is also in the cloud. It has the advantage of being accessible from anywhere and centralizing compute costs, but it also has the problem of potentially being out of the user’s direct control. Tiny Corp has the idea of creating a 15k super-computer that sits in your garage and runs LLaMA (an open-source LLM) out of the box. This might partially solve the privacy problem since you literally own the hardware and physically control the data on it. Some companies such as Apple have put “privacy-first” as part of their business model, and this is likely to be important for any companies generating these types of tools. UX designers will also need to be part of data-privacy-architecture discussions with their teams, to ensure the the AI can understand the difference between various levels of private data, verify recipients of shared data, and understand risks of data exposure to their primary user.

#11 As a Human, I expect for the modality, formatting, display and interaction with me to be customized to best accomplish my goal most effectively

• Everyone learns differently and some tasks are more naturally explored using speech, writing, drawing, pictures, diagrams, sound, smell, touch, or other modalities. Your AI should know what you prefer, and make the right modal choices based on context.
• There will be times when you want to draw something for your AI, or when you want to whisper to it, or when you want to use a hand signal to tell it something. It should be constantly aware and flexible with both input and output.
• There will be times when the output given is not entirely clear, and the user wants it in a different format to understand it better. Currently, in any LLM available (e.g. ChatGPT, Bard, Bing, Claud), if you ask for code, it will give you that code in its own formatting scheme. If you ask it to highlight the recent changes it suggested in red, it can’t do that. This is because the input content and the output content are considered flexible, but the output format for displaying code is not. There is no reason that the format for displaying code should be locked — its just that the designers haven’t gotten to the meta-level of UI design — which lets the user redesign the UI via natural language commands. This would be very useful.
• DESIGN RECOMMENDATIONS:
  Designers should erase more of the boundaries around what is or is not the “interface”. LLMs have already shown that you can ignore the GUI and just tell the computer what you want. Go a step further and realize that the GUI for interacting with the LLM chat is just another interface that should bend to user intent. If the user wants to speak, let them speak. If they want to type, let them type. If they want it in blue, or in CAPS, or with particular content highlighted, or with numerical relationships rendered as charts, let them do that. Designers need to relinquish control over the design and instead provide defaults and frameworks to work within. The interface is about to become much more fluid.

#12 As a Human, I expect to be able to ask the product how to use it, verbally give it tasks, and have it understand itself

• “Smart” products are often quite dumb. My Roomba often can’t connect to its own iPhone app and can’t tell me its status. It is capable of flashing different cryptic colored warning lights, but you have to decipher what they mean. It sometimes knows something is broken in its hardware, but it doesn’t know exactly what, and consequently suggests replacing everything. There is room for improvement here.
• If we apply the LLM-based interaction model (think: natural language, verbal, ongoing) to hardware products, we get a cleaning robot that we can direct to go clean a room (already possible using Alexa, but very constrained and prescriptive). We could also ask it if it is happy (i.e. are you experiencing any problems.) We could ask it if it is capable of automatically learning the layout of rooms, and it should know if that is how it technically operates. We could ask it if it saw the dog ball I’m looking for. We could ask it why it didn’t clean the kitchen the last time it went out. We could ask it for troubleshooting options if its right wheel isn’t rotating. In short, you can ask the product about itself and tell it what to do. You don’t need an “interface”.
• DESIGN RECOMMENDATIONS:
  What we are talking about here is a mini-LLM, trained on the product documentation, engineering specs, and help forum ideas, which resides on the robot hardware itself, possibly with over-the-air updates. It is a product that is genuinely smart, self-aware (to a degree), able to process commands directly via speech (not via an app or external buttons), and is able to accomplish user goals or troubleshoot itself. This is what all products should do, and none of them do it yet.
  Take this idea to its logical end and you get a shampoo bottle that you can verbally query to see how much is left in the bottle, or a jar of pickles where you can ask if it is expired, or a front door you can ask if it is locked. Basically anything could be able to talk to you at your request, and it should understand its own state and how to help you use it effectively. Perhaps “don’t speak until spoken to” would be an additional design guideline to consider.

Summary

Good products and good designs are highly focused on target personas and use cases. AIs will increasingly be their own persona, and will be able to accomplish advanced use cases, often in collaboration with the user. The high-level use cases listed here provide clues into the types of interactions that may take place and how to design for them.