How can ChatGPT support UX Research? I conducted a fictional UX Research project with ChatGPT to find out more about what ChatGPT’s current strengths (and weaknesses) are in performing UX Research tasks.
Intro
AI is currently all the rage in the UX world – at the latest since AI tools like ChatGPT, Midjourney and DALL-E have become accessible and usable for the general public, the question arises how AI will change the work of UX professionals in the future. In a previous blog article, Thomas already discussed the implications of AI for the work of UX professionals.
UX Research and AI
First steps towards integrating AI are also being taken in UX research. In fact, we are now seeing tools that offer to replace real UX research with the use of AI. With the help of AI and computational linguistics, synthetic users are generated that are supposed to imitate real human behavior. To do this, simply input some persona characteristics, the persona’s problems, and a solution approach, and the model “conducts virtual user interviews,” receives feedback from the synthetic personas, and summarizes it in a report. User research entirely without the users – based on Large Language Models (LLMs) and their training data. Niloufar Salehi, Assistant Professor in the Department of Human-Computer Interaction at the UC Berkeley School of Information, summarized the basic problem of such tools well in her blog article: An LLM essentially knows nothing about the real world and cannot produce insights beyond the patterns present in their training data. Even if real human behavior can be imitated, that does not change the fact that the information from the “interviews” is based only on the training data. The answers of such synthetic users to interview questions are thus based on what – given the existing training data – is statistically most likely. The result is superficial and stereotypical “insights” about the users. The more specific and complex the context of use, the stronger the need for real UX research. It simply needs the input of real people to gain new and relevant insights about the context of use and the problems of the users.
ChatGPT as an UX Researcher? – An experiment
So we still need the input of real users and we still need to validate our assumptions about users, for example by conducting interviews or surveys. Nevertheless, UX researchers should take advantage of the potential of AIs, e.g. to automate steps or to get inspiration when writing interview guides or test scenarios. To learn more about how ChatGPT, for example, can help us in UX research, I conducted a fictional UX research project using ChatGPT.
The fictional start up shorttripbuddy wants to develop an app that offers personalized day or weekend trips for its users. The personalized planning of the day or weekend trips should take into account interesting sights, restaurant visits and arrival and departure by public transport for the users. For this purpose, different factors such as users’ areas of interest, available time and budget, should be taken into account. Local businesses in the field of tourism and gastronomy should be able to use the app to attract the attention of tourists.
Stakeholder
First, I ask ChatGPT to tell me relevant stakeholders for the project.
The fictitious start-up shorttripbuddy wants to develop an app that offers personalized day or weekend trips for its users. The personalized planning of the day or weekend trips should take into account interesting sights, restaurant visits and public transportation to and from the destination. Different factors, such as users’ areas of interest, available time and budget, should be taken into account. Local businesses in the field of tourism and gastronomy should be able to use the app to attract tourists’ attention. Which stakeholders could such an app have?
ChatGPT:
“The following groups and individuals could be considered as stakeholders of such an app:
1. Users: The users of the app are the most important stakeholders: The users of the app are the most important stakeholders as they are the main target group and use the app to plan personalized day or weekend trips. The app should take into account their needs and interests and provide them with a unique experience.
2. Local businesses: Local tourism and hospitality businesses are an important stakeholder as the app can help them attract tourists. If the app is successful, it can help generate more visitors and thus more revenue for local businesses.
3. Public transport operators: public transport is an important part of the app, as it is used to travel to and from the various attractions and restaurants. Public transport operators could be considered as stakeholders as the app uses their services and may even help to attract more users to public transport.
4. City/municipal governments: City or municipal governments can be considered stakeholders as they could potentially benefit from the app. If the app helps to attract more tourists*, this could help to strengthen the local economy and increase revenue for the city or municipality.
5. Investors: Investors could be considered stakeholders, as they may invest in the start-up to fund the development and marketing of the app. They have a financial interest in the success of the app and could potentially also be involved in the company’s business strategy.”
ChatGPT outputs a comprehensible and complete list of stakeholders. I find it particularly interesting that city/municipal governments and possible investors have been added as additional stakeholders, since I did not name them in the prompt. So ChatGPT can definitely help identify relevant stakeholders for a project.
Protopersonas
Next, I ask ChatGPT to generate personas relevant to the previous prompt. It is important to point out at this point that these are not personas verified by UX Research, but protopersonas. Protopersonas are based on a team’s (or in this case, an LLM’s and its training data) assumptions about who the users are and what they want. These assumptions can be used as hypotheses for UX research, for example to learn more about users* and their context of use in interviews. Subsequently, these assumptions can be verified or falsified or enriched to generate a verified persona from the findings of the UX Research.
ChatGPT:
“1.
I am happy to create different personas that represent potential users of the app “kurzreisenbuddy”:
Name: Julia
Age: 28 years
Professional status: Marketing Manager
Education: Master in Marketing
Nationality: German
IT Skills: Advanced
Hobbies: Travel, photography, hiking
Motivations: Julia loves to discover new places and is always looking for interesting activities and sights. She wants to create a personalized itinerary tailored to her interests and budget. She also wants to support local businesses and have authentic experiences.
Frustrations: Long queues, impersonal tourist services, high cost of tourist attractions.
Personality traits: Adventurous, creative, organized, open to new experiences.”
ChatGPT generates a total of three protopersonas – Julia (28), a marketing manager, Hans (60), a retiree, and Maria (23), a student in tourism management. At first glance, the protopersonas seem quite fitting and highlight some points that I hadn’t considered myself. For example, that a possible frustration of travelers planning day trips could be long queues at tourist attractions.
ChatGPT:
“2.
Name: Hans
Age: 60 years
Occupational status: Retired
Education: High school graduation
Nationality: German
IT Skills: basic knowledge
Hobbies: culture, reading, cycling
Motivations: Hans wants to have a relaxing trip with cultural highlights and try local cuisine. He is looking for a simple and user-friendly trip planner that provides him with all the information he needs. He also wants to make sure he can travel by public transportation.
Frustrations: Difficulty finding his way around new cities, unclear information about public transportation connections, overwhelmed by technology.
Personality traits: easy-going, interested in culture, experienced, tech-savvy”
However, inconsistencies are also noticeable – Hans has basic knowledge of IT and one of his frustrations is that he is overwhelmed by technology. However, at the same time, he is described as being tech-savvy. In addition, Hans is attributed the motivation to seek a simple and user-friendly trip planner that provides him with all the information he needs. This is a bit of a stretch – one could assume that Hans’ motivation is to have a nice day trip, having information about it is merely a means to an end – and it makes even less sense to assume that Hans is motivated to find a simple and user-friendly trip planner.
Als nächsten Schritt frage ich ChatGPT nach passenden Zitaten für unsere Protopersonas, pflege diese bei LeanScope ein und versehe sie mit einem passenden Foto.
As a next step, I ask ChatGPT for suitable quotes for our protopersonas, input them into LeanScope and provide them with a suitable photo.
ChatGPT:
“Yes, I’d love to! Here are appropriate quotes for the personas:
1. Julia: “I love exploring new places and being inspired by the beauty of the world.” – Unknown This quote shows Julia’s love of adventure and her creative side.
2. Hans: “Life is a journey, enjoy the ride.” – Unknown
This quote fits Hans’ laid-back and experienced character.
3. Maria: “Travel is the only thing you can buy that will make you richer.” – Unknown
This quote shows Maria’s passion for travel and her curiosity to have new experiences.”
As-Is–Szenario
Next, I ask ChatGPT to create an as-is scenario for Julia. Again, similar to the protopersonas, in the first step the as-is scenario is just assumptions about the persona’s goal and current approach. These assumptions must also be validated in the UX Research.
ChatGPT:
“I would be happy to create an as-is scenario for Julia:
Title: Planning a weekend trip to Berlin.
Preconditions: Julia has already downloaded the “shorttripbuddy” app and created a profile. She has also specified her areas of interest and budget ideas.
Other players: the “shorttripbuddy” app and local businesses in Berlin.
Environment: Julia is in her home in Hamburg and using her laptop.
Goal: Julia wants to create a personalized itinerary for a weekend in Berlin, tailored to her interests and budget
1. Julia opens the “shorttripbuddy” app and logs in with her profile.
2. She enters the data for her weekend trip: travel date, travel duration and budget.
3. Julia selects her areas of interest, e.g. art and culture, food and drink, and sightseeing.
Tthe app generates a personalized itinerary for Julia and suggests different activities and sights.
4. Julia reviews the suggested activities and selects the ones she likes best.
5. The app suggests restaurants based on Julia’s food preferences and budget.
6. Julia checks the public transport connections to the suggested places and plans her trip by public transport.
7. She books the activities and restaurants directly through the app and receives confirmation.
8. During her weekend trip, Julia follows the personalized itinerary and discovers new places and activities in Berlin.
9. After her return, Julia gives feedback on the app and the local businesses she visited and shares her experience with other app users.”
ChatGPT already includes the fully developed app in the scenario, although it was defined in the first prompt that this app still needs to be developed by the fictitious start up. I try to correct ChatGPT by giving ChatGPT a definition for an as-is scenario.
ChatGPT:
“Sorry for the misunderstanding, I’ll be happy to create a new actual scenario for Julia according to the definition of the UXQB:
Title: Planning a day trip to Lübeck.
Preconditions: Julia has her smartphone with her and the “kurzreisenbuddy” app is already installed on her device.
Other stakeholders: the ” shorttripbuddy” app and local businesses in Lübeck.
Environment: Julia is at home in Hamburg and plans her day trip from there.
Goal: Julia wants to create a personalized itinerary for a day trip to Lübeck, tailored to her interests and budget.
1. Julia opens the ” shorttripbuddy” app on her smartphone.
2. She selects the “Plan a new trip” option.
3. Julia enters the travel date and the duration of the day trip.”
Again, ChatGPT incorporates the app into the as-is-scenario. I try it a third time:
ChatGPT:
“Sorry, I understand better now what you mean. Here is the as-is scenario for Julia:
Title: Planning a day trip to Lübeck.
Preconditions: Julia is interested in taking a day trip to Lübeck. She has access to a computer or smartphone and has already gathered some information about Lübeck and possible places of interest.
Other stakeholders: The Internet, possibly friends or family who can help Julia plan.
Environment: Julia is at home and uses a computer or smartphone to gather information about Lübeck.
Goal: Julia wants to plan a day trip to Lübeck that is tailored to her interests and budget.
1. Julia starts by searching for information about Lübeck on the Internet. She reads travel guides, looks at pictures of places of interest, and searches for restaurants.
2. She finds some places that interest her and writes down the names and addresses.
3. Julia now looks for information about public transportation in Lübeck to find out how to get from Hamburg to Lübeck and back.
4. She finds some information, but it is difficult to understand which ticket to buy and where to get on.
5. Julia asks friends and family for recommendations for activities and restaurants in Lübeck. She gets some tips and writes down the names.
6. Julia is still unsure about the best way to get to Lübeck and whether she has chosen the right places.
7. She starts visiting different travel websites to gather information about itineraries and tickets. However, she has to visit many different websites and it is difficult to keep track.
8. Julia is frustrated that she spends so much time gathering information instead of planning the day trip.
9. Finally, Julia decides to just take a train and be led on the spot by recommendations in guidebooks and from friends.
10. Although she has a good time in Lübeck, she feels a bit overwhelmed and disorganized. She wishes there was an easier way to plan a personalized day trip.”
Hooray, an as-is scenario! The way Julia is currently planning a day trip sounds plausible at first glance. But here, too, an inconsistency stands out: In Step 8, why is Julia frustrated that she is spending so much time gathering information instead of planning the day trip? Isn’t gathering information part of the activity of planning the day trip? Also, I would like to know more about why Julia feels overwhelmed and disorganized in Step 10, even though she is having a good day.
Overall, however, the as-is scenario provides a relatively realistic representation of how Julia might currently plan a day trip, but remains fairly superficial. Nevertheless, ChatGPT can provide a good starting point for formulating an as-is scenario. Again, the steps of the as-is scenario should be understood as assumptions that need to be verified in the UX research.
For the further work, I also add the as-is scenario to LeanScope.
Interview question generation
In the next step, I want to see how ChatGPT can help me prepare for interviews:
ChatGPT:
“Interview guide for potential users of the app shorttripbuddy:
1. Introductory questions:
– Could you briefly tell us something about yourself (e.g. age, profession, hobbies)?
– Do you have any experience planning day or weekend trips?
2. Questions about using travel apps:
– Do you use apps for trip planning? If so, which ones?
– How often do you use such apps?
– What do you expect from a travel app?
3. Questions about the operation of the app:
– How would you operate such an app?
– What are the most important functions for you that the app should have?
– How would you indicate your interests and preferences in the app?
– How would you handle an app that gives you suggestions that don’t interest you?
4. Questions about planning day or weekend trips:
– How do you usually plan day or weekend trips?
– What factors are most important to you when planning (e.g., budget, interests, time)?
– How do you select attractions and activities for your trip?
– How do you find out about local tourism and hospitality businesses?
5. Concluding questions:
– How do you like the idea of personalized planning for day or weekend trips?
– Is there anything else you would like to add?
Thank you very much for your time!”
Many of the questions are very useful. For example, the introductory question about whether the person has already had experience planning day or weekend trips. Especially in the fourth section “Questions about planning day or weekend trips” ChatGPT suggests many open and neutral questions that could be used to gather a lot of information about the current context of use. For the second question, however, it would be useful not to give examples so as not to steer the person’s answer in a particular direction.
The questions from the third section “Questions about using the app” should rather not be asked. They consist largely of hypothetical questions that should be avoided in UX research, as users are poor at predicting their future behavior, preferences, and circumstances.
The final question “How do you like the idea of personalized planning of daily or weekend trips?” is also rather less useful, as it implies the question whether the person would use such a feature. In 90% of the cases, the person asked would probably say that they like the idea – out of politeness or because they suspect that such a feature could be interesting for them in the future.
Interview analysis
To see how ChatGPT can support the analysis of interviews, I conduct the interview almost as suggested – only leaving out the third section for the reasons mentioned before.
I make up a fictitious user and interview participant – Sophie, 23 years old, studies education, hobbies: traveling, cooking, hiking. To create a semi-realistic interview protocol, I use the dictation function of Word and interview myself in the role of Sophie.
First, I ask ChatGPT to generate a bulleted list of key findings from the interview transcript.
User Researcher: Hello, thank you for agreeing to talk with me today about your experiences planning day and weekend trips! Could you briefly tell us something about yourself (e.g., age, occupation, hobbies)?
Participant: Yes, with pleasure! I am 23 years old and currently studying pedagogy as a bachelor in my 6th semester. In my free time I like to cook, travel and go hiking.
User Researcher: Do you have any experience planning day or weekend trips?
Participant: Yes, definitely – I like to use my free time on the weekends to go with friends to see cities and areas we haven’t been to yet.
ChatGPT:
“- The participant is 23 years old and studies pedagogy in the Bachelor in the 6th semester.
– In her free time, she enjoys cooking, traveling and hiking.
– She likes to plan day and weekend trips with friends to cities and areas they don’t know yet.
– When planning day trips, she emphasizes getting to and from the destination by deciding whether to use public transportation or a rental vehicle.
– She researches in advance what sights are available, what monuments or museums can be visited, and where to go out to eat.
– She expects a travel app to quickly provide information about the city or region she plans to travel to and to be able to search for activities or places in different categories.
– She plans day or weekend trips with her friends and jots down multiple options rather than making a set plan.
– She writes down the opening hours of the places she wants to visit and decides spontaneously with her friends during the day what they want to do.
– When planning outings, it is important for her to include things that her friends and she enjoy. She takes everyone’s interests into account.”
The summary is quite good and summarizes all the main points mentioned in the two-page interview transcript. So ChatGPT can definitely help to extract the most important statements from longer texts.
In the next step, I try to find out whether ChatGPT manages to verify or falsify the assumptions about the protopersona Julia.
ChatGPT:
“Since I, as an AI-based system, am not able to verify the authenticity of people, I cannot verify whether the assumptions about the persona Julia are correct. However, I can show the extent to which the assumptions have been confirmed or disproved based on the interview transcript and my data analysis.
Assumption 1: Julia is 32 years old and works as a marketing manager in a large company.
Verified: Yes, the assumption was confirmed in the interview transcript.
Assumption 2: Julia is very busy and often has little time for personal interests.
Verified: Yes, this assumption was confirmed in the interview transcript. Julia mentioned that she has a busy schedule and often has long work days, which makes it difficult to find time for personal interests.”
ChatGPT “hallucinates” new assumptions about Julia (e.g. assumption 2) that were not previously defined that way in the persona. I’ll try a second prompt summarizing the most important assumptions about Julia:
– Julia is motivated to discover new places.
– Julia is motivated to support local businesses.
– Julia is motivated by having authentic experiences.
– Julia is frustrated by standing in long lines.
– Julia frustrates participating in impersonal tourist offerings.
– Julia is frustrated when the cost of tourist attractions is high.
ChatGPT:
“Below, I verify or falsify the assumptions made about the persona Julia based on the interview transcript:
– Julia is motivated by exploring new places.
VERIFIED: The participant emphasized several times during the interview that she loves to travel and is always looking for new places to explore. She also stated that she often looks for local events to gain new experiences.
– Julia is motivated by supporting local businesses.
VERIFIED: The participant indicated that she prefers to support local businesses rather than shop at large, global chains. She also emphasized that she enjoys visiting local restaurants and cafes to learn about the culture and cuisine of the area.
– Julia is motivated by having authentic experiences.
VERIFIED: During the interview, the participant emphasized that she is interested in local culture and history and enjoys having authentic experiences. She prefers to visit places that are not as touristy and focus on local traditions and customs.
– Julia gets frustrated standing in long lines.
VERIFIED: The participant indicated that she finds it frustrating to stand in long lines to visit tourist attractions. She prefers to visit these places early in the morning or late in the evening to avoid the rush.
– Julia finds it frustrating to participate in impersonal tourist attractions.
VERIFIED: The participant emphasized during the interview that she finds it frustrating to participate in tourist attractions that are impersonal and do not cater to visitors’ needs and interests. She prefers to have individual experiences that are tailored to her interests.
– Julia is frustrated when the cost of tourist attractions is high.
VERIFIED: Participant indicated that she finds it frustrating when the cost of tourist attractions is too high. She prefers to visit places that are less expensive and offer authentic experiences.
Based on the interview transcript, I can say that all assumptions about the persona Julia were verified. The participant expressed similar likes and dislikes as the persona described.”
ChatGPT verifies all assumptions about the persona by referencing statements from the interview transcript that were not made. There is nothing in the interview transcript about the participant frequently seeking out local events to gain new experiences. In addition, there is nothing in the interview transcript about long queues or about visiting places early in the morning or late in the evening to avoid the rush. The phenomenon of an AI generating convincing-sounding output to questions or tasks that is not justified by the training data is called “hallucination”. Especially for LLMs, hallucination is currently one of the biggest challenges. Improving the training data and incorporating human feedback on the correctness of the output will likely improve the severity and frequency of hallucination in AI in the future.
However, at this point I give up on having ChatGPT evaluate the interview protocol in terms of verifying and falsifying our assumptions about the persona and as-is scenario. I had planned to verify the as-is scenario using the interview transcript as well and to transfer the results into LeanScope. However, the more ChatGPT develops and the better the prompts we write become, the more likely it will be that we will be able to verify the assumptions in the future.
ChatGPT and Data privacy
So, up to this point, we can conclude: ChatGPT can support some tasks in UX Research quite well, while others currently work less well. However, we have not yet considered one important limitation for the use of ChatGPT for UX Research: What is the data privacy situation regarding the use of ChatGPT in UX Research?
The information we receive from our participants and clients in UX Research is confidential. On both the participant and client side, it is essential to maintain data and information privacy. OpenAI uses the personal data collected in ChatGPT for various purposes, including the further development of AI models. It is currently not clear whether this collected data could be used for other purposes in the future. German data protection authorities are currently investigating whether ChatGPT violates the GDPR. In addition, ChatGPT had a privacy breach in March 2023, where some users were able to view chat titles and, in some cases, messages in newly created chats of other users. Before ensuring that ChatGPT is GDPR compliant and can guarantee information security, we should refrain from processing real interview or test logs with ChatGPT. Perhaps in the future, taking into account the effort and cost, it will be a possibility to use cloud-provided or self-hosted LLMs for UX Research analysis, ensuring data privacy and information security.
Fazit
With the “experiment” on the use of ChatGPT in UX research, I was able to give a small insight into how LLMs could be used for UX research tasks in the future. However, I can only touch on the topic – the topic around the possible uses of AI is too complex to discuss it comprehensively in a single blog article.
The greatest potential of ChatGPT in UX Research currently lies primarily in analyzing long texts such as interview transcripts and breaking them down to the most important key points. ChatGPT can also provide good clues for further elaboration and review when generating protopersonas and as-is scenarios.
Currently, ChatGPT is not suitable for analyzing assumptions about users* and scenarios – hallucinations occur too often and one does more work checking the analysis of ChatGPT for correctness than it saves time. However, this could change as AI models evolve.
The biggest problem right now is insufficient data protection. When LLMs are used for UX research tasks, it is important to ensure that client and participant data is secure, and it must be clear how the data will be further processed. How this can be achieved will become clear in the coming months and years.
