How to use AI to plan and run usability tests
Where AI saves real time in usability testing, from test plans and scripts to note-taking, and where you still need a human in the room.

Most small teams know they should test their designs with real people, and most of them skip it. Not because they think it is useless, but because the prep feels heavy: writing a plan, inventing realistic tasks, drafting a script, booking people, taking notes, then turning a pile of notes into something the team will act on.
AI tools are good at exactly that kind of preparation and clean-up work. They are not good at being your users. Used in the right places, they can cut a week of testing overhead down to a couple of days. Here is how we split the work between the model and the humans.
What AI should and should not do in a usability test
Start with a clear line, because it is easy to drift. AI is a strong assistant for structure and writing. It is a weak substitute for watching a real person struggle with your interface.
- Good uses: drafting the test plan, turning goals into tasks, writing neutral questions, building a note-taking template, summarising transcripts, grouping observations.
- Bad uses: pretending to be a participant, predicting what users will do, deciding which problems matter most without you reviewing the evidence.
A model can tell you what users often struggle with in checkout flows in general. It cannot tell you that your users miss the coupon field because it sits under the fold on their phones. Only a session shows you that.
Turn a fuzzy goal into a test plan
Most tests fail before they begin because the goal is vague, such as "see if the new onboarding works". Give the AI your context and ask it to sharpen the goal into questions you can actually answer.
A prompt that works well for us:
Here is our product, our audience and the flow we changed. Write a one-page usability test plan with three research questions, the tasks that would answer them, what success looks like for each task, and five participant criteria. Keep it plain and short.
Then edit hard. Remove research questions you cannot act on this month. Check that each task maps to a real decision the team needs to make.
Write tasks that do not lead the user
Task wording is where tests go wrong most quietly. "Click the Upgrade button to change your plan" tells the participant exactly where to go. "You want to move to the plan with more storage. Show me how you would do that" tests whether they can find it.
AI is useful here as a second reviewer. Paste your draft tasks and ask it to flag any that use words from your interface, hint at the answer, or contain two tasks in one. It catches more of these than you would expect, because you know your UI labels too well to notice them.
Draft the script, then read it out loud
Ask the model for a full moderator script: a short welcome, consent and recording notes, a reminder that you are testing the design and not them, warm-up questions, the tasks, and closing questions. It will give you a solid first draft in seconds.
Then read it out loud once. AI scripts tend to be too long and slightly formal. Cut until it sounds like you talking to a person. Keep the warm-up to two or three questions so the session does not run over.
Take better notes during the session
During the session, your attention should be on the participant. If your tools and your participants' consent allow it, record and transcribe the session so you are not typing everything live. Keep a simple paper or digital sheet for the moments a transcript misses:
- Where they hesitated, and for how long
- Where they clicked first, even if it was wrong
- Facial expressions or sighs at specific screens
- Anything they said that surprised you
Always tell participants how the recording and transcript will be used, and do not paste personal details into AI tools your company has not approved.
Synthesise with AI, decide as a team
After five or so sessions, you will have transcripts and notes that are tedious to read side by side. This is where AI saves the most time. Ask it to list every observation per task, with the participant number and a short quote, and to group similar issues across participants.
Then check the output against the source. Models sometimes merge two different issues into one or invent a cleaner quote than the person actually said. Spot-check a few quotes against the transcript before you trust the summary.
The final step stays human: decide which issues are severe, which are cheap to fix, and what you will change before the next round. A simple grid of severity against effort works well for that conversation.
A lightweight weekly rhythm
The real benefit of AI-assisted prep is that testing becomes small enough to do regularly. A rhythm we like for small teams:
- Monday: update the plan and tasks with AI, recruit three to five people.
- Wednesday: run the sessions, 30 minutes each.
- Thursday: synthesise with AI, review the evidence together.
- Friday: ship the top fixes or put them in the next sprint.
Use AI to remove the friction around testing, not the testing itself. The plan, the script and the summary can all be faster. The moment a real person gets stuck on your screen is still the most useful thing you will see all week.


