Please, test it.
The fact you believe in your idea, or that your investor do as well, it means a lot about creating a wonderful team and culture.It does not mean your customer will buy into that.
There is a long way between what the market opportunity (as the market analysis and the founder’s vision is) and what the actual product is. Of the thousands of products in different countries that seems to solve the same problem (you may call them competitors if you want), which one will win the customers’ favor now and in the future?
Usability test helps to provides insight on that.
Or in other words. usability test tells you, at the current state of mind, your customer will most likely find your product to be value if you do … in the following way … and … in the following way ...
As you can see, these findings can be translated to what you can include in your MVP, why a certain design or feature did not meet the expectation (or will ever meet) and what you may decide to remove so you can focus on what actually matters.
In my company, we started doing usability test every week, Here, I want to share with you 9 points I learnt from the process:
Point 1: Recruit the Right Candidates
You will probably say, “of course, otherwise why do I bother”.
As for us, we had given a very wide range of potential user group “people between 30 to 65 years old; shop online regularly; and use smartphone and/or computers regularly”. Do you know how many people this means in Hong Kong in year 2016, 4.07 million of them.
Not all of them are good testers, you want to find a representative sample, that include male and female, all age, different professions, interact with smartphone and computers in different ways (e.g. some shop on mobile, some shop on PC etc). You need to know all that before they turn up in your office for the test to decide what to test on. And above all, they should be open to talk. You never want to have people coming in and say “well it is good” to everything you do. As encouraging as it is, for me that is not the most valuable advice I am trying to get out of the test.
I found that getting on a chat with the tester is extremely useful. In Hong Kong we are very used to texting on Whatsapp, WeChat and Messenger. I just get their number and chat for 5 minutes. For the first few rounds when I am not sure who gets me the best results, this has been proven to be extremely helpful. And then, based on the finding, I wrote the screening questions and I task my team to recruit for the next rounds.
The reason is simple, remember my potential user group? Screening by those criteria is so ineffective that is useless. A 5 minutes chat gives you the same amount of info equally to around 50-60 screening question and more importantly, which of the 10 out of the 50-60 you can use for later recruitment.
Point 2: Have Simple and Short Scenarios
At least in the beginning, break down the user journey and test them separately. If you have to, plan some stops on the way and ask some questions. When people are clicking through your screens and there are a lot of them, they will eventually go into the “auto-pilot” stage and don’t remember what they did. I would not suggest more than 5 steps in a scenario or at least stop and talk about something. Another reason I do this, is that it is also a good opportunity for the instructor to quickly reflect and catch the questions before they vanish.
Point 3: Ask the Tester to Give Examples
Let me use an actual case to explain what it is about, one of the questions I have been asking is about how people go about making decision on what to buy online, but I am not going to throw this question out to them because it is likely I cannot do anything with the finding, they will be generic and thus not executable.
Instead, I asked, “Can you describe your recent online purchase of apparel to me? “ or “Can you describe your recent online purchase of personal electronics to me?” (These are just examples, you will find from time to time the tester remember too little details to be meaningful, so you change the subject to get more insights; and remember, as why they cannot remember).
Another example, I found some testers are struggling on a page; so I ask them to give me some examples of other applications they find to have a less confusing layout, as well as some that are equally confusing or even more confusing. What I am trying to get are materials I can compare later to understand the actual reason for the confusion.
Point 4: It is Equally Important to Identify What is Missing as What is Wrong
Outliners are brilliant, they are very likely your solution to the problem.
That is how antibody and vaccine comes from.
My most recent use case was rather simple and honestly stupid，I was testing the product listing page and want the testers to click on the share button. Most of the testers had problem to share button, few of them did not. So now I am missing the “be confused” part, I found the icon I took reference of is also used by many of them who also use Flipboard, ASOS etc which all use the same icon, they recognize the icon. While the majority do not use these application. So I choose to go with the other alternative which is equally common, tested it and got a much better result.
Another thing that is often missing is the time, I mean, when a tester takes longer than expected on a certain page or to complete a flow. You probably should ask what caught their attention. You may not find problems, maybe they just like the design, or they do not understand a word you included in the text (a problem you don’t want to tackle at the moment), but it is valuable to know what happened.
Point 5: Don’t ask What They Want, Ask Which They will Choose
"What do you like”, is a pandora box. You will get a lot of wonderful things, most of them not about the problem you are trying to solve. I once tried to ask, among my friends, what they would like the color of Facebook icon to be, I had hoped that most will come back and say, I think it is pretty good the way it is; instead, I had 10 different answers from a sample size of 10 and nobody would prefer it to stay like the way it is.
Most of the time, you are just going to take one approach, at least to start with, there is no need to complicate this from the beginning when the system is not stable, to complicate the matter by giving a lot of options, unless that is your core business. Just give them a choice. An A/B test, A or B, the tester have to choose one.
Point 6: Have some Questions to Help You Further Verify and Enrich Your Persona
I picked up this advice from Oliver Hass when I went to a General Assembly workshop on User Experience. Since then I have been using that with my testers and find it to be very helpful. I ask what Facebook group they follow, what Instgram people they follow, how do they read news etc. Then I have a look at what they usually see if there is any relationship between how they live and the test results
Point 7: Be Friendly; Not to Be Judgmental
Well, at the end of the day, we try to have fun together and collect some useful insights right? There is absolutely nothing wrong about what the testers are doing. And actually, the more mistakes they make, the better your products can be in the future.
Some opening questions can help relax the tester, after all, it is an unfamiliar environment, facing some cold machines, we all feel a bit uneasy. Ask them how was their week, as about their shopping (as I am working on an e-commerce project) etc, these questions help. I often take the tester to the coffee machine and we make a coffee together, this provides to be helpful as well (you can make a ice cream, I guess that works even better)
Point 8: Find the Commonality Behind all the Noise
Finally, this is the most difficult part; after all the tests you have done, let’s see what the end result give us. In a perfect world, I hope to transcript all the interviews, and match them with the video recording, but I work in the startup, recourses are limited. If you have the same problem as I do, my suggestion is to get a good video-to-text software. A free one is actually Youtube, upload your videos there and you can download the subtitle, it is terrible obviously if you are using it to watch the video, but for me it is good enough for key word analysis.
I also share the videos and the findings with the team, and literally take them through the research summary phase. All videos, mockups, transcripts and summary presentations are available on company shared directory. I love when they tell me what they are interested in and want me to ask. I love feedbacks, I don’t get them a lot though…
At the end, I am trying to get out of this exercise, are answers to the following 5 questions:
- What elements of the different design the testers prefer
- What elements caused confusion
- What alternatives I can use to try to reduce the confusion
- What is missing that is critical for testers to proceed to the next step
- What is not absolutely necessary that we can consider to develop later on
Nevertheless, what usability test cannot give you, most of the time, is an absolute answer to the above questions. Well, even in the case you have surveyed 100% of your user group, you will probably not find that they overwhelmingly prefer one over the other (it is not entirely impossible, especially when the product you are testing is very revolutionary). You should take a positive and reasonable attitude towards the usability test. It does not tell you what to do, it helps make it better.
What it is also not going to do, is to find all the problems, it will find the major ones, and after you fixed them; you find the tier 2 problems; it goes on. This is the great journey because you can clearly see the product is getting better, more lean and smooth. Good luck with your usability tests, if you have any specific problems or questions, you can reach out to me on Facebook (April YU) and I am more than happy to discuss.
Comments
Post a Comment