Evaluating my initial implementation
Earlier this (academic) year I looked at the existing behaviour of football fans on the internet. After that I created a prototype in several iterations (evaluating, adapting, evaluating, …).
I’m currently working on a good method for evaluating the user experience of my initial implementation (implementation of the first version is more or less done). The basis for this method will be the way my group evaluated the Facebook app we made for last years user interfaces course (think-aloud protocol). I also used this method for evaluating the paper prototypes.
I’ve been looking back at how I evaluated my prototype. While I tried to do it thoroughly (several iterations), I think there is room for improvement. There are some lessons to be learned. Each iteration about 8 persons evaluated my prototype. Several of them had no affinity towards football and I now believe this isn’t ideal. When it comes to usability, those evaluations were very useful. However, when it comes to the broader concept of user experience, I think its much better to focus on persons that are interested in football. I don’t think I can ask non-football fans to pretend to be one (it doesn’t work). A system can be very good from a usability point of view, but still have a poor user experience. Also the opposite is possible. There is research that indicates Facebook isn’t very good from a classic usability point of view (using the classic heuristics) even though Facebook users do like the user experience it provides.
The user experience I’m trying to create is (obviously) aimed at football fans. I believe I should evaluate it with football fans to get the best idea of how the user experience is working out (without losing sight of usability).
I think it is important to determine some goals for the evaluation. Of course my intention is to evaluate the initial implementation completely but I believe extra focus should be put on certain parts to get more information about them because they are vital to the design. Next to the think-aloud procedure, I will ask the test persons questions about certain aspects of the implementation. I did this for the prototype as well and I believe this was a good way of getting specific additional information on things the test persons otherwise maybe didn’t even say anything about.
There are two main components to the design. There is the news side by which football fans can keep themselves up to date about their favourite clubs, players, competitions, national teams,… in football. This content will be provided by users themselves (links to news sites etc.).
The other component is the social networking component specific to football. This involves ways to share their favourite football teams, players, their football-related activities, their activity on the news component (submitted stories, voted on stories, comments on stories).
I want to verify as good as possible if the combination of these components is something football fans are interested in. In the prototype evaluation this was certainly the case but a significant part of my evaluation group wasn’t interested in football so more evaluation of the main concept should be done.
Next to evaluating these two components in general, I will evaluate them in detail. If the design choice of going for these two main components still proves to be the correct one, It is important to look at the way they are designed. Is the approach for the news system right? Is the procedure for submitting news good? Do people want to submit news like this? Is the method for sorting the news good? Does the site offer something better in terms of news compared to alternatives? What should be put on the profiles? What activity of their friends interests them? (and a lot more)
I believe that a thorough evaluation using the think-aloud protocol will get users to give a lot of very useful information but in addition to this I also want to ask some questions about crucial parts of the application. Maybe a user won’t say anything about the profile content itself, and is a question required to get more information about this.
My approach now will be to further determine which parts of the components need special attention to get more information about what users want (by composing a list of questions and goals). Then I’ll do one or two evaluations to see if my approach itself is working, after which I start evaluating the initial implementation. The deadline for this is March 1st.
To summarise: I think it’s important to do these evaluations very consciously. I need to have a good plan to make these evaluations as useful as possible. I need to determine what works and what doesn’t and in order to do this I need to get the right kind of information. Based on the information gathered during this first round of evaluations, I will need to make some decisions about how to move forward.
Filed under: Uncategorized | Leave a Comment
No Responses Yet to “Evaluating my initial implementation”