To evaluate the user interface and user experience of the first version of my social network site I conducted an evaluation experiment. 9 football fans participated in this evaluation which consisted of an evaluation by think aloud procedure and a questionnaire. These evaluation sessions, on average, were about 40 minutes long. In this post, I will discuss the problems that became apparent during these experiments. I’ll also give my thoughts on some of the issues/suggestions cause I have to decide on which to act.
I will be doing one more evaluation experiment on wednesday, with my thesis supervisor prof. Duval. I am mentioning this because it means some more issues could be found, and I could change my mind about which things to focus on based on his thoughts.
1 General concept
The purpose of the site is to enable football fans to get the football news that interests them all in one place and to connect with each other. Users submit news they come across on sport websites, newspaper websites, blogs, … and vote on what they like. Users can see what their friends are submitting and voting on. The news is tagged and this categorization is used for the navigation between news streams.
All the test users liked the general concept. No fundamental problems, problems with the general idea behind the site were signaled. The test persons provided great feedback to improve the site. Even though they considered the site to be user friendly in general, there are a lot of things that can be improved.
For each part of the site, I will list the information gathered during the evaluations. These are issues, suggestions, remarks,…. Some issues were reported by multiple test persons (or I noticed it by analyzing their behavior), others only by one. I will definitely act on the issues reported by several users. Some things that were reported by only one test user had a more preferential nature. It is not a good idea to get into all of those. I will look at them on an individual basis and judge whether they are things that should be fixed or added in case of suggestions.
For each part of the site, I will list the issues and suggestions. Ideas relating to issues or things that should be improved will be put under issues as well. Suggestions will be the kind of things that are more about new features then about current features or solutions to address issues.
2 General
2.1 Issues
• Two test persons found the order of items in the top menu to be illogical. They thought putting “About” in beween “Join” and “Login” was weird. This will be changed. You can see the problem in the image below.
• At first it isn’t very clear that news isn’t automatically fetched from news sources but its submitted by users. I believe things like this are important issues for every website. Its about communicating purpose, about writing copy. This is something that will require an ongoing effort, not something that can be fixed easily.
• Redirects on actions that require user login don’t go back to original page after user logs in. This should obviously be fixed.
• Theres too much blue in some screens, especially the profile / friends feed (pictured below). This will be addressed by doing something about the background colour of story titles.
• Having to login right after signing up.
2.2 Suggestions
• Add widgets with latest news items of sports sites: My opinion is that it could be nice to have, but its a threat to the focus of the site as well.
• Add a way to list fans of a certain club. This is a way to stimulate social discovery and I think it should be added.
• Have the algorithm that takes care of sorting the news stories take into account the number of friend votes. There are several things to consider here. It adds complexity to integrate this to the algorithm, it makes the site less straight forward. There are advantages to having a uniform ranking system. I think its a good idea to show how many friends voted on a story but at this point I don’t want to change the algorithm in this way.
• Integrate friends feed as a category rather then a separate page. I think I like this idea. It increases focus by eliminating a separate page. At the moment there aren’t many kinds of friend updates implemented yet, this is a factor in this decision as well. I believe the friend feed still requires a lot of polishing anyway.
3 Categorization
3.1 Issues
• There is room for improvement when it comes to the top menu (picture below). There is a preset menu for people that are not logged in but logged in users can customize it to fit their interests. The feedback I received is that it isn’t visible enough and that it would be a good idea to use different colours (for example different colours for clubs, players, leagues,…). Will be implementing this idea.
• There is a category “World Wide” which every story is part of (can be seen in the first screenshot of this blog post). Several users noted that this wasn’t a very good description (“World wide”). Definitely need to rename this.
• There currently is no way to search for categories or an overview page containing them. These things are lacking at the moment and will be added.
• There should be a way to organize ones favourite categories. At the moment it isn’t possible to arrange the favourite categories, this was something not included in the initial implementation but it should be possible (preferably enabling the user to do this by dragging them around in some kind of edit mode, similar to arranging apps on the iphone). It should also be easy to add/remove categories in this edit mode.
• The way in which related categories are shown isn’t good (too small being one factor):
3.2 Suggestions
• A recommended category containing all the news from all your favourite categories combined. This probably is a very good thing to add. I had it in original designs but didn’t implement it for this version.
• Use the source as a category as well. This way people can look at the most relevant news items from a certain source.
• Custom categories. This could be used to for example make a list of players you want to follow and put them under one category “favourite belgian players”. This is definitely something interesting, its a bit a more advanced feature though
• Change the category system so that the news overview shows the news from the currently active categories. The user would be able to activate/deactivate categories at the top. Doing this would mean a big change in the sites concept. Definitely not implementing this suggestion. It could be an alternative later on if the current system would not be effective (but then that problem should have emerged during the evaluations).
4 News items
4.1 Issues
• Some users thought that clicking the headline of a story would open the comments section rather then open the original article. I’m unsure what can be done about this. The users had problems finding how to go to the comments too, the icon at the top left corner isn’t visible or good enough. Need to think some more about this!
• There could be a bit of a learning curve for the sharing icons (e-mail, facebook, twitter), especially the e-mail one (that one isn’t very straight forward it seems.
• Maybe the description for the votes (ie. “154 fans” and “like it”) could be better. Only one test person brought this up, I don’t think its bad now but I should continue to look for something better (tried several things with the paper prototypes already).
• Not possible to go to source website when clicking on it. Don’t know if this is something I want to enable.
• Categories, submitter and source should be more visible.
• The popup that opens when sharing a story on twitter is a bit weird in dimensions. Should be fixed.
• Share using e-mail opens the users default e-mail program, maybe a form is better here.
4.2 Suggestions
• Show number of comments somewhere, and perhaps number of friend comments. I thinks this improves the social factor. I want to include this.
• make it possible to exclude certain sources from the news stream (so when a user doesn’t like a certain site, their stories won’t show up in his/her stream). This is a nice to have, not a priority.
• Show which friends liked a certain story. Like the first suggestion, it adds some more social behavior so I want to include this.
5 Discussion
5.1 Issues
• Nothing happens when a non-logged in user tries to moderate a comment (should ask the user to login or sign up).
• Moderating your own comments is possible and shouldn’t be.
• Long user names aren’t being displayed properly here:
• User names don’t link to the profiles, should be fixed.
5.2 Suggestions
• Highlight friend comments. I like this one.
• Display the best comments (as decided on by the users). Could be nice, not really a priority.
6 Profile
Profile (feed)
Profile (info)
6.1 Issues
• Feed/info links are not clear to several of the test persons. Definitely need to change the way to change between these two parts of a profile page.
• The link to enable edit mode on your profile is hard to find (arrow on 2nd screenshot)
• Need to put up a default picture when the user hasn’t submitted his/her picture yet.
• The form for editing profile information needs to improve. For example, the favourite clubs field should have autocomplete based on the tags related to clubs.
• Date of birth not showing.
• Change “Recent stories” into “my recent stories” or “Wouter’s recent stories”, to make clear that its about the user which profile is showing.
6.2 Suggestions
-
7 Update feeds
7.1 Issues
• Voting on stories doesn’t work from the feed pages (bug)
• It somehow says “wouter submitted” when another user submitted a story.
• Distinction between news pages and the friends/profile feed possibly isn’t clear enough for some users. Didn’t seem to be too much of a problem during the evaluations though.
• Blueness of the page
7.2 Suggestions
• There should be a way to easily see which items are new since last visit. Possibly by placing a dot next to them or so. Would be a good idea I believe.
8 News submission
8.1 Issues
• Adding a story should be a lot easier. Adding content to sites often is already a barrier high enough for a lot of users. Title should be fetched from the site after pasting url, maybe even tags and summary. There are several more ways to approach this to make it a lot easier (bookmarklet, buttons for websites to use) which I should implement.
• http:// is required, it shouldn’t be.
• Form should display how many characters are required for the fields.
• Certain tags are in the autocomplete twice.
8.2 Suggestions
• Preview of how the news item looks after submission. Think this is a very good idea, will implement it.
9 Fuboh?
I asked the users what they thought about the name. No one thought it was great, only one user thought it was outright bad. The rest was somewhere in the middle. Its hard to find a good name for something like this, especially a fairly short one. The obvious requirement for the name is its domain availability. Ideas are very welcome! If I can’t find a better alternative, I’ll stick to this one.
10 Are the test users interested in using the site?
This was one of the questions I asked. The responses were consistent: It depends on the content, the community and whether friends use it. Most users do not seem eager to submit news themselves, especially not if its as much of a chore as it is right now. It is extremely important to optimize this process.
11 Lesson learned from evaluation procedure
My mouse, which test persons used to do the evaluation, is a bit defective (I don’t notice it anymore, its a small problem). I don’t think this interfered with the evaluation but its definitely something to avoid.
There probably were too many additional questions. People participating in the experiment weren’t expected to fill in all of them, but it still seemed too much work (for some users I wrote the answers down, others filled it in themselves).
12 Conclusion
These evaluations have provided me with a lot of information to move forward with. I was happy to learn that the main concept of the site is good and the main components (categories, …) are too. There are some things that need to be improved a lot (news submission), there are quite some bugs to iron out, there definitely is room for improvement in the design (usability issues). There are some other things beside the improvements listed in this post that need to be done before I can make the site available publicly. I need to implement e-mail (for activating accounts for example), I need to make the site cross browser compatible (currently only webkit-compatible, no major issues with other browsers though), improve the sorting algorithm and I need to create the required tags in the system (using an API).
There is one thing I should definitely look out for, that is the simplicity and focus of the site. I think the site should be more simple then it currently is (even though there were no problems in that area reported by users). Implementing new features can be a threat to to this simplicity. I will have to be vigorous about it.
Filed under: Uncategorized | 1 Comment
Earlier this (academic) year I looked at the existing behaviour of football fans on the internet. After that I created a prototype in several iterations (evaluating, adapting, evaluating, …).
I’m currently working on a good method for evaluating the user experience of my initial implementation (implementation of the first version is more or less done). The basis for this method will be the way my group evaluated the Facebook app we made for last years user interfaces course (think-aloud protocol). I also used this method for evaluating the paper prototypes.
I’ve been looking back at how I evaluated my prototype. While I tried to do it thoroughly (several iterations), I think there is room for improvement. There are some lessons to be learned. Each iteration about 8 persons evaluated my prototype. Several of them had no affinity towards football and I now believe this isn’t ideal. When it comes to usability, those evaluations were very useful. However, when it comes to the broader concept of user experience, I think its much better to focus on persons that are interested in football. I don’t think I can ask non-football fans to pretend to be one (it doesn’t work). A system can be very good from a usability point of view, but still have a poor user experience. Also the opposite is possible. There is research that indicates Facebook isn’t very good from a classic usability point of view (using the classic heuristics) even though Facebook users do like the user experience it provides.
The user experience I’m trying to create is (obviously) aimed at football fans. I believe I should evaluate it with football fans to get the best idea of how the user experience is working out (without losing sight of usability).
I think it is important to determine some goals for the evaluation. Of course my intention is to evaluate the initial implementation completely but I believe extra focus should be put on certain parts to get more information about them because they are vital to the design. Next to the think-aloud procedure, I will ask the test persons questions about certain aspects of the implementation. I did this for the prototype as well and I believe this was a good way of getting specific additional information on things the test persons otherwise maybe didn’t even say anything about.
There are two main components to the design. There is the news side by which football fans can keep themselves up to date about their favourite clubs, players, competitions, national teams,… in football. This content will be provided by users themselves (links to news sites etc.).
The other component is the social networking component specific to football. This involves ways to share their favourite football teams, players, their football-related activities, their activity on the news component (submitted stories, voted on stories, comments on stories).
I want to verify as good as possible if the combination of these components is something football fans are interested in. In the prototype evaluation this was certainly the case but a significant part of my evaluation group wasn’t interested in football so more evaluation of the main concept should be done.
Next to evaluating these two components in general, I will evaluate them in detail. If the design choice of going for these two main components still proves to be the correct one, It is important to look at the way they are designed. Is the approach for the news system right? Is the procedure for submitting news good? Do people want to submit news like this? Is the method for sorting the news good? Does the site offer something better in terms of news compared to alternatives? What should be put on the profiles? What activity of their friends interests them? (and a lot more)
I believe that a thorough evaluation using the think-aloud protocol will get users to give a lot of very useful information but in addition to this I also want to ask some questions about crucial parts of the application. Maybe a user won’t say anything about the profile content itself, and is a question required to get more information about this.
My approach now will be to further determine which parts of the components need special attention to get more information about what users want (by composing a list of questions and goals). Then I’ll do one or two evaluations to see if my approach itself is working, after which I start evaluating the initial implementation. The deadline for this is March 1st.
To summarise: I think it’s important to do these evaluations very consciously. I need to have a good plan to make these evaluations as useful as possible. I need to determine what works and what doesn’t and in order to do this I need to get the right kind of information. Based on the information gathered during this first round of evaluations, I will need to make some decisions about how to move forward.
Filed under: Uncategorized | Leave a Comment
Post-exam period update
Over the last couple of weeks I’ve managed to put a good amount of work into my thesis. I wrote this post to give an update on my activities. From now on, I will post more often (and shorter).
My last update was the presentation at the end of last semester. At that point I was far advanced in the design and I had evaluated it several times. Since then I’ve only made minor changes to the design. Something else I’ve been doing is reading some articles on metrics in order to get a better understanding of this field and its most important concepts. After my presentation in December Prof. Berendt suggested she could point me into the right direction for materials on metrics and the references she provided were very useful.
Two weeks ago I started learning Ruby on Rails. I wanted to develop in a language/framework that is well suited for agile programming. Two frameworks that are usually linked to this quality are Ruby on Rails and Django (for Python). I’ve compared the two and didn’t find a clear reason to choose one over the other. I had limited experience in Ruby on Rails already and this is why I decided on RoR. A decision I haven’t regretted so far. Ruby on Rails provides a lot of useful things to the developer, it feels “right” so far.
After an extensive tutorial (the book Agile Web Development On Rails) I started implementing my design. It is hard to say how far into development of the first version I currently am but if I had to make a guess it would be about 30-40%. There is a good base to work from and I’ve implemented some of the most important components already. I’m very satisfied with the work I’ve been able to do considering a lot of time was spent on exams (obviously). I want to get the first version done as soon as possible. After that I want to evaluate it thoroughly.
Filed under: Uncategorized | Leave a Comment
Second thesis presentation
The slides of todays presentation:
Filed under: Uncategorized | Leave a Comment
) and I’ll do my best in creating something good.
.Filed under: Uncategorized | Leave a Comment
To get some level of understanding on the behavior of football fans on the Web I came up with a short on-line survey. I spread it through the forums of 4 Belgian Jupiler League teams (RSC Anderlecht, KRC Genk, SV Zulte Waregem, Cercle Brugge) and one (unofficial) Liverpool fan forum. This proved an easy way to get a good amount of participants and the visitors of such forums could well be the people my project should target. I also asked some friends of mine with an interest in football to fill it in. 192 people participated.
Age of the participants
No surprises here I believe. Over 60% of the participants are under 30 years old.
Even though the survey was done by visitors of football forums these results are surprising to me. About 30% of the participants indicated they spend 2 hours or more on average on visiting football-related websites a day.
Over 75% of the participants indicate they regularly visit 2-5 football-related websites. There are only a couple of participants who regularly visit over 10 football-related websites. Also, the chart resembles a blue pac man .
I included this question cause it seemed relevant to me not only how much time people spend in general but also how often the participants browse football-related sites a day. Over half of the participants indicate they visit football sites 6 times a day or less.
For each activity, the participants selected from 6 choices (from “not relevant” with weight 0 to “very important” with weight 5). The chart displays the average values. “News” comes out on top and “live scores”, “live game footage” and “to discuss football” aren’t far behind.
There are a couple of very good sites and mobile applications for live scores (livescore.com for example). These results make me wonder about live game footage. As far as I know there currently aren’t many football matches you can follow on-line. Some media companies offer the option to people who already subscribe to their sports channel (for example Sky in the UK). I’m afraid most of the people who indicate they watch games on-line watch illegal streams. I think this is another example of a slow response to changing demands, similar to what happened with the music industry and music downloads.
The survey also included an open ended question “Which football-related sites do you visit?”. The answers on these questions showed, like these results, that discussion is an important activity. A lot of users mentioned discussion forums in their answers.
“Statistics (league tables, goal-scorers,…)”, “interviews” and “video’s” are considered quite important (averages just below 3) as well. I assume a lot of football fans watch match reports and post game interviews on-line.
This is another view on the same results. Here you can see the different answers in perspective instead of the averages.
I listed the most prominent football social networks as answers to this question. I think these results show that the existing football social network sites are not popular. 37 out of 192 participants indicate they know FootyTube.com. In my post on existing football SNS I indicated that I thought FootyTube is one of the best football SNS that exist. Maybe this is because its most important feature is video’s. The results of the previous question indicate that watching video’s is regarded as quite an important activity.
My intention behind asking this question was to see whether other football fans had the same feeling as me; that it takes quite some effort to get the information on football that interests you. The answers are interesting yet not very conclusive. I think this was to be expected. Nonetheless, I believe it is interesting that 70 out of 192 participants agree (or “completely agree”) with this statement.
Nothing surprising here, most of the participants are already using a forum to discuss football.
Not much to conclude here in my opinion. I guess it depends on what the football SNS is going to be like.
Conclusion
I believe conducting this survey was useful. It provided me with some indications of how fans use the internet for their interest in football. However, it is important to keep the way in which this survey was conducted in mind (on-line, spread through fan forums).
Filed under: Uncategorized | Leave a Comment
Presentation november 10
Here are the “slides” for the presentation I did on tuesday. I wanted to embed them but didn’t find a way that worked.
Filed under: Uncategorized | Leave a Comment
Social Media = Business
Video: Social Media = Business
Last month I saw Gary Vaynerchuck speak at FOWA London. For the people who haven’t heard of him, you can find his biography on his website. I believe he has some interesting views. In this video he expresses his opinion about whether social media are a genuine business. He says a lot of people think it is not a business, but something extra to add on top of your business. Gary emphasizes on the fact that the Internet is changing things like company branding and media significantly.
How is this relevant to my thesis subject? Among many things Gary talks about how you have to listen to your customers/users and not the other way around. I think this is a crucial element in developing a social network site (which effectively is a form of social media). You need to listen to your users in order to build something that people want to use. He stresses that it is important to care about your work, and that you can be successful if you are passionate about something and if you are willing to work hard on it. I don’t think anyone will disagree that you will be much better at what you do if you care about it. He also talks about how you will find an audience in your niche if you know what you’re talking about and provide quality content.
I just picked this video but he has several more interesting ones on his website (garyvaynerchuk.com). I just wanted to point out that I believe there is a lot of truth in what he says. It is important to keep things like this in mind when developing a social network site. Gary has a way with words and sometimes his choice of words can be a bit explicit but I think it doesn’t take anything away from the message. Of course in some ways what he says is a bit abstract and vague but he is the living example that what he stands for works. He’s successful in the wine business, he has recently published a book, he’s a popular speaker at events,…
To summarize my objective this year in his words: I have to “crush it” for my thesis on social network technology for football.
Here is a another video by him on his book tour in which he discusses his views about changes resulting from the Internet. It is part of his longer talk on fora.tv.
Filed under: Uncategorized | Leave a Comment
SNSs usually offer the same basic functionality: network of friends listings (showing a person’s “inner circle”), person surfing, private messaging, discussion forums or communities, events management, blogging, commenting (sometimes as endorsements on people’s profiles), and media uploading.
Footytube
3nil
Goalpostr
Extrafootie
Yourleague
Footbo
Ole ole
Conclusion
Filed under: Uncategorized | Leave a Comment
Search
-
Blogroll
Recent Entries
- Slides of mondays presentation
- First version of football SNS: evaluation
- Evaluating my initial implementation
- Post-exam period update
- Second thesis presentation
- In reaction to “Taking a look at existing football social network websites”
- Football on the Web: a survey of football fans
- Presentation november 10
- Social Media = Business
- Taking a look at existing football social network websites
- Exploring the Facebook Experience: A New Approach to Usability
Categories
- Uncategorized (14)
