Customers – DANFORTH

Soliciting Quantitative Feedback

“A successful person is one who can lay a firm foundation
with the bricks that others throw at him or her.” – David Brinkley

User surveys and feedback forms are arguably the easiest and most inexpensive methods of gathering information about your users and their preferences. Like competitive analysis, surveys are also used in a marketing context. However, UXD surveys contain more targeted questioning about how usable a website or software is, and the relative ease with which people can access content. There are a few different types of user surveys. Some are different questioning methods; others approach the user at a different point in time.

Feedback forms – Feedback forms are the most common way to elicit input from users on websites, but they can be implemented for desktop applications. A feedback form is simply a request for input on an existing application or set of functionality. A feedback form can be a single, passively introduced questionnaire that is globally available on a website for users to fill out. They can also be short one- or two-question mini-forms placed throughout an application in relevant places such as a help/support topic, or when an application unexpectedly terminates.

Tips for Feedback:

Keep it simple. A shorter, simpler survey will be completed more frequently.
Always offer a comment field or other means for users to type free-form feedback. Comments should be monitored and categorized regularly.
If your marketing team has already established a feedback form, try to piggy back off of their efforts. If not, consider collaboration. Much of the information collected in a feedback mechanism will be useful to both groups.

Surveys – A survey is not all that different from a feedback form, except that they are usually offered for a limited amount of time, involved some sort of targeted recruitment effort, and the content of the questions are not necessarily related to an existing site or application. Surveys can be implemented in a few ways, depending on the goals of your research.

Intercept Surveys – Intercept surveys are commonly used on web sites. An intercept survey is a survey that attempts to engage users at a particular point in a workflow, such as while viewing a certain type of content. Intercept surveys usually take the form of a pop-up window or overlay message. Because these are offered at a specific point in a process (e.g. the jewelry section of an e-commerce department store.), the results are highly targeted.

Online Surveys – Online surveys can be conducted to learn about a website, application, kiosk, or any other type of system. However, the survey itself is conducted online. There are many advantages over traditional phone or in-person surveys, not the least of which the ease of implementation. There are a number of online survey tools available that will make it easy for you to implement your survey in minutes. Most offer real-time data aggregation and analysis.

Traditional Surveys – There is definitely still a place for surveys conducted in person or on the phone. While a bit more costly and time consuming these methods can reach users who would not otherwise be reachable online or those who normally will not take the time to fill out a survey on their own.

Why Conduct User Surveys?

User Surveys are an effective, economical method of gathering quantitative input.
You can poll a very large number of users when compared to the number of users who participate in user testing or other research methods.
Since surveys (online versions in particular), are usually anonymous, you are more likely to get more honest and forthright responses than from other forms of research.
Surveys can be implemented quickly, allowing you to test ideas in an iterative process.

When is a Survey Most Useful?

When you have specific targeted questions that you would like answered by a statistically relevant number of people. Polling attitudes, etc.
When there is a specific problem that you want to investigate.
When you want to show the different between internal attitudes and perceptions of an issues verses the attitudes and perceptions of your target market
When you want direct feedback on a live application or web site.
When you want to use ongoing feedback to monitor trends over time, and gauge changes and unexpected consequences.

Development Life-cycle

Surveys are useful in the requirements gathering stage as well as follow-up feedback elicitation during the maintenance stage, i.e. post release.

Limitations of User Surveys

User surveys are entirely scripted. So while skip logic[1] and other mechanisms can offer a degree of sophistication, there is little interactive or exploratory questioning.
Users usually provide a limited amount of information in any open ended questions.
Because there is little room to explain or clarify your questions, surveys should be limited to gathering information about tangible, somewhat simplified concepts.
In the case of passive feedback forms, most people won’t think to offer positive feedback. In addition, not everyone with a complaint will take the time to let you know.

How to Conduct a User Survey

Define your research plan. As in any research effort, the best starting point will be to define and document a plan of action. A documented plan will be useful even if it’s just a bullet list covering a basic outline of what you will be doing. Some considerations for your user survey research plan are:

What are the goals of your survey? What are you hoping to learn from participants? Are you looking for feedback on what currently functionality, to explore concepts and preferences, or to troubleshoot an issue?
Who will you survey? Are you polling your existing users? Do you need to hear from businesses, teens, cell phone users, or some other targeted population?
How many responses will you target? While even a small number of responses are arguably more valuable than no input at all, you will want to try for a statistically relevant sample size. There are formulas that will tell you how many survey participants are needed for the responses to be representative of the population you are polling. For most common UXD uses, however, 400-600 participants should be sufficient.[2]

Select a survey tool. There are a number of online survey tools to choose from so you’ll need to select an application that fits your goals and budget. Even if you will be conducting the surveys in person or on the phone, consider using survey software as they offer substantial time savings in processing and analyzing the results. Once you have selected the survey software, take some time to use the system and get to know what options are available. A basic understanding of your survey tool can help reduce rework as you define your questions.
Develop your survey. This is usually the most time consuming part of the process. While seemingly straightforward, it’s quite easy to phrase questions in a way that will bias responses, or that are not entirely clear to participants—resulting in bad or inaccurate data.

Define your Questions. Consider the type of feedback you would like to get and start outlining your topics. Do you want to learn how difficult a certain process is? Do you need to find out if people have used or are even aware of certain functionality? Once you have a basic outline, decide on how to best ask each of your questions to get useful results. Some possible question types include:

Scale – Users are asked to rate something by degree i.e. 1 to 10, Level of Agreement. Be sure to include a range of options and a “not applicable” where relevant.
Multiple Choice – Users are asked a question and given a number of pre-set responses to choose from. Carefully review multiple choice answers for any potential bias.
Priority – Users are asked to prioritize the factors that are important to them, sometimes relative to other factors. A great exercise to help users indicate priority is the “Divide the Dollar” where users are asked to split up a limited amount of money by dividing it among a set of features, attributes etc.
Open Ended – Users are asked a question that requires a descriptive response and are allowed to provide an open response, usually via text entry. Opened ended questions should be used sparingly in this type of quantitative study; it’s difficult to get consistent responses and they are difficult to track and analyze.

Edit & Simplify. Your goal is to write questions that the majority of people will interpret in the same way. Consider both phrasing and vocabulary. Try to write in clear, concise statements, using plain English and common terms. Your survey should be relatively short and the questions should not be too difficult to answer. In addition, be sure to check and double check grammar and spelling. I once had an entire survey question get thrown off due to one missing letter. We had to isolate the data and re-publish the survey.
Review for Bias. Review the entire survey, including instructions or introductory text. Consider the sequence of your questions; are there any questions that might influence future responses? Take care not to provide too much information about the purpose of the survey; it might sway user’s responses. Keep instructions clear and to the point. As with all research, you need to be ever vigilant in identifying areas of potential bias.

Would you like a piece of candy?
Yes (95%)            No (5%)

Would you like a piece of caramel?
Yes (70%)            No (30%)

The Surgeon General recently came out with a study that shows candy consumption as the most common cause of early tooth loss. Would you like a piece of candy?
Yes (15%)            No (85%)

Thinking Ahead
Recruit for future studies. At the very end of your survey, ask users if they’d like to participate in future studies. If they agree, collect their contact details and some additional demographic information. This will help you build up your own database of participants for more targeted future testing. Many users will want to be assured that their information will not be connected to their survey responses, and that you will not resell their data.

Implement & Internally Test. Implement the survey using the survey tool you selected. Next, test the survey internally, this can mean you and a few colleagues, or a companywide involvement. Testing the survey internally will help ensure your questions are clear, without error, and that you are likely to get the type feedback you want.

Our Users, Ourselves

Data from an internal test can be additionally helpful. Getting feedback from people in your company or department will likely produce biased results. While it’s important to isolate that data before conducting the actual survey, you can use feedback from your internal group to illustrate any differences (or notable similarities) in their responses from those of your users. E.g. 90% of our employees use Twitter, while only 35% of our users do.

Recruit Respondents. Your research plan should have outlined any appropriate demographics for your survey. But now that you are ready to make your survey live, how will you reach them? If you have a high traffic web site or large email list, you can use these to recruit participants. If not, you may need to get creative. Consider recruiting participants through social networking sites, community boards or other venues frequented by your target population.
Prepare the Analysis. – One of the nice things about the online survey tools is that the analysis is usually a breeze. You may, however, want to put together an overview of some of the insights and interpretations you gathered from the data, e.g. “Since a relatively small number of our customers tend to use social networking sites for business referrals, we may want to lower the priority of getting Twitter integrated onto our website in favor of other planned functionality.” As with all UXD research, present your findings and publish any results.

Additional Resources

Sample Size Calculator (www.surveysystem.com/sscalc.htm) – a sample size calculator that will determine the sample size required for a desired confidence interval.
SurveyMonkey (www.surveymonkey.com) – offers a popular online hosted survey tool that works well for basic surveys.
SurveyGizmo (www.surveygizmo.com) – is comparable to SurveyMonkey, but offers a somewhat less robust reporting at a slightly lower fee.
Ethnio (www.ethnio.com) – is primarily a recruitment tool, but offers some basic survey functionality. Ethnio can be used to drive users from your website to another third party survey.

[1] Skip logic refers to the ability to skip a question or set of questions in a survey based on the users response to a preceding question.

[2] For a survey of the general population where the data has a 95% degree of confidence and 4-5% margin of error.

How to Develop Basic User Personas

“Before I can walk in another person’s shoes,
I must first remove my own.” – Brian Tracy

This article discusses the basics for developing of a set of user personas. User personas are not a research method, but a communication tool. They are a visual, creative way to convey the results of research about the users of your software or website. Personas are fictional characters developed to represent aggregate statistical averages to profile a user group. They do not express all types of user or their concerns, but can give a reasonable representation of who might be using your product and what some of their goals or issues might be. Personas are not fixed precise definitions of your users; they are more of an empathy creating tool. They can help you get into the mindset of a user type and communicate that mindset to others.

One created, personas are a way of humanizing your users and providing a canvas on which you can superimpose ideas on how a given user type might interact with your system. A little bit like criminal profiling (minus the crime and vilification), you can think of it as role playing with the intention of gaining new insights into the other party. User Personas should not be confused with engineering use cases or marketing demographic profiling.

Why develop user personas?

A key benefit to developing user personas is to provide user-centric objectivity during product design, and reinforce the idea that you and your colleagues’ version of a great user experience is not necessarily the same as your end users.
User Personas can dispel common stereotypes about users, e.g., the site needs to be so easy to use the VP’s grandmother can use it.
Later in the design process, user personas can help you communicate the usefulness of a specific design to stakeholders.

While I used to be somewhat limited in using persona, I did create them a few years ago for a top-five business school. The project was in the context of a site redesign that included an overhaul of the site’s information architecture. I conducted a fairly exhaustive audit of the MBA space which included a competitive review, expert review, and stakeholder interviews. At the time, I didn’t create personas for myself, I just used the underlying audit data to restructure the site’s content and define a simplified navigational model. However, later, when preparing to present the new information architecture to the client, I took the time to create a set of user personas based on the original data I used to make my design decisions. They were created out of a need to help illustrate to the client how the new structure better addressed the needs of their prospective students, and to assure them that user’s concerns were being addressed. After realizing how effective persona can be for stakeholder buy-in and alignment I have since used them much more frequently in a range of UX projects.

When are personas most useful?

User personas are useful when there are a number of unsubstantiated assumptions floating around about who your users are. E.g. early adopters, soccer moms, the VP’s grandma, etc.
When there is a tendency for developers and product managers to make design decisions based on; their own personal preferences, technical constraints, chasing short term sales, or any other reasoning that does not consider the end user.
When there is a general lack of clarity about who your end user is, or when the data you have regarding your users does not seem to directly translate into insights about their behavior.
Whenever you need a communication tool to advocate and encourage empathy for end users.

Development Lifecycle

Persona creation is best done during requirements gathering stage. However, getting the most out of your user personas means that they should also be referenced during development and testing, as questions and changes arise. They should also be compared to, and updated against actual user testing and post-release feedback. If properly evolved and maintained, personas can be an effective guidepost throughout the development and product lifecycle.

Limitations of User Personas

It’s easy for personas to be taken too literally or become a stereotype. Personas should never be considered a definitive archetype of your users, doing so runs the risk of turning your definitions into pseudoscience, akin to phrenology. Direct user testing and feedback insights should take precedence over anything user persona derived.
User personas are not precise; they are limited by the innate prejudices, interpretations, and reference points of the authors of the personas and any subsequent user scenarios.
User personas are not an appropriate communication tool for everyone; some people prefer a well crafted presentation of the source data over a persona which can be perceived as containing superfluous data.

How to implement user personas

With a little research, a set of basic personas is actually fairly easy to put together. The first step is to pull research from multiple sources to gather data about potential users. Next, synthesize this data to form user groups and select representative data from each user group to form a user data profile. Finally, add supplemental character information to create a believable persona. As a general rule, you should start with trying to define three to five user types, adding additional types as appropriate. I would recommend against defining more than about eight or 10. Beyond this number, you start to exceed people’s ability to reasonably keep track of the information.

Consider This…

If you think you need more than ten personas, or have already been heavily relying on personas to drive development consider as a next step investing in an ethnographic study. Real profiles and insights derived from the results of an ethnographic study will provide more actuality than fictitious personas.

Conduct Research. Pull together any reliable data you can find might describe your users or give insight into their goals, concerns or behaviors. Collect any potentially relevant demographics such as; gender, age, race, occupation, household income. Let’s, for example, take the case of developing personas around prospective MBA students. In addition to client interviews I was able to pull a large amount of key data from sources on the Internet. Some references I found were:

Graduate Management Admissions Council
GMAC: 2004 mba.com Registrants Survey
The Black Collegian: ‘The MBA: An Opportunity For Change’
The Princeton Review: ‘Study of Women MBAs’
National Center for Education Statistics (NCES)

Form Relevant User Groups. To continue with the business school case study, research on MBA students showed that there were a growing number of minority women enrolling in MBA schools. In addition, the client indicated a desire to further attract this type of student. Armed with that initial seed data, I defined a user group of minority women who were considering, had applied to, or attended business school. Further research allowed me to compile a list of statistical information about this group (such as average age, current occupation, family status, etc) as well as formulate some ideas about their interests, concerns and expectations. When compared to more “traditional” MBA students, these women were; a few years older on average, more likely to be married with children, and were highly-focused on what specific opportunities this type of degree offered post graduation.
Create and Embellish the Persona. Start by turning the data you have for the user group into a single individual. The “Minority Women” user group was converted into a thirty six year old, African American woman named Christine. At this point it’s ok and actually preferable to fill in the gaps with plausible details about this person to bring them to life a bit; e.g. “Christine Barnhart is a marketing communications manager who balances her time between work and family…” Since this is in large part a tool to facilitate empathy for the user group, as much realistic information as possible should be incorporated to develop the character. As long as the “filler” is not negated by other data it should be fine. Even still, after you have finished writing your personae take a step back and ask yourself; “Is it reasonable to expect this person to exist?” Alter your description if necessary, or move on to the next persona.
Develop and Test User Scenarios. Now that you have your personas and an idea of what your user types are, we can make some assumptions about what functionality or information they might find useful. You will need to develop some User Scenarios to help test any hypothesis you have. Unlike a use case, which is a description of how system functionality interacts (i.e. contact data is retrieved, error message is returned), user scenarios focus on the user’s perspective. A simplified User Scenario for our “Christine” persona might look like:

While successful in her career, Christine is not entirely sure she has the background and experience to be admitted to a top business school. In addition, she has a lot of specific questions about the program’s culture. She is concerned that it may be too aggressively competitive for her to get the most out of her business school experience.Christine goes to the program website and immediate looks at the eligibility and admission requirements. She takes her time looking through the information and prints out a few pages. Next, she starts to browse around through site, trying to get a feel for the culture. She finds a section that has video interviews with students and so she watches a few and begins to get a sense of whether this school is a fit for her or not.

User Stories Not to be interchanged with a user scenario (or use case), a user story is a short two or three sentence statement of a customer requirement. It is customer-centered way of eliciting and processing system requirements within Agile development methodologies[1].

Basic Persona Template

There are a wide range of user persona templates, and they can get rather complicated. Considering that the intention of this type of tool is effective communication, I tend to gravitate toward the clean, simple layouts that are one page or screen. Below is a basic example template. Don’t get too focused on templates however, if you find information that you think is useful or relevant but it doesn’t fit your template, rework the template to fit the data.

[1] More information on user stories can be found at: http://www.extremeprogramming.org/rules/userstories.html