5 Reasons to Evaluate Your Product Experience Using a Longitudinal Research Method
At Aritzia we’ve recently started conducting longitudinal usability studies to observe how our customers use our mobile website over time. Aritzia has been building an eCommerce presence since 2012, and as our customers’ device usage turns more mobile, we’re aware that our user research methods will need to capture their behavior and attitudes wherever and in whatever context they’re shopping. We’ve been using Lookback to run week long studies where our participants are asked to use the website at least once a day, for at least 10 minutes, for 5 days. At the end of each day they send me a text with their impressions of the day (diary study style). We then wrap up each week with a post evaluation questionnaire. We’ve found that these studies yield completely different insights than any other user research method we employ, and are excited to share the value we’ve found in this type of study through the lens of eCommerce.
Here are five reasons to foray into the method of longitudinal user research:
1. You have yet to identify your customer’s key flows. Learning about your customer’s key flows, means you’re then able to track them down through analytics. When synthesizing longitudinal studies I like to transcribe each session video by page and then mark recurring sets of three. For example PLP (grid of products), PDP (product page), back to PLP (grid of products). In a recent study I pulled out 6 primary flows and 3 secondary flows for us to then measure quantitatively through analytics. Knowing your customer’s most common flows allows your team to better design how they move forwards and backwards through those flows, and understand the most common scenarios as the frequency of these flows changes with the implementation of new features.
2. Pest control. Bugs crawl to the surface most when a digital product is being used in a real life context, and reporting bugs is a lot more common in an app then on a website. Not only that, but certain demographics will be naturally less inclined to help you out by reporting issues.
3. You’re hosting a bag, wishlist or collection space. There are an infinite number of engagement metrics you can tie to features like the bag and wishlist, but which are the most important? By watching our customers shop on Aritzia.com over the course of the week we learned that our customers were adding products to their wishlist via their bag due to the visibility (or lack thereof) of our wishlist CTA on products. We watched as users hit the sign in wall of our wishlist, and immediately turned their bag into a higher stakes evaluation space. We were able to hypothesize that US customers used these features less than CA customers, and set up an analytics queries to confirm. We also noticed that the more time a user spent on our site, the more they used the wishlist. All of this has helped us clearly define the scenarios when the wishlist is used, and map ways that we could remove the wishlist barriers to increase engagement metrics on Aritzia.com.
4. You want to monitor your site during the release of new products, collections, or seasons. Imagine watching the faces of your customers the day your campaign is launched. In February we ran a longitudinal evaluation during a week where we had three spring launches released. Watching how our customers managed their shopping during this time inspired the exploration of new metrics to measure the success of future seasonal blasts. For example comparing product add to bag rate by campaign, against sales dollars per campaign. Though your sales numbers may be your key indicator of success, watching users interact with your content in real time allows you to identify success and failure in new ways and react immediately.
5. Your team needs to get real. Seeing and hearing the people you’re building things for is like flicking a switch in a dark room. Watching where they use your site, learning why they use your site, and observing the friction that they experience on your site is an intimate experience. At the end of the day empathy is inspiration for design, and the most brilliant and meaningful solutions come out of human need.
For our team at Aritzia, these longitudinal studies are exploratory in nature, and live within the context of evaluative research for the purpose of problem discovery. We’re not running a longitudinal study for the purpose of medical science, and these aren’t six month long studies where we strive for repeatable results. For example we don’t need to observe twenty participants experience the same awkward friction point in order to acknowledge that it’s a problem. The magnitude of any problem identified can be determined by following up with an analytic inquiry. As people who work in agile environments, we exist in a world that’s sprint driven, iterative, and motivated by getting value out the door. These first few longitudinal studies we’ve conducted at Aritzia have lasted a week in duration, with less than 10 participants per study. We’ve aimed to keep it lean, iterate on this methodology as we go, and not let fear of imperfection hold us back from deriving new insight.
