Case Study 2:
Evaluative

I knew that we needed a quick turnaround and already had an abundance of under-analyzed data. So, I decided to forgo talking with users for now and instead conduct an evaluative study that would leverage the quantitative and qualitative data that we already had: platform analytics and user session clips. I would also evaluate copy and design elements to identify quick wins. So, I created a research plan in Notion, asked my team for feedback, and got to work.

My approach to this study was to move from the broad to the specific. I would start by using quantitative methods to identify the most problematic webpages and steps in the workflow through funnel analysis as well as where on these “problem pages” users were having trouble through analyzing heatmaps and form field data. Finally, I would do qualitative work by watching user session clips to uncover specific details about design interaction and user behavior to understand exactly what problems were occurring and why. This mixed methods approach would create richer insights than using qualitative or quantitative methods alone.

So, I knew that the IDV process was likely causing drop off, but why were users dropping off? I decided to compare IDV conversion rates for the current and old flows.

Interestingly, users were equally likely to start IDV regardless of flow order. In the old flow, 38.2% of users on the feature landing page began the IDV process. This number “jumped” to 40.9% for the current flow, but this difference was not statistically significant. Imagine the misjudgment that would have occurred if someone saw those numbers and didn’t check for statistical significance!

My hypothesis was that the most demanding, intrusive parts of the onboarding process (identify verification and bank account linking) were too cumbersome and occurred too early in the onboarding process.

I knew that other things needed to happen first in the product adoption process and believed the key was to provide users with opportunities to trial the product so they could get to that “Aha moment”–that instant where the user realizes that the product solves their problem–before we ask them to complete sensitive, higher effort tasks.

I also conducted a “form data analysis,” where I looked at the rate of successful entry for all form fields in the IDV process. Since much of the IDV process happened through the Stripe API rather than on the RentSpree domain, we didn’t have analytics data or user session clips for this part of the flow.

Luckily, I was able to poke a hole in this “data blindspot” by accessing the limited analytics that Stripe provides to customers on their platform, analyzing the data that our users entered into form fields, revealing user challenges as well as opportunities to remove fields and improve copy.

Sometimes, my study will yield insights that apply to stakeholders across several teams and functions. In those cases, I may start by hosting what I call an “open sharing session,” where I invite a large and diverse group (in several cases, it’s been the whole company) with the goals of education and inspiration. To accomplish this, I explain the widely relevant insights, address questions, help participants identify potential actions to take, and try to secure commitments from individuals to take the ball and run with it.

Conversely, I’ve also facilitated an open sharing session that was exclusive to our team of 10 designers, where I went deeper into the particulars of user needs and led them in exploring how the insights may apply to the design process.

If insights apply to another researcher’s team, I always reach out to that researcher to discuss the relevant insights and make sure they’re ready to bring them to their team. I’ve been known to do guest appearances to support a fellow researcher and their team, explaining relevant insights from my research and how they might apply to their work.

I only needed to host a single sharing session for this study since insights primarily applied to my immediate team. However, for studies with broadly applicable insights, I tend to host multiple sharing sessions (my record is 5!).

I try to make research sessions inclusive by providing various ways for people to share their thoughts and questions. This includes providing 5-10 minutes of silent time for adding your ideas as sticky notes before we work together to thematically group ideas and discuss. I’m also a bit obsessed with ideation frameworks. I’ve facilitated teams in using a bunch of them: eg, SCAMPER, Amazon Working Backwards, How-Now-Wow, Crazy 8’s, Lean Coffee, Six Thinking Hats, etc.

Inclusive techniques can empower neurodivergent and other colleagues. At RentSpree, inclusivity proved especially valuable, as two-thirds of our employees were Thai and many were not native-level speakers. Because I had lived in Japan, I had lots of experience interacting with non-native level English speakers. Additionally, I understood what it was like for them because I had first-hand experience. Navigating complex group conversations in Japanese wasn’t easy for me! So, I knew how to modify my speech and structure sessions to facilitate a more inclusive environment.

As an additional measure, I always encourage colleagues to contact me anytime to discuss research on a one-on-one call or through Slack messaging.

A designer once made my month by messaging me to say that several of her Thai colleagues remarked on how thankful they were for how I tailored sessions to their needs. It was amazing!

Our team agreed to redesign the landlord onboarding experience and apply a number of my recommendations immediately. So, once we aligned with the Director of Product and product manager on the scope of the changes, the designer and I worked together to design the prototype. This would include moving the IDV process completely onto the RentSpree domain to give us complete control over design and remove the data blindspots that came with using the Stripe API for part of the IDV process. Once the designer and I were happy with the prototype, we shared it with the product and engineering managers for feedback.

This study was quite successful. Within just a few weeks, I had produced insights and recommendations that were immediately actionable. We quickly rallied around the study’s core conclusions, aligned on an action plan, created a prototype, ran an A/B test with excellent results, and compelled the marketing team to add a redesign of the landing page as a shared roadmap item. We had accomplished our goal of increasing conversion, improving usability and satisfaction in the process.

To make sure we continued in the right direction, I designed an in-product survey through Survicate to ask non-converting users why they didn't complete product setup. So, this was just the beginning.