Online workflow mental models
in-home testing, technology, multi-country, survey
Project Lead, Moderator, report writer
Bold insight
CHALLENGES
A global technology company wanted to better understand search engine user mental models of web and native apps in order to gain directional insight on potential UI changes aimed at solving user problems. This company believed - due to prior foundational research - that there are particular benefits to using native apps that are worth preserving but are unsure if investing in these benefits is worth additional time and effort.
"How might we use existing mental models of native apps to identify functions and features meaningful to users that we can implement in the web app experience?"
Process
We collaborated with our client stakeholders to identify a multi-method process that would provide small-sample qualitative insights backed by large-sample quantitative data.
​
-
Step 1 - Discovery and literature review: The client provided our team documents from prior research to review. This helped provided our team a foundational understanding of current client expectations and areas of uncertainty. We learned that there were existing assumptions about how younger users perceived web apps and native apps becaue of their exposure to touch screens and technology earlier in life. Our team also discovered that the power and functionality of web apps was comparable to their native app counterpart.
-
Step 2 - In-home contextual interview: After literature review, our team created discussion guides aimed at answering the research questions. Due to client budget constraints, we opted for no more than n=8 in-home sessions in both the US and the UK. Our screener helped us recruit knowledge workers who were regular users of creativity or productivity apps. We also wanted to talk to a mix of users whose first device was a touch screen or non-touch screen device. We conducted pre-interview introduction meetings to allow participants to get to know the researchers from our team who would be coming into their home, set expectations for the study beforehand, and allow the participants to ask us any questions. During in-home interviews, our two-person team used a simple technology setup of a few microphones, a GoPro, and laptop to capture backup recordings. The 90-minute sessions were centered around understanding impressions and assumptions around working in a browser or working in a standalone native app. After this, we presented a series of hypothetical scenarios meant to understand the limitations of their mental models and when they would want to work in a web versus native app.
-
​Step 3 - Survey creation and launch: Our team gathered preliminary qualitative findings based on the in-home interviews and used this to design a survey for the large sample portion of our study. We worked with an external survey vendor to launch this survey to 200 knowledge workers across the US and UK, and to support the quantitative analysis of the survey data. After survey deployment, we worked closely with this vendor to ensure that the survey launch was successful, that the data was flowing in as anticipated, and that we aligned on expectations for survey data analysis and reporting.​
analysis
Both the US and UK teams created daily debrief documents with participant summaries of their in-home interviews, which helped to guide the survey creation.
​
Our survey vendor supported us in analyzing the survey data and provided us a comprehensive report outlining the quantitative findings.
​
Our team then created a comprehensive report that combined learnings from the initial literature review, participant interviews, and survey data.
outcomes
The findings from the study were successful in supporting decisions in upcoming feature design sprints and UI updates. The client was also able to identify development areas of high interest and risk, allowing them to effectively prioritize effort for their upcoming development roadmaps.