Research operations in product development
TECH FLEET RESEARCH
Overview of My Research
Client: Morgan D
As founder of Tech Fleet, Morgan wanted to know what skills UX candidates need to be more marketable when looking for their first UX job. Our Team worked with Morgan to conduct User Interviews on UX Candidates and Hiring Managers.
Our Team
Our team consisted of a total of 7 UX Researchers and our Product Manager, where we coordinated to conduct Card Sorting activities and User Interviews, taking turns conducting the interviews and taking notes.
My role: UX Researcher
My main role was a UX Researcher, where I worked with the other researchers on our team to understand what UX candidates need in order to get their first role in UX.
RESEARCH STRATEGY:
Our project was aimed at performing a hybrid quantitative and qualitative study to gain insight of all target participants' attitudes, wants, and behaviors towards UX skills.
STAKEHOLDER INTERVIEW
Stakeholder- Morgan D.
UX Research Methods:
Card Sorting
A list of skills for UX Candidates and UX Hiring Managers and Recruiters.
-
Must Have Experience Skills
-
Must Be Familiar With Skills
-
Nice To Have Skills
-
Unnecessary Skills.
Survey Questions
-
Get Background information for analysis
-
Screen participants to recruit later for user interviews
User Interviews
UX Candidates
-
To understand their experience in their job search process.
UX Hiring Managers and Recruiters
-
To understand what they are looking for in candidates.
Survey Questions:
-
We created survey questions to get a better understanding of the experience and demographics of the UX candidates in the job search process. We also created questions to get background information on the UX hiring managers. Then we used the responses to reach out to the candidates and hiring managers that fit the demographics we wanted to talk to. We wanted a variety of UX candidates who just graduated and those switching careers. We also wanted a wide range of UX hiring managers from in-house and from recruiting agencies.
Card Sorting
I was assigned to analyze the UX Research job descriptions, while another team member (Marija) analyzed the UX Designer job descriptions.
-
Objective: Review job descriptions and develop a quantified database of frequently requested hard and soft skills
-
Criteria: 30 UX Design and 21 UX Research Junior/Mid-Level positions
During this process, we:
-
Employed a word frequency counter to analyze job descriptions.
-
Focused on specific UX-related skills and tools.
-
Collaborated with others to compare and contrast the refined lists of skills.
-
Determined the selection of skills to be used for the card sorting activity.
PARTICIPANTS:
-
Junior UX Candidates
-
No professional UX experience
-
Have (at least) started to look for UX job opportunities
-
-
UX Hiring Managers & Recruiters
-
Mix of inhouse and agency
-
Must participate in hiring process or
-
Have experience interviewing or hiring UX candidates
-
-
Junior UX Candidates:
-
39 UX Candidates
-
10 - UX Design
-
10 - UX Research
-
16 - Interested in both
-
3 - Unsure
-
-
-
UX Hiring Managers & Recruiters
-
9 In House
-
3 - UX Design
-
4 - UX Research
-
3 - Mix
-
-
1 Agency
-
1 - UX Design
-
-
LIMITATIONS:
SCREENERS AND CARDSORT
(cardsort below)
UX CANDIDATES
SCREENER/QUESTIONNAIRE/CARD SORTING
Activity
Created a card sort to evaluate what Junior/Mid-level UX candidates think are the most desirable skills hiring managers and recruiters look for
Objective
Get quantitative data on what skills UX candidates believe they need to have with regard to the role
Number of Participants
Total: 70, rejected: 21, abandoned: 10, completed: 39
Participant Criteria
UX Candidates pursuing a Junior/Mid-level UX position
HIRING MANAGERS
SCREENER/QUESTIONNAIRE/CARD SORTING
Activity
Created a card sort to evaluate what hiring managers and recruiters value most in Junior/Mid-level UX candidates
Objective
Get quantitative data on what skills hiring committee members think UX candidates need to have with regard to the role
Number of Participants
Total: 29, rejected: 9, abandoned: 10, completed: 10
Participant Criteria
Involved in hiring of Junior/Mid-level UX candidates
CARD SORTING ANALYSIS
After looking at the questions that we asked them, I decided to focus on four specific comparisons:
-
All UX Candidates vs. All UX Hiring Managers
-
UX Research Candidates vs. Hiring Managers for UX Research
-
UX Design Candidates vs. Hiring Managers for UX Design
-
UX Research Hiring Managers vs UX Design Hiring Managers.
All UX Hiring Managers Vs. All UX Candidates
For this comparison, I looked at All the UX Candidates and all the UX Hiring Managers.
1. UX Hiring Managers may think that these types of research is something they just need to be familiar with it because it is some of the simpler parts of UX, because they are easy to learn on the job.
2. When it comes to Design Software and tools, the need for experience depends on what type of company they work at, and how much they rely on those software programs.
3. UX Hiring Managers seem to care more about how UX candidates think, that the specific research methods can be easily learned.
4. UX Candidates tend to underestimate some of the Design aspects of UX and the moderation of research and other UX activities.
Conclusions:
UX Hiring Managers seem to want candidates to be familiar with the different aspects of the Design part of UX, and how to moderate UX activities, to know what needs to be done in order to get results.
UX Research Hiring Managers
Vs.
UX Research Candidates
For this comparison, I looked at All the UX Candidates and all the UX Hiring Managers.
UX RESEARCH CANDIDATES VS. UX RESEARCH HIRING MANAGERS
1. UX Researchers overestimate having experience in the specific research methods of UX Research, like Heat Mapping, A/B Testing, Tree Testing, and Quantitative Research Methods.
2. UX Hiring Managers may think it’s something they just need to be familiar with it, and not have experience in, because they are easy to learn on the job.
3. UX researcher candidates underestimate the skills of data analysis of UX. They are underestimating specific design methods, and the need to be familiar with Product Design and Moderation.
4. UX Hiring Managers seem to care more about the way UX Researchers think, rather than the research methods.
They do want UX Researchers to have experience in the DESIGN methods, since UX Research is all about researching the Design of UX, and them understanding and having experience in certain Design methods can help increase their understanding of UX Research.
UX Design Hiring Managers
Vs.
UX Design Candidates
UX DESIGN CANDIDATES VS. UX DESIGN HIRING MANAGERS
1. UX Designers overestimate needing to have experience in or being familiar with the research methods of UX Research and using analytic tools. UX Hiring Managers may care less about UX Designers having experience in UX Research Methods or analytic tools, due to them not having to be the ones that conduct them.
2. UX Designers seem to really overestimate needing to have being familiar with Adobe Creative Suite. Other programs may be more popular in the UX field, or it may be easier for them to learn.
3. UX Hiring Managers may also like UX Designers to be familiar with the Agile design process.
4. UX Design candidates seem to be underestimating having experience in specific design methods. UX Hiring Managers seem to want Designers to have some hands-on experience with these methods before beginning their career.
5. UX Hiring Managers seem to want UX Designers to be familiar with moderation. They seem to want UX Designers to understand how the user research methods are used, possibly to help them direct their decisions in product design.
UX Research Hiring Managers
UX RESEARCH HIRING MANAGERS VS.
UX DESIGN HIRING MANAGERS
1. UX Research Hiring Managers seemed to be split on requiring the basic UX research methods and moderation between candidates needing experience with them or being familiar with, while UX Designer Hiring Managers considered them less important.
2. When it comes to using analytic tools, UX Research Hiring managers seem to think the experience is nice to have, while UX Design Hiring managers find them unnecessary. This makes sense because UX Researchers tend to be the ones that moderate the activities with users and to use analytic tools to understand the findings.
UX Design Hiring Managers
1. UX Designer Hiring Managers think UX Designers should have experience with Wireframes/Prototypes, whereas UX Research Hiring Managers are split about it.
3. Coding can help with some UX Design work, so Hiring Managers may think it’s nice to have while for UX Researchers find it unnecessary.
4. UX Design hiring managers want UX Designers to be familiar with Agile Design, however UX Research hiring managers seem to think it’s unnecessary.
2. UX Designer Hiring managers think UX Designers should be familiar with Interaction/Visual/Graphic design and working with/creating design systems, and it would be nice for UX Designers to know coding. UX Research Hiring managers need those skills less, especially for coding.
5. UX Designer hiring managers require portfolios, whereas UX Research hiring managers were not as certain, meaning UX Researchers portfolios may not be as important as UX Designers.
UX DESIGN HIRING MANAGERS
VS.
UX RESEARCH HIRING MANAGERS
Skills In Which There Was No Consensus
UX Research Hiring Managers
-
Design Thinking
-
Experience creating Journey Maps/User Flow Charts/Storyboards
-
Personas
UX Design Hiring Managers
-
Presentation Skills
-
Management, Planning, Organizational Skills
-
Project Management Skills
-
Product Design
-
Quantitative Research Methods
SUMMERIZING RESULTS
After analyzing the results of the card sort, I created a presentation of the results, talking about the trends I found and the insights that I drew from the results. I participated in an hour-long presentation in which I talked about my analysis for about 15 minutes.
What Hiring Managers Want
1. UX Hiring Managers seem to care more about how UX candidates think.
2. UX Hiring Managers seem to want candidates to be familiar with the different aspects of the Design part of UX, and how to moderate UX activities, to know what needs to be done in order to get results.
3. Hiring managers want UX Researchers to have experience in the DESIGN methods.
4. Hiring Managers may care less about UX Designers having experience in or knowing UX Research Methods or analytic tools.
What Tech Fleet Should Teach
1. Tech Fleet should focus on teaching UX Researchers on how to show they know and have experience in Design Methods, to show hiring managers they know how to work with designers.
2. So, Tech Fleet should be focused on teaching candidates on learning how to get results in projects, and how to show their work.
3. Tech Fleet should be focused on getting UX Designers actual experience in Design Methods, and some experience of Moderation of UX Research Methods.
In conclusions, our data suggests that both UX Designers and UX Researchers need more experience in Design methods, and a greater understanding on how to get and show their results. It may be because many are trying to break into the field and think that experience is not as important as knowing the processes, but UX Hiring Managers seem to them to have some hands-on experience with these methods before beginning their career.
User Interviews
INTERVIEWS
During the screener survey for both the UX Candidates and the Hiring Managers and Recruiters, they were asked if they wanted to participate in a half hour interview. The participation was voluntary and was scheduled using Calendly tool.
In addition to contributing to the creating and editing of the interview questions, I moderated one interview for the UX Candidates (was scheduled to do two, but one participant dropped out), and took notes for one other interview for the UX Candidates. I also moderated one interview for the UX Hiring Manager Interviews, and took notes for a different interview.
INTERVIEW PARTICIPANTS
UX CANDIDATE INTERVIEWS
Activity
Perform virtual interviews through Zoom with recruited participants
Objective
Get insights into what UX candidates' experience is like when searching and interviewing for UX jobs
Number of Participants
Total: 9
Participant Criteria
Candidates seeking a Junior/Mid-level UX position with no professional UX experience
UX HIRING MANAGER INTERVIEWS
Activity
Perform virtual interviews through Zoom with targeted participants
Objective
Collect qualitative data on what hiring managers are looking for in potential UX candidates
Number of Participants
Total: 5
Participant Criteria
Involved in hiring Junior/Mid-level UX positions
INTERVIEW ANALYSIS
The interview Analysis was done by other members of the team, Krishika and Marija. Their presentation is shown below.
PRESENTING OUR KEY TAKEAWAYS AND RECOMMENDATIONS
After analyzing the results, we gave a presentation the founder of Tech Fleet, Morgan, other members of Tech Fleet, and other members of the UX community. The presentation was about an hour long, where I presented my analysis of the card sorting data for about 15 minutes. The following information below was the summarization of our results and recommendations and our recommendations for future research.
What We Learned
UX candidates focus (too much) on the wording of job postings
Job descriptions are a general guide for creating a repository of potential candidates
Referrals play an important role in recruitment
Portfolio is paramount and instrumental for assessing candidate’s competence and potential
Hard Skills Hiring Managers are looking for
Understanding of the foundations of the UX design process
Application of appropriate research methods
Research principles, research ethics, and best-practices
Soft Skills Hiring Managers are looking for
Collaboration with team members and cross-functional stakeholders
Problem-solving
Communication
Articulation of thinking process
Receptiveness to feedback/criticism
Adaptability and flexibility
Ability to learn
Cultural and values fit
Study Limitations
Recruitment limitations
Small sample size
Samples were not representative of the target populations
Limited resources
Recruitment
Lessons Learned
An $$ incentive might drive more engagement
Try different channels to reach recruiters
Consider a NDA to increase participant confidence in privacy
What We Still Need to Learn
How do skills sets requirements change with organization type and size?
How do requirements as well as prioritization of skills differ based on the regional differences?
If, then how has the pandemic changed the UX skills requirements?
PROJECT RESULT
Our project was the first of many iterations of research on how Tech Fleet can better prepare UX Candidates on getting ready to get into the UX field (Linkedin post of our project below). With our project being the breaking off point, future members of Tech Fleet can expand our research and explore more ways of discovering how Tech Fleet will move forward in it’s goals of improving entry level UX’ers.
Tech Fleet Recommendations
RECOMMENDATIONS FOR UX CANDIDATES
Gain a deeper understanding about the differences of UX positions depending on organization size and type
Use portfolio to demonstrate your problem-solving skills, ability to articulate your thinking process through iterations, and share your hows and whys
Customize your portfolio
Don’t think of UX as an isolated field
Measure and articulate the impact
Keep networking
RECOMMENDATIONS FOR HIRING MANAGERS
More transparency into the role you are looking to fill and your organization’s values
“I just gave you like a very broad brush of what I have and then I cross my fingers, and hopefully through my portfolio you're able to see the keyword that you're looking for.” (Participant–15, UX Candidate)
RECOMMENDATIONS FOR FURTHER RESEARCH
Expand sample for quantitative analysis
Data on recruiters’ perspective
Larger and more representative sample
Other research methods
Observation
Diary study
Longitudinal study
Research the impact
How does apprenticeship in Tech-Fleet influence career path and goals of the volunteers?
