ZipRecruiter: Project eLLMo

ZipRecruiter: Project eLLMo

TL;DR

In this case study, we explore the development of eLLMo, an innovative AI-assisted chat conversational UI designed to enhance the job search experience for ZipRecruiter users.

As the VP of Product Design I led the strategy and product design for this project, two designers helped create the designs; Yutong Feng and Jihye Woo, I worked cross-functionally to create a solution that addresses a significant challenge in the job search process: helping users discover suitable job titles when they don't have a specific role in mind.

eLLMo is live; try it out at ZipRecruiter.com

Problem Area

ZipRecruiter faced a challenge: over half of our visitors arrived without a specific job title in mind. These "blank searchers" struggled with a site designed for title-based searches and an activity-driven matching system. The result? Often irrelevant recommendations. Our mission became clear: craft an intuitive, language-based matching approach that understood each job seeker's unique skills and aspirations.

The Users

Our target audience comprised US job seekers aged 18-65, primarily 25-40-year-old professionals, including early career individuals and career switchers in both skilled and unskilled roles who possessed diverse skills and preferences but were uncertain about specific job titles.

The process

Cross-functional Collaboration

I co-led a dynamic, cross-functional team of over 50 innovators, uniting product marketers, user researchers, data scientists, engineers, product managers, and two lead product designers to tackle this transformative challenge.

Dream team

  • Blake Crosley

    VP of Product Design

  • Yutong Feng

    Product Designer

  • Jihye Woo

    Senior Product Designer

The product designers on this project were Jihye and Yutong. Jihye was the lead designer for our day-to-day onboarding experience and iterated on the flow multiple times. Yutong lead the conversational UI creation for the eLLMo project, as well as, helped define how the system would respond to our users working with our data scientists on the prompt engineering.

My role was to co-lead the project along Nishok Chetty, the Director of Matching Product. I was instrumental in the creative direction of the product design, prompt principles, and communication with the Product organization. The design was created by the work of these excellent designers and their major effort. We worked together daily for this eight-week sprint.

User Research and Insights

Our approach to gathering insights was comprehensive and multi-faceted. We started with quantitative surveys from our product marketing team, which provided a broad overview of both ZipRecruiter users and US job seekers in general. To add depth to these findings, our Lead User Researcher, Tianyu Koenig, conducted in-depth interviews, offering rich, qualitative data.

Finally, we analyzed the current site experience and user behavior, giving us a clear picture of how people were actually interacting with our platform. This three-pronged approach ensured we had a well-rounded understanding of our users' needs and pain points.

Experience principles

Adaptive questioning based on user input

Integration with existing preference data

System confirmation for low-confidence results

Flexibility to handle varying data quality

Structured questions and answers from the LLM

Confidence scoring for title recommendations

Explainability for recommendations

Responsiveness to user feedback

Principles Methodology

I created these principles while prompt engineering in the early stages of the project. Below are screenshots from the OpenAI playground where I explored the boundaries of the technology at the time GPT 4.0.

My base prompt was 1,397 tokens long and developed over a week of experimentation. Imagining how the LLM might empower a unique and helpful user experience was critical to the project's success.

Screenshots from my OpenAI Playground experimentation

Iterative Design

Our design process was methodical yet dynamic. We started with established principles, but quickly adapted. Data scientists and product managers provided crucial input, reshaping our initial concepts.

To test our ideas, we built a sandbox version. Real users interacted with it, offering valuable feedback. This led to multiple rounds of user in-depth-interviews in the sandbox and further refinements.

Each iteration brought new insights. We didn't just create a design; we built a solution that evolved with user needs.

Our iterative design process led to several key improvements. We switched to multiple-choice questions and limited their number to five, reducing user effort. A new "Tell me about yourself" prompt was added to personalize the experience. We also incorporated preference settings directly into the conversation flow, making customization more seamless.

Finally, we refined the copy throughout the experience for clarity. These changes aimed to create a quicker, more engaging, and personalized interaction, balancing gathering necessary information and respecting users' time and preferences.

Implementation and Testing

After finalizing the design, we worked closely with our development team to ensure precise implementation. Every detail was carefully translated from design to code. Once the new version was ready, we launched an A/B test to compare it with the original. This allowed us to validate our design choices with real user data before fully rolling out the changes.

The Solution: eLLMo

eLLMo is an innovative matching system that uses advanced language models to connect users with job opportunities. It operates through a conversational interface, engaging users in a natural dialogue to gather information about their preferences and qualifications.

At its core, eLLMo leverages artificial intelligence to analyze user inputs and match them with suitable job prospects. One of its standout features is the ability to provide personalized job title recommendations based on the information collected during the conversation.

Key features of eLLMo include:

1. An AI-powered chat interface that feels natural and intuitive to use.

2. Adaptive questioning that adjusts based on user responses, ensuring relevant information is collected efficiently.

3. Seamless integration with existing structured preference data, enhancing the accuracy of matches.

4. A confidence scoring system for title recommendations, along with explanations for why certain jobs are suggested, providing transparency in the AI's decision-making process.

Results and Impact

eLLMo's launch brought remarkable changes. User satisfaction soared, with 96% happy with job recommendations and 82% praising the innovative approach.

Engagement skyrocketed—first-time job clicks more than doubled, and registration completions rose from 82% to 89%.

Despite added steps, users reported less friction, better personalization, and easier navigation. eLLMo didn't just improve our platform; it fixed the blank job search experience.

First-time Job Clicks Comparison

Satisfaction with job title recommendations

persistence paid off

Integrating LLM technology into our product faced initial resistance from leadership. To overcome this, I took a three-pronged approach. First, I used OpenAI's playground to demonstrate the technology's capabilities in real-time. Then, I presented a clear vision of potential user benefits. Finally, I collaborated with our design team to create an intuitive UI for AI interactions. This combination of practical demonstration, user-focused vision, and sleek design ultimately won over the CTO and COO, paving the way for LLM adoption.

Screenshot of my playground example showcasing how we could leverage LLM technology in our product.

Leading by example

In conclusion, as co-lead of Project eLLMo, I spearheaded our first venture into LLM-based AI product design. I persuaded leadership to adopt this technology and presented our progress to the CEO and a 350+ person cross-functional team. Our team developed crucial skills in prompt engineering and successfully merged AI capabilities with user-centric design principles.

We created a unique and practical user experience that significantly improved ZipRecruiter's platform. eLLMo's success enhanced user satisfaction and opened doors for more AI-driven innovations at Zip. By demonstrating our ability to create cutting-edge solutions, we secured support for additional LLM projects, positioning our team at the forefront of AI integration in the company.

Research and Design celebrating eLLMo’s success