Beta Testing Community

Customer obsessed, constantly

Introduction

In anticipation of the launch of the "Journeys" feature in the Calm app, our team faced the challenge of collecting authentic, actionable user feedback to refine and enhance the user experience with a rapidly approaching, high visibility launch. In response to this we decided to create a dedicated beta testing community for this new Calm feature—and if successful, for future product iterations as well. This initiative aimed to foster a collaborative environment where real users could engage with the new feature, share feedback, and contribute to its development.

Objective

Our primary goal was to gather rich user feedback on the "Journeys" feature to address three main user challenges before its public launch:

• Most users not knowing what to do in the app
• Most content being too difficult for them to complete
• Users not making/feeling progress in the app and not coming back

We wanted to create a dynamic and engaging beta testing community that would provide valuable insights into user behavior, preferences, and the overall usability of "Journeys."

Why Create a Beta Community?

Prior to launching the Beta Community we were internally testing the new feature with Calm employees. The question came up of why would we want to test externally when we were capturing so much feedback internally? Here's why.

Diverse Perspectives

• User Feedback: Represents a wide range of user demographics and preferences.
• Internal Employee Feedback: Limited to internal perspectives and biases.

Validation of Assumptions

• User Feedback: Validates assumptions through real-world usage.
• Internal Employee Feedback: May reinforce internal assumptions without external validation.

Authentic User Experience

• User Feedback: Reflects real-world usage scenarios and authentic user experience
• Internal Employee Feedback: May be influenced by internal knowledge and assumptions.

Unbiased Evaluation

• User Feedback: Provides unbiased insights into user satisfaction and usability.
• Internal Employee Feedback: Could be biased by personal preferences or company culture.

Strategy

Community Platform Selection

After evaluating several platforms, we chose Discord for its real-time communication capabilities, fostering a sense of immediacy and connection among participants. This choice facilitated ongoing discussions, live feedback, and a structured yet flexible environment for user interaction.

Participant Recruitment

We targeted a diverse group of users, including current subscribers with varying lengths of engagement with the app and interested non-subscribers. Recruitment was strategically designed to capture a wide range of perspectives, ensuring the feedback would be representative of our broader user base.

We also invited internal employees to the community so they could observer and interact with the beta testers. We had cross functional participation from design, product management, engineering, content, licensing, Calm Studios, growth marketing, marketing, customer support, data, and UX copy.

Engagement and Feedback Collection

To maximize engagement and obtain structured feedback, we organized the community around specific channels for different aspects of the "Journeys" feature. Regular prompts encouraged users to share their thoughts on onboarding, personalization, activity selection, and progression. Moderated Zoom interviews provided deeper insights, while in-app behavior was monitored through TestFlight builds linked to users' Calm accounts.

Key Challenges and Solutions

Building Engagement

Initial engagement was boosted by offering incentives, such as a free year of Calm subscription for early participants. Regular, scheduled prompts and engaging discussions facilitated by our team kept the conversation lively and focused.

Managing Feedback

The diversity of feedback presented a challenge in terms of categorization and actionability. We addressed this by implementing a structured approach to feedback collection, using tagging and prioritization to identify key themes and areas for improvement.

Integrating Insights into Development

Rapid iterations based on user feedback required close collaboration between the design, product, and engineering teams. Regular syncs and a dynamic roadmap allowed us to quickly implement changes and test new ideas within the community.

Outcomes

The Calm Beta Testing Community provided invaluable insights that significantly influenced the final "Journeys" feature. Key outcomes included:

• Enhanced personalization and user guidance within "Journeys"
• Improved clarity and motivation around progression and levels
• Introduction of new, highly appreciated activity types based on user suggestions
• Development of a clearer, more engaging onboarding process

Conclusion

The creation and management of the Calm Beta Testing Community were instrumental in the successful launch of the "Journeys" feature. This case study not only highlights the importance of community-driven development in creating user-centric digital products but also showcases the effectiveness of leveraging real-time platforms like Discord to foster meaningful user engagement and collaboration.