A Brief Project History

Setting the stage

I joined the team at a time when reporting and analytics ranked highest on an opportunity assessment conducted in partnership with the customer advisory board in close contact with customers.

We had to start with our users as our north star, not market validation.

I shifted the narrative from high level market and customer validation to problem definition. How do we enable our users to solve the problems they have and the jobs they are trying to accomplish with analytics? Problems that, up until this point in time, had not been fully identified.

There are common denominators in the data that marketers need to derive insights from (think the obvious ones, open rate, click rate, etc.). However, the ways in which marketers prioritize and put this data work were drastically different, given the size of their department, the industry they operate within, level of sophistication, etc.

Bringing an MVP analytics offering to market that accomplishes table stakes doesn’t mean we’ve advanced our position in the market, or even caught up. Because it doesn’t mean that we’ll meet user expectations - Emma user’s expectations.

The problems for marketers today

I studied the competitive market

The ESP market is highly competitive and saturated with ESPs positioned within the SMB space to mid-market space. In order to understand where Emma could carve out a competitive advantage, I had to familiarize myself with other ESP offerings.

I discovered user perspectives

Emma comprises of two separate platforms. One is the ESP, which serves subaccount owners, typically small business owners. The other is HQ, which serves larger mid-market franchises, typically with multiple locations.

After conducting an initial round of interviews to uncover user needs, I synthesized and clustered the data, uncovering two primary personas with very different goals, use cases, and needs.

I developed our target user personas

After conducting discovery research, I guided the team towards two primary personas: manager level markets of a single sub-account and Director level personas who oversaw multiple subaccounts.

Prioritizing, inclusively

As a team, we collectively shared ideas, and prioritized.

User surveys

User prioritization of feature offerings

User prioritization of metrics offerings

Design

I tested two approaches, embedded content versus cards.

Once I defined the approach, I tested different layouts.

Lo-fi iteration towards the optimal information design

I tested multiple layouts with users to find the right information design and visual hierarchies that aligned with their preferences.

I used a mix of research methods to test prototypes

Our initial release started small and then scaled

After the initial release, feedback overall was positive. One main issue was the call-to-action and ease-of-access to detailed data. I tested different button positions, but ultimately decided to pivot.

Iterations in hi-fi

Pivoting based on user analytics

Tracking customer satisfaction score

We tracked CSAT scores from post-launch through multiple releases and post-MVP improvements in a very systematic way to understand if the improvements we were release were in fact increasing customer sentiments.

Post-launch adoption

We defined adoption as accounts who used insights at least once per week consecutively over a one-month cycle post-launch. 34% adoption rate for Plus and Premium accounts (our target audience) wasn’t bad. Our target was 30%.

6-mo post-launch drop

After around 6 months, we noticed that overall use was starting to drop. This didn’t have us too alarmed, but we dug in, to see what was going on.

Deepening our understanding

Understanding the organization

Aligning around a hypothesis

Based on the research, one central theme surfaced. Users were pleased with the high level overview of growth and engagement, but couldn’t action on that data in a way that helped the team and helped them adjust their marketing strategy.

Introducing the jobs to be done framework

I used Clayton Christensen’s Jobs to be Done framework. In doing so we discovered that there were six (6) primary jobs that users conducted on a frequent basis.

Will it help me compare my performance?

Users were comparing performance, engagement, and growth, one timeframe to another. We tested different approaches to enable easy comparison in product.

Will it help me share data across my team?

We learned that users were often sharing reports, on specific days for weekly / monthly meetings, or generally, across the marketing org. So, we helped automate that process by giving users the ability to send weekly reports out to member on their team. Working with our branding team, we designed an emailed report, that displays the metrics of their choosing.

First, I tested approach - a modal, a wizard, or a page

The user data suggested a modal. Then, I a/b tested a tabbed navigation

Against a walkthrough navigation

The walkthrough won, due to higher completion rates and lower error rates

Will it help me send at the optimal time to increase engagement?

One of main jobs users were trying to do, was to figure out when was the best time in the day and week to send a campaign.

Will it help me personalize my content based on engagement?

Another primary job users were trying to do, was to break down their audiences into different segments representing their level of engagement, so that they could then market to those audiences with customized content.

Will it help let me compare campaign engagement and audience growth?

Users needed the ability to understand their campaign performance at a high level, and at a detailed level, and quickly move in and out of those contexts as necessary.

Project outcomes

1. Over 30% healthy customer adoption at launch

2. Drove additional 12% adoption, up to over 40% through additional post-MVP features

3. 15% reduction in professional service requests

4. 10% reduction in support cases

5. 8% increase in segmentation use

6. Similar increases to other tangential feature set, such as organization of channels (groups, forms, etc.)

7. 9% reduction in compare mailings use

8. 5% reduction in campaign level metrics

9. 10% reduction in exports among target segments

10. 2nd highest viewed webinar in Emma history

11. Trending over 70% CSAT scores

12. Positive NPS correlation to insights use, which is a driver for retention

Key performance indicators

12-mo post launch adoption increase

12-mo post launch growth

We accomplished an increase in average number of monthly visitors, accounts and clicks. In other words, scaling the insights experience with the features we decided to prioritize increased use and stickiness.

Customers who adopted insights were over 20% more likely to renew.

Upward trending CSAT scores

Over the course of a year, we tracked CSAT scores and fell below to 60 to trending above 70. Increases correlated to post-launch feature releases.

Vision explorations

Throughout the project, we received numerous requests for a centralized dashboard view from both our least sophisticated and our more sophisticated marketers. Both ultimately wanted to control custom views into marketing performance.

Furthermore, both our more sophisticated marketers started requesting more advanced aggregate metrics for social, impressions, shares, conversion rates. At this stage, we were confident that we enough data, validation and interest to justify building out a dashboard to accommodate these needs, and offering it as a solution for our premium subscription.