Measuring on Mobile – Best Practices

Standard

Last week I spoke at 500startup’s #500distro event on best practices for mobile analytics. It was an interesting event with lots of great talks – 500Startups did a fantastic job of arranging an in-depth look into the challenges of acquiring and retaining users. I heard the word retention over and over again – this is a huge change from the way people were speaking about analytics a couple of years ago. It was refreshing to witness the shift from “traffic” to metrics that really matter.

My talk focused on the nuances between web and mobile analytics. These differences stem largely from the fact that mobile offers much less real estate, but benefits from higher user acceptance of authentication. This means that:

1. The signup funnel is worth obsessing over

2. A/B testing is much harder, but so very important with that limited real estate

3. Retention has a steeper drop-off, but it’s easier to measure – and push notifications become an invaluable tool for re-engagement

Here is the full deck:

http://www.slideshare.net/500startups/500distro-measuring-on-mobile-making-the-most-of-the-signup-funnel-other-analytic-hot-spots

And the video is here:

https://www.youtube.com/watch?v=6W7gbFQwzRY&list=UUsFOY-wGT9IOC4Okrq6UBfA


Screen Shot 2014-08-16 at 11.00.09 AM

Sitecatalyst v Mixpanel

Standard

I’m finding that suddenly lots of enterprises are moving from Adobe Sitecatalyst to Mixpanel. Hallelujah! Let’s get data out of the hands of data scientists and into the hands of the entire organization. Let’s democratize data!

What might you gain from migrating to Mixpanel?

1) Mobile - First off, Mixpanel is excellent at mobile, so if you care about marrying engagement metrics from your website and your mobile apps, Mixpanel is the only option.

2) Ease of Implementation - Mixpanel is much easier to implement than Sitecat. No need to deal with eVars and sProps – Mixpanel has one type of property which can describe any event or person. Our easy-to-implement Javascript library and mobile SDKs make us very developer-friendly.

3) Ease of Use - Sitecatalyst requires training classes to use, which means there tends to only be a handful of people in any company that use the tool. Mixpanel is extremely easy to use and comes with unlimited user seats, which democratizes data access across the organization.

Compare these two queries – which looks easier to decipher?



4) Arbitrary Segmentation on the fly - With Sitecatalyst, you have to pre-define custom relationships to segment one variable by the other. In Mixpanel you can segment by unlimited properties on the fly.

5) Unsampled Raw Data - If you add Discover onto Sitecatalyst, you gain access to a java desktop app to run analysis on sampled data. With Mixpanel, you can get all of your unsampled data out instantly via API, for no additional fee. 

6) Lacking Features - Sitecatalyst does not offer cohort analysis, conversion history, live view, surveys, people profiles, or notifications.

7) Instant Reporting - Data hits Mixpanel immediately, and reports over millions of data points run in less than a second. With Sitecatalyst, your reports take up to 24 hours to run.

8) Cross-device Analytics - You can send data to Mixpanel from any platform, and it’s easy to manage identity. This means you can get a holistic view of your users across web, mobile, your back-end, or even a device (if you’re a hardware company).

8) Default Properties - Mixpanel automatically captures information about the user’s OS, referrers, browser, device, app version, etc. With Sitecatalyst it’s a great deal of work to capture and parse this data from the user agent on your own.

I think we’ll see more and more companies moving from this large, slow-moving incumbent to scrappier, easier-to-use tools like Mixpanel.

What do you think?

User Quality: Engaged users matter more than MAUs

Standard

I hear it over and over again – people have no idea how often their users visit their app, or which ones visit the most frequently.

really like me

The stats that people tend to be able to rattle off include number of sessions each month and number of monthly average unique visitors. These are nice vanity metrics to give your investors and are useful to measure growth over time. Where these numbers are less helpful is in understanding why you are growing.

To understand the why, we need to dig into user engagement. An increase in user acquisition can be attributed to marketing, but an increase in user engagement either means you’ve done something to make your product better or you’ve found a superior audience. There are two questions you should be asking:

1) Did engagement increase when I made a change to my product?

2) Is there a cohort of users that is more engaged than others? If yes, what characteristics does this cohort share?

Cohort analysis allows you to understand who comes back to your app each month, each week, each day, or even each hour. Mixpanel calls this hourly cohort analysis addiction factor and makes it easy to create cohorts based on anything you know about your users. 

For app developers, for example, a quick and easy report to run is retention or addiction by app version; rather than waiting for your users to update their apps and looking at cohorts purely based on time of signup, you can see immediate impact based on app version.

Screen Shot 2014-06-01 at 6.45.54 PM

For both apps and websites, you can run A/B tests and track which cohort is most engaged. What could possibly matter more to your product decisions than how they impact user addiction?

You might spot some variation across devices, browsers, operating systems, and screen size, for example. These insights can help you tweak low-performing variations to increase addiction.

From a marketing perspective, it’s interesting to dig into whether demographics impact engagement. Is there a gender or age difference? Are certain countries or cities more engaged? Understanding who gets the most of your app is key to effectively targeting your advertising dollars.

What this all boils down to is this: simply focusing on the number of unique visits each month (MAUs) is less helpful than focusing on the quality of those MAUs. We don’t just care about acquiring users, we care about gaining engaged, or addicted, users. In other words, it’s not just about user acquisition  – it’s about user quality. Do your users really like you?

 

 

The Art of the Freemium Sale

Standard

Considering going freemium? There are two questions you should ask yourself:

1)   Is your product really, really great?
2)   How difficult is it to setup? If your answers are “yes” and “a breeze,” then go right ahead!

If your answer to the first question is “maybe” then freemium is NOT the right model for you.
If your answer to the first question is “yes” but your answer to the second question is something along the lines of “well, it depends…” then, you’ve got a few items to take into consideration.

Freemium image

I would put products like Salesforce, Marketo, and Google Analytics in this category. The first two companies require annual contracts up front, with no trial period. They are quite difficult to setup, requiring system integrators, consultants, and full-day training sessions.  Google Analytics, on the other hand, takes the freemium approach – though it must be admitted that Analytics is not Google’s main source of cash flow. I would also argue that Google Analytics is certainly not really, really great; it is mediocre at best.

The difficulty with going freemium on a product that requires technical resources is that the implementation or setup must occur before the sale. This means that you’ll need sales engineers on hand to assist non-paying clients, and that your sales people will require a higher technical comfort level.

A traditional sales cycle goes something like this:

1)   A lead comes in and is qualified by an SDR
2)   The SDR hands off the lead to a Sales Rep
3)   The Sales Rep shows the lead a demo, perhaps with the assistance of a sales engineer
4)   The Sales Rep negotiates the deal, paid in-full, up-front, with an annual commit and no trial period
5)   The deal closes and the client is on their own to get up and running.

The Sales Rep is no longer involved with the deal, so has very little vested interest in the quality of the implementation. The Sales Rep, in other words, really just plays referee and negotiator. His job is to look for buying signals, hunt for needs and pitch benefits, name-drop competitors, and navigate the decision-making and budget discussions. He does not really need to understand the product in depth.

referee

With a technical freemium product, the process is much different.

1)   A lead comes in and is qualified by an SDR
2)   The SDR hands off the lead to a Sales Rep
3)   The Sales Rep shows the lead a demo, perhaps with the assistance of a sales engineer. The main goal of the demo, beyond further qualification, is to get the lead to integrate the product.
4)   The Sales Rep oversees the integration process, with the assistance of a sales engineer. A poor implementation leads to a lost deal, so the Sales Rep is
motivated to make sure the implementation is executed well.
5)   The Sales Rep works to help the lead see the value in their implementation, which requires familiarity with the product. The focus is less on the negotiation, because presumably the client is happy with the product – when they go over their free limit, they will simply pay or lose access.

The Sales Rep, in this role, is playing more than referee and negotiator; the Sales Rep is playing consultant. Google Analytics gets around this by not selling their free version – it’s completely up to users to figure it out. Their paid version comes with an “Expert Consultant” and runs $150k per year.  Adobe Sitecatalyst charges $15,000 for implementation work. Marketo charges a setup fee, and Salesforce pushes contractors at their clients.

Here at Mixpanel, we’ve taken a different approach – we’ve hired very smart, technical Sales Reps who can serve as experts and consultants throughout the implementation process. We don’t charge fees for these services. Finding these people has not been easy, however; we’ve interviewed hundreds of salespeople over the last couple of years, and only hired a handful. So, consider this: how hard is it to get up and running on your product?  If it’s a great product, and easy to get started (Dropbox, Docusign, Yammer) – then by all means, go for it! If it’s a bit trickier to implement, do you think you can find the salespeople you’ll need to be successful?