Fishpool

To content | To menu | To search

Tag - business

Entries feed - Comments feed

Thursday 18 July 2013

The difference in being demanding and acting like a jerk

Sarah Sharp is a member of a very rare group. She's a Linux kernel hacker. Even among that group, she's unique - not because of her gender (though that probably is distinctive in many of the group's social gatherings), but because she's brave enough to demand civil behavior from the leaders of the community. I applaud her for that.

Now, I have immense respect for Linus Torvalds and his crew. I've been a direct beneficiary of Linux in both professional and personal contexts for soon to be two decades. The skills this group demonstrate are possibly only matched by the level of quality they demand from each other. However, unfortunately that bar is often demonstrated only on the technical level, while the tone of discussion, both in-person and on the mailing lists, can turn quite hostile at times. It's been documented many times, and I can bring no value to rehashing that.

However, I wanted share some experience from my own career as a developer, manager of developers, and someone who has been both described as demanding and who has needed to report to others under very demanding circumstances. I've made some of these mistakes myself, and hope to have learned from them.

Perhaps not so surprisingly, the same people in the community who defend hostile behavior also, almost by rule, misunderstand what it means to behave professionally. There's a huge difference between behaving as in a workplace where people are getting paid, and behaving professionally. The latter is about promoting behaviors which lead to results. If being a asshole was effective, I'd have no problem with it. But it's not.

To consistently deliver results, we need to be very demanding to ourselves and to others. Being an uncompromising bastard with regards to the results will always beat accepting inferior results, when we measure technical progress on the long run -- though sometimes experience tells us a compromise truly is "good enough".

However, that should never be confused with being a bastard in general. Much can (and should) be said about how being nice to others means we're all that much happier, but I have something else to offer. Quite simply: people don't like to be called idiots and having personal insults hurled at them. They don't respond well to those circumstances. Would you like it yourself? No? Don't expect anyone else to, either. It's not productive. It will not ensure delivering better results in the future.

Timely, frequent and demanding feedback is extremely valuable to results, to the development of an organization, and to the personal development of the individuals. But there are different types of communication, and not all of it is feedback. Demanding better results isn't feedback, it's setting and communicating objectives. Commenting on people's personalities, appearance, or otherwise, let alone demanding them to change themselves as a person isn't feedback nor reasonable. Feedback is about observing behavior and demanding changes in the behavior, because behavior leads to results. Every manager has to do it, never mind whether they're managing a salaried or voluntary team.

However, calling people names is bullying. Under all circumstances. While it can appear to produce results (such as, making someone withdraw from an interaction, thus "no longer exhibiting an undesirable behavior"), those results are temporary and come with costs that far outweigh the benefits. It drives away people who could have been valuable contributors. What's not productive isn't professional. Again - I'm not discussing here how to be a nicer person, but how to improve results. If hostility helped, I'd advocate for it, despite it not being nice.

The same argument can be said about using hostile language even when it's not directed at people, but to results. Some people are more sensitive than others, and if by not offending someone's sensibilities you get better overall results, it's worth changing that behavior. However, unlike "do not insult people", use of swearwords toward something else than people is a cultural issue. Some groups are fine with it, or indeed enjoy an occasional chance to hurl insults at inanimate objects or pieces of code. I'm fine with that. But nobody likes to be called ugly or stupid.

In the context of Linux kernel, does this matter? After all, it seems to have worked fine for 20 years, and has produced something the world relies on. Well, I ask you this: is it better to have people selected (or have the select themselves to) for Linux development by their technical skills and capability to work in organized fashion, or by those things PLUS an incredibly thick skin and capacity to take insults hurled at them without being intimidated? The team currently developing the system is of the latter kind. Would they be able to produce something even better, if that last requirement wasn't needed? Does that help the group become better? I would say hostility IS hurting the group.

Plus, it's setting a very visible, very bad precedent for all other open source teams, too. I've seen other projects wither and die because they've copied the "hostility is ok, it works for Linux" mentality, but losing out on the skills part of the equation. They didn't have to end that way, and it's a loss.

Monday 29 April 2013

Analytics infrastructure of tomorrow

If you happen to be interested in the technologies that enable advanced business analytics, like I am, the last year has been an interesting one. A lot is happening, on all levels of the tech stack from raw infrastructure to cloud platforms and to functional applications.

As Hadoop has really caught on and is now a building block for even conservative corporations, several of its weaknesses are also beginning to be tackled. From my point of view, the most severe has been the terrible processing latencies of the batch- and filesystem-oriented MapReduce approach, rather than solutions designed on top of streaming data. That's now being addressed by several projects. Storm provides a framework for dealing with incoming data, Impala makes querying stored data more processing-efficient, and finally, Parquet is coming together to make the storage itself more space- and I/O efficient. With these in place, Hadoop will move from its original strength in unstructured data processing to a compelling solution for dealing with massive amounts of mostly-structured events.

Those technologies are a bear to integrate and, in their normal mode, require investment in hardware. If you'd prefer to get a more flexible start to building a solution, Amazon Web Services has introduced a lot of interesting stuff, too. Not only have the prices for compute and storage dropped, they now offer I/O capacities comparable to dedicated, FusionIO-equipped database servers, very cost efficient long-term raw data storage (Glacier), and a compelling data warehouse/analytics database in the shape of Redshift. The latter is a very interesting addition to Amazon's already-existing database-as-a-service offerings (SimpleDB, DynamoDB and RDS), and, as far as I've noticed, gives it a unique capability other cloud infrastructure providers are today unable to match - although Google's BigQuery comes close.

The next piece in the puzzle must be analytical applications delivered as a service. It's clear that the modern analytics pipeline is powered by event data - whether it's web clickstreams (Google Analytics, Omniture, KISSMetrics or otherwise), mobile applications (such as Flurry, MixPanel, Kontagent) or internal business data, it's significantly simpler to produce a stream of user, business and service events from the operational stack than it is to try to retrofit business metrics on top of an operational database. The 90's style OLTP-to-OLAP Extract-Transform-Load approach must die!

However, the services I mentioned above, while excellent in their own niches, can not produce a 360-degree view across the entire business. If they deliver dashboards, customer insight is impossible. Even if they're able to report on customers, they don't integrate to support systems. They leave holes in the offering that businesses have to plug with ad-hoc tools. While it's understandable, as they're built on technologies that force nasty compromises, those holes are still unacceptable for a demanding digital business of today. And as the world increasingly turns more digital, what's demanding today is going to be run-of-the-mill tomorrow.

Fortunately, the infrastructure is now available. I'm excited to see the solutions that will arrive to make use of the new capabilities.

Thursday 28 February 2013

What if movies were designed for free-to-play?

This tweet from Ben Cousins over at ngmoco (looking forward to The Drowning!) got me thinking:

In a response, I said that's true if you consider the entirety of the movie industry (where some people buy everything they watch, while some pirate it - and yet another group pirates the movies and buys a lot of movie-related merchandise), but that on the level of any one movie, if they're analyzed from a free-to-play angle, they're terrible businesses.

I guess that obligates me to write something about how would a free-to-play movie work. Not being very well versed in the details of how movies get produced today, I guess I'm either way off in the deep end, or in an advantageous position to speculate about it. Take your pick, shoot me down in the comments. It's entirely possible it's not even possible to make every movie work as a standalone free-to-play (in which case we're back to something like Netflix as the freemium business model for movies), but since we did figure it out for games, why shouldn't we try to figure it out for movies?

What's a good free-to-play product design like? A quick summary:

  • A basic version of the product should be available for free. If someone's motivated enough, they should be able to enjoy the full experience without opening their wallet, but they'll have to contribute in some other way. Pirated movies don't count, that's not contributing. Ad support is a weak solution - better than nothing, though.
  • "Basic" doesn't mean low quality, because the free product should be as engaging as the premium version. A low-rez online clip doesn't cut it, it just drives people back to piracy.
  • The bar to spending money should be really, really low and well incentivized. An Amazon $0.99 rental for 24h counts, iTunes store $15 download does not. The incentives still need work, though - ease of access, good recommendations, easy streaming to the big screen are a good start.
  • The upper limit to how much one customer can spend on the product should be high enough to be practically unlimited. Spending more should always result in some additional marginal value.
  • High value customers come in two shapes: those who buy something really expensive once (such as a collector's edition, like Ben was linking to), or those who keep spending, again and again.

In some markets and for some movies, the industry does manage to capture the middle. However, these are not optional points for a free-to-play design. You have to consider all of them, or you're turning away customers. A free-to-play design typically expects a few percent of the audience to pay for their experience.

On the level of basic free edition, the easy suggestion to make is to have each movie be viewable from its own site in exchange for a Facebook Like or a retweet. By doing that, free viewers are contributing viral visibility to the product. As I mentioned above, this should not be a crappy low-rez edition, but a real, enjoyable stream. Done this way, movies would have to have stable, long-term addresses, rather than the marketing campaign sites they now have, but that would be a good thing. Free-to-play is a lot about the long tail, in both volume and time.

That site can sell offline copies as DVD or BluRay for someone who (for whatever their reasons) can't or doesn't want to stream. That may be quaint, but hey, people still buy vinyl, too. It can also rent the movie for streaming to something else than a computer. Clearly, there would need to be several incentives for someone to want to contribute a couple of bucks for the regular edition of the movie, and this is probably the hardest thing to get right. Eg, you could charge for pause function, but that would be a pretty dick move, likely to drive away people who would otherwise enjoy the experience. Perhaps the free edition should only come to play a month after release, until which streaming always costs.

Stuff like comment tracks, making of, etc can be a paid extra. They're made for true fans, and true fans are by definition willing to pay for the work. Some of that stuff can reasonably be priced much higher than it typically is, today.

Selling merchandize and collectors editions are obviously something the site should feature. It should also have exclusive items, such as limited edition access to the production crew. Just look at any of several successful Kickstarter campaigns to see what might a $5000 edition of a movie be packaged with. Today's featured documentary on Kickstarter about the Arab Spring has 10 premier night tickets next to the crew for that price, and another reward for double that (check it out yourself). The Kickstarter rewards are time-limited, but a free-to-play movie should have similar items available for fans throughout its distribution lifetime. They will need to be refreshed. Free-to-play is a service, not a product.

A re-watch would need to have some special features for it, all of which could be paid stuff. This would benefit some movies much more than others - and create an incentive for artists to create more movies like that. I wouldn't mind!

Now, I haven't even tried to run any numbers on this thought experiment, and I don't know where to pull the reference data. According to Box Office Mojo, last year's top grossing movie was The Avengers at $623M US, and on position #100 was The Five Year Engagement at $29M US box office revenue. The same site estimates US ticket prices today at $8.05, so that would mean 78 million US viewers for Avengers and 3.6 million for "5 year". However, those figures probably do not include rentals or online, and almost certainly do not include merchandize, which I would guess is a substantial extra for Avengers (and included in my suggested model above), so basing any comparisons on those data points would be very flawed.

A blockbuster film like Avengers collects most of its revenue very close to the release date, but other movies, like the perennial favorites Sound of Music, The Wizard of Oz and It's a Good Life, or somewhat more recent examples like Pulp Fiction, Inception or The Fight Club would keep racking views and revenue for years, even decades. So, would the Avengers ever get its current revenue as free-to-play? Perhaps not. Would Five Year Engagement? I don't see why not. Would Pulp Fiction or Fight Club, neither of which apparently make it to the all-time top 200 grossing movies on Box Office Mojo be able to generate a billion dollars off their engaged fan base over time? Of course they would.

Are you prepared to deal with a negative test result?

Someone recently asked me, whether he should do a limited-market test launch for a product he knows isn't finished yet, in order to learn more from actual users. A worthy goal, of course. Perhaps you're considering the same thing. I have, many times. Before you decide either way, consider the following:

What do you expect to learn? If you need to see people using your product, throwing it on the App Store won't help you achieve that objective. Better you go a nearby meetup of people you'd like to see using your product, introduce yourself and ask them to test it while you're looking. If you can't take the product with you or need to have an entire team experience the end-user feedback first hand, invite 10 people over for some pizza (either to your office or some more cozy environment) and record the event on video. I promise you, it'll be an eye-opening experience.

Do you have a hypothesis you're trying to verify? Can you state how you're verifying it? Can you state a test which would prove your hypothesis is false? Are those tests something that you can implement and measure over a launch? Awesome! If not, then you need to think harder and probably identify some other way of gaining the insight you're looking for.

Are you just trying to gather experience of something not directly related to the product itself? Such as, you don't yet know first hand how to manage an App Store launch. Well, you could launch your baby -- or you could quickly create another product with which to learn what you needed to learn. This tactic has the added benefit that such side-products are typically simpler, so they're easier to analyze for the understanding you're after.

But most importantly, what will you do if your test comes back with a negative result? Far too often, this hasn't been given any consideration, and when it does happen (as it typically does, if you haven't thought through the process), the response is "oh, the test failed, never mind, we'll just go ahead anyway". Unfortunately, in most such situations, it was not the test which failed. Rather, it successfully proved that the hypothesis being tested was incorrect. This is a completely different thing, and going ahead without changes would be a mistake after such a result. You have to be prepared to make the hard decisions. For example, Supercell killed Battle Buddies after their test launch showed it would not convert enough people.

You should test often and early. You should gather market data to support significant further investments of time or resources to any development you're undertaking. But you should also be prepared to take any necessary actions if the tests you're running show that your assumptions were incorrect, and the product doesn't work the way you intended. Those are not easy decisions to take, if you're invested into the product, as most creators would be. Think it through. A launch is not a test.

Wednesday 12 December 2012

A marriage of NoSQL, reporting and analytics

Earlier today, I mentioned querymongo.com in a tweet that fired off a chat covering various database-related topics each worth a blog post of their own, some of which I've written about here before:

One response in particular stood out as something I want to cover in a bit more detail that will fit in a Tweet:

While it's fair to say I don't think MongoDB's query syntax is pretty in the best of circumstances, I do agree that at times, given the right kind of other tools your dev team is used to (such as, when you're developing in a JavaScript-heavy HTML5 + Node.js environment) and the application's context is one where objects are only semi-structured, it can be a very good fit as the online database solution. However, as I was alluding to in the original tweet and expounded on in its follow-ups, it's an absolute nightmare to try to use MongoDB as the source for any kind of reporting, and most applications need to provide reporting at some point. When you get there, you will have three choices:

  1. Drive yourselves crazy by trying to report from MongoDB, using Mongo's own query tools.
  2. Push off reporting to a 3rd party service (which can be a very, very good idea, but difficult to retrofit to contain all of your original data, too).
  3. Replicate the structured part of your database to another DBMS where you can do SQL or something very SQL-like, including reasonably accessible aggregations and joins.

The third option will unfortunately come with the cost of having to maintain two systems and making sure that all data and changes are replicated. If you do decide to go that route, please do yourself a favor and pick a system designed for reporting, instead of an OLTP system that can simply do reporting, when pushed to do so. Yes, that latter category includes both Postgres and MySQL - both quite capable as OLTP systems, but you already decided to do that using MongoDB, didn't you?

Most reporting tasks are much better managed using a columnar, analytics-oriented database engine optimized for aggregations. Many have spawned in the last half a decade or so: Vertica, Greenplum, Infobright, ParAccel, and so on. It used to be that choosing to use one might be either complicated or expensive (though I'm on record saying Infobright's open source version is quite usable), but since last week's Amazon conference and its announcements, there's a new player on the field: Amazon Redshift, apparently built on top of ParAccel, priced at $1000/TB/year. Though I've yet to have a chance to participate in its beta program and put it through its paces, I think it's pretty safe to say it's a tectonic shift on the reporting databases market as big or bigger as the original Elastic Compute Cloud was to hosting solutions. Frankly, you'd be crazy not to use it.

Now, reporting is reporting, and many analytical questions businesses need to solve today really can't be expressed with any sort of database query language. My own start-up, Metrify.io is working on a few of those problems, providing cloud-based predictive tools to decide how to serve customers before there's hard data what kind of customers they are. We back this with a wide array of in-memory and on-disk tools which I hope to describe in more detail at a later stage. From a practical "what should you do" point of view though -- unless you're also working on an analytics solution, leave those questions to someone who's focused on that, turn to SaaS services and spend your own time on your business instead.

Thursday 1 March 2012

Increase engagement with social analytics

Last week I discussed segmentation as a method for identifying and differentiating customers for their specific service needs. Whether used for young cohort's introductory period service, high-value segments special treatment, or to identify the group on a transitionary path to high value and help accelerate that process, segmentation is a very versatile tool for business and product optimization. It can be approached with many techniques and I'll go on to more implementation details on those. But first, an introduction to the next topic after segmentation: social metrics.

While social behavior is not historically strongly featured by many products in either the gaming space or in the wider scope of freemium products, your customers and users are people, and thus they will have social interaction with others you can benefit from. If you can capture any of that activity in your product measurement, it can serve as a very valuable basis for in-depth analytics. Today, I will focus on those products and services in which their audience can interact among each other - that is, there is some sort of easily measured, directly connected community.

Any such product will probably have user segments such as:

  • new users who would benefit from seeing good examples of effective use of the product, guidance on the first steps, or some other introduction beyond what the product can do automatically or what your sales or support staff can scale to
  • enthusiasts who would like nothing better than to help the first group
  • direct revenue contributors who either have a lot of disposable income, or otherwise find your service so valuable to them that they'll be happy to buy a lot of premium features or content
  • people who, though they're not top customers themselves, find innovative ways to use premium features for extra value
  • people who are widely appreciated by the community for their contributions, "have good karma"
  • people whose influence within the community is on the whole negative due to disruptive behavior

and many, many others. Two of these groups are easy to identify simply based on their own history, I'm sure you'll recognize which two. The other four are determined largely by their interaction with the rest of the community and other users' reaction to their activities. How do you find them? This is a rapidly evolving field of analytics with constantly growing pool of theoretical approaches and practical tools, and can look daunting at first. The good news, there are many practical tools already, and while theoretical background helps, the first steps aren't too hard to make.

You'll need to develop some simple way to identify interaction. The traditional way to begin is to define a "buddy list" of some sort similar to Facebook friends network, Twitter following, or a simple email address book. However, I find a more "casual" approach of quantifying interactions works better for analytics. Enumerate comments, time in the same "space", exposure to the same content, common play time, or whatever works for your product. At the simplest level, this will be a list of "user A, user B, scale of interaction" stored somewhere in your logs or a metrics database. This is already a very good baseline. With the addition of time/calendar, you'll be able to measure the ebb and flow of social activities, but even that isn't strictly necessary.

Up to data set of about 100k users and half a million connections or so, you'll be able to do a lot of analysis just on your laptop. Grab such a data dump and a tool called Gephi and you're just minutes away from fun stuff like visualizing whether connections are uniformly defined or clustered into smaller, relatively separate groups (I bet you'll find the latter - social networks are practically always have this "small world" property). This alone, even though it isn't an ongoing, easily comparable metric, will be very informative for your product design and community interaction.

In terms of metrics and connected actions, here's a high-level overview of some of the more simple-to-implement things:

  • highly connected users are a great seed for new features or content, because they can spread messages fast and giving them early access will make them more engaged. While in theory you'd want to reach people "in between" clusters, the top connected people are an easy, surprisingly well functioning substitute.
  • those same people with a large number of connections are also critical hubs in the community, and you should protect them well, jumping in fast if they have problems. This is independent of their individual LTV, because they may well be the connection between high-value customers.
  • high clustering coefficient will indicate a robust network, so you should aim to build one and increase that metric. Try introducing less-connected (including new) people to existing clusters, not simply to random other users. A cluster, of course, is a set of people who all have connections to most others in the cluster (i.e., a high local clustering coefficient).
  • Once someone already has a reasonable number of semi-stable relationships (such as, 4-8 people they've interacted with more than once or twice), it's time to start introducing more variance, such as connecting them to someone who's distant in the existing graph. Most of these introductions are unlikely to stick, but the ones that do will improve the entire community a great deal.
  • if you can quantify the importance of the connections, e.g. by measuring the time or number of interactions, you can further identify the top influencers apart from the overall most connected people.
  • finally, when you combine these basic social graph metrics to the other user lifetime data I discussed previously, you'll get a whole new view into how to find crucial user segments and predict their future behavior. This merged analysis will give you measurable improvement far faster than burying yourself into advanced theories of social models, so take the low-hanging fruits first.

That's it for yet another introductory post. Time for feedback: what other analytics areas would you like to see high-level explanations about, or would you rather see this series dive into the implementation details on some particular area? Do let me know, either via comments here, or by a tweet.

Tuesday 21 February 2012

There's no such thing as an average free-to-player

A quick recap: in part 1 of this series, I outlined a couple of basic ways to define a customer acquisition funnel and explained how it falls short when measuring freemium products, in particularly free-to-play. In part 2, I continued to explain two alternative measurement models for free-to-play and focused on Lifetime Value (LTV) as a key metric. A core feature emerged: the spread of possible LTV through the audience is immense, ranging from 0 to, depending on product, hundreds, perhaps even thousands of euros. 

This finding isn't limited to just money spending, but is seen over all kinds of behavior, and is well documented for social participation as the 90-9-1 rule. From a measurement point of view, one of the most overlooked aspects is how it destroys averages as a representative tool. At the risk of stating the obvious, an example below.

When 90% of a freemium product players choose the free version and 10% choose to pay, the average LTV is obviously just 1/10th of the average spending of the paid customers. However, when there's not just a variety of price points, but in fact a scale of spending connected to consumption (or if we're valuing something else than spending, such as time), the top 1% is likely to spend 10x or more than the next 9%. Simple math will show you that the top 1% would be more valuable than the rest of the audience in total, as illustrated here:

The average? 0.19. Now, can you identify the group that is represented by "average spending of 0.19" in the above example? Of course you can't - there is no such group. Averages work fine when what you're measuring follows some approximation of a standard distribution (e.g., heights of people), but they break down with other kinds of quantities. Very crucially, they break down on behavioral and social metrics. Philip Ball's book Critical Mass goes to some length on the history of these measurements, if you're interested in that.

Instead of measuring an average, you should identify your critical threshold levels. Those might be the actions or value separating the 90% and 9% value players, and equally, 9% and 1%. Alternatively, you might already have a good idea of your per-user costs and how much a customer needs to spend to be profitable. Measure how many of your audience are above that level. Identify and name them, if you can. Certainly try to remember them over time to address them individually. This goes deeper than simply "managing whales", to use the casino term. Yes, the top 1% are valuable and important to special case, but it's equally if not more important to determine what are the right strategies for developing more paying customers from the 90% majority.

This is why it's important to measure everything. If you only measure payment, the 90% majority will be invisible to your metrics, and it's usually very hard to identify ahead of time which other measurements are important for identifying the activities that lead to spending. Instrumenting your systems to collect events on all kinds of activities on a per-user basis (rather than just system-level aggregates) enables a data mining approach to the problem. Collect the events, aggregate them across time for each player (computing additional metrics, when appropriate), and then identify which pre-purchase activities separate those players who convert to paying from those in the same cohort who do not. There are several strategies for this, from decision trees to Bayesian filtering to all kinds of dimensionality reduction algorithms. The tools are already pretty approachable, even in open source, whether as GUIs like Weka, in R, or with Big Data solutions like Apache Mahout, which works on top of Hadoop.

Essentially, this approach will surface a customer acquisition funnel akin to what I described earlier, but using the raw measurement data. It will probably reveal things about your product and its audience you would not have identified otherwise, and allow you to optimize the product for higher conversions. The next step in this direction is to replace the "is a customer" criteria above with the measured per-player LTV value. Now, instead of a funnel, you will reveal a number of correlations between types of engagement and purchase behavior, and will be able to further optimize for high LTV. Good results depend on having a rich profile of players across their lifetimes. A daily summary of all the various activities in a wide table with a column for each activity, and a row per player per day is a great source for this analysis.

Friday 10 February 2012

Developing metrics for free-to-play products

In my previous post, I outlined a few ways in which a "sales funnel" KPI model changes between different businesses, and argued that it really doesn't serve a free-to-play business well. Today, I'll summarize a few ways in which a free-to-play model can be measured effectively.

Free-to-play is a games industry term, but the model is a bit more general. In effect, this model is one where a free product or service exists not only as a trial step on the way to converting a paying customer, but can serve both the user as well as the business without a direct customer relationship, for example by increasing the scale of the service, making more content available. From a revenue standpoint, a free-to-play service is structured to sell small add-ons or premium services to the users on repeat basis - in the games space, typically in individual transactions ranging from a few cents to a couple of dollars in value.

As I wrote in the previous article, it's this repeated small transaction feature which makes conversion funnels of limited value to free-to-play models. Profitable business depends on building customer value over a longer lifetime (LTV), and thus retention and repeat purchase become more important attributes and measurements. Here is where things become interesting, and common methodologies diverge between platforms.

Facebook games have standardized on measuring number and growth of daily active users (DAU), engagement rate (measured as % of monthly users on average day, ie DAU/MAU), and the average daily revenue per user (ARPDAU). These are good metrics, primarily because they are very simple to define, measure and compare. However, they also have significant weaknesses. DAU/MAU is hard to interpret as it pushed up by high retention but down by high growth, yet both are desirable. Digital Chocolate's Trip Hawkins has written numerous posts about this, I recommend reading them. ARPDAU, on the other hand, hides a very subtle, but crucially important fact regarding the business - because there is no standard price point, LTV will range from zero to possibly very high values, and an average value will bear no reflection on either the median nor the mode value. This is, of course, the Long Tail like Pareto distribution in action. Why does this matter? Well, because without understanding the impact of the extreme ends of the LTV range to the total, investments will be impossible to target, implications of changes impossible to predict, as Soren Johnson describes in an anecdote about Battlefield Heroes ("Trust the Community?").

Another way of structuring the metrics is to look at measured cohort lifetimes, sizes and lifetime values. Typically, cohorts will be defined by their registration/install/join date. This practice is very instructive and permits in-depth analysis and conclusions on performance improvement: are the people who first joined our service or installed our product last week more or less likely to stay active and turn into paying users than the people four weeks ago? Did our changes to the product help? Assuming you trust that later cohorts will behave similarly to earlier ones, you can also use the earlier cohorts' overall and long term performance to predict future performance of currently new users. The weakness of this model relates to the rapidly increasing number of metrics, as every performance indicator is repeated for every cohort. Aggregation becomes crucial. Should you aggregate data older than a few months all-in-one? Does your business exhibit seasonality, so that you should compare this New Year cohort to the one last year, rather than to the one from December? In addition, we have not yet done anything here to address the fallacy of averages.

The average problem can be tackled to some degree by further increasing the number of cohorts over some other feature than the join date, such as the source by which they arrived, their geographic location, or some demographic data we may have of them. This will let us understand that French gamers will spend more money than those from Mexico, or that Facebook users are less likely to buy business services than those from LinkedIn. This information comes at a further cost in ballooning the number of metrics, and will ultimately require automating significant parts of the comparison analysis, sorting data to top-and-bottom percentiles, and highlighting changes in cohort behavior.

Up until now, all the metrics discussed have been simple aggregations of per-user data into predefined break-down segments. While I've introduced a few metrics which can take some practice to learn to read, the implementations of these measurements are relatively trivial - only the comparison automation and highlight summaries might require non-trivial development work. Engineering investments may already be fairly substantial, depending on the traffic numbers and the amount of collected data, but the work is fairly straightforward. In the next installation, I will discuss what happens when we start to break the aggregates down using something else than pre-determined cohort features.

Tuesday 7 February 2012

Metrics, funnels, KPIs - a comparative introduction

I know, I know - startup metrics have been preached about for years by luminaries like Mark Suster, Dave McClure, Eric Ries and many, many others. The field is full of great companies and tools like KISSmetrics, ChartBeat, Google Analytics, to name but three (and do a great disservice to many others). Companies like Facebook and Zynga collect and analyze oodles (that's a technical term) of data on their traffic, customers and products, and have built multi-billion dollar business on metrics. Surely everything is done already, and everyone not only knows that metrics matter, but also how to select the right metrics and implement a robust metrics process? There's nothing to see here, move along... or is there?

Metrics depend on your business as much as your business depends on them. No, more, in fact. It is possible (though hard) to build a decent, if not awesome business purely on intuition, but it is not possible to define metrics without understanding the business. Applying the wrong metrics is a disaster waiting to happen. In fact, in some ways this makes building a robust metrics platform more difficult than building the product it's supposed to measure. Metrics can't exist ahead of the product, but are needed from the beginning. Sure, with experience you will learn to pick plausible candidates for KPIs, and may even have tools ready for applying them to new products, but details change, and sometimes, with those details, the quality of the metrics changes dramatically. This is obviously true between industries like retail vs entertainment, but it's also true between companies working in the same industry.

This is a big part of why metrics aren't a solution to lack of direction. They can be a part of a solution in that well-chosen metrics will make progress or lack of it obvious, and may even provide clear, actionable targets for developing the business. Someone still needs to have an idea of what to do, and that insight feeds back into all parts: product, operations, measurement. I've never liked the phrase "metrics driven business" for this reason. Metrics don't do any driving. They're the instrumentation to tell you whether you're still on a road or what your speed is. You still have to decide whether that's the right road to be on, and whether you should be moving faster, or perhaps at times slower.

What to do, then? Well, understanding the differences helps. Lets start with a commonly applied metrics model, the sales funnel.

In a business-to-business, face to face sales driven business, a traditional funnel may begin with identifying potential customer opportunity, then measuring the number of contacts, leads, proposals, negotiations, orders, deliveries and invoices. A well managed business will focus on qualified customers and look for repeat transactions, as the cost per opportunity will likely be lower, and the revenue per order may be higher, leading to greater profitability. They will also look at the success rate between the steps of the funnel, trying to improve the probability of developing an opportunity into a first order.

This model is often adapted to retail: advertising, foot traffic, product presentation, purchase decision. For some businesses that's it - others will need to manage the delivery of the product, and may see further opportunities in service, cross-sales, or otherwise. Online retail businesses measure every step in much greater detail, simply because it is easier to do so. Large retail chains emulate that measurement with very sophisticated foot traffic measurement systems. But even in its simplest form, while the shape is similar, the steps of the funnel are very different.

Online businesses have developed a variety of business models, among which two large categories are very common: advertising and freemium.

The advertising-funded two sided market model is two different funnels: a visibility - traffic - engaged traffic - repeat traffic page view model, and a more traditional sales funnel for the advertisers, though even that one has been, through automation, turned to look more like an online retail model than what advertising industry is used to. This model is further enhanced by traffic segmentation and intent analysis, allowing targeted advertising and a real-time direct marketing product, the sales funnel of which is even less familiar to the sales funnel I described at the beginning.

Freemium isn't even one business model: a B2B service with a tiered product offering and a free time- or feature-limited trial product may ultimately use the traditional sales model, only so that the opportunity to prospect part of the model is fully automated. Often it's entirely automated to the point a customer never needs to (or perhaps even wants to, assuming a simple enough product) talk to a sales rep. Still, the basic structure holds: some of the prospective leads turn into customers, and carry the business forward. Free service, be it for trial only or for the starter-level segment, is a marketing cost and a leads-qualifier, enabling a smaller sales force.

On the other hand, the free-plus-microtransactions model, one which we pioneered with Habbo Hotel, and has since been used to great success by many, including Zynga, can certainly be described as a funnel, but to measure it with one requires significant violence to many details. The most important of these is that because individual transactions are typically of very low value, building a profitable business on top of a model which aims for, and measures one sale per customer is practically impossible. This class of business doesn't just benefit from repeat customers, it requires them. Hence, a free-plus (or, as it is called in the games industry, free-to-play) business model must replace counting a "new customer" metric or measuring individual transaction value with the measurement of customer lifetime value. Not just measuring it on average, but trying to predict it individually - both to try to develop 0- or low-value users (oh, how I hate the word) to higher value by giving them better value or experience, and by identifying the high value customers to serve and pamper them to the best of the company's ability (within reason and profitability, of course). And once you switch the way you value revenue, you really need to switch the way you measure things pre-revenue.

Funnels change. There are business models where funnels really can't provide the most instructive KPIs, even though they still may be conceptually helpful in describing the business. As this post is getting long, more on the details of KPIs of free-to-play in the next episode.

Thursday 13 October 2011

Where the chips fall - platform dominator for 2012

It's been about a year since I put my prognosis skills on the line and tried to predict where technology and consumer products are heading. Since today is National Fail Day in Finland, perhaps it's time to try again. Lets see how right or wrong I end up being.

Last year I noted a couple of things about mobile platforms and of the software environments best suited for creating apps on them. While this year has seen a lot of development on those fronts, little of it has been in surprising directions. HTML5 is coming, but not here yet. If WebGL and Intel's River Trail project were supported by the Big Three (IE, Firefox and WebKit, ie Safari/Chrome), that'd make an amazing game platform - but at least the latter is research-only at this point, and IE9 isn't going to support either. In the meantime, Adobe finished Flash 11, which now has hardware-accelerated 3D in addition to a pretty good software runtime, and, after only 10 days out, already has 42% reach for consumer browsers (at least judging by stats on habbo.com). Like I've said a long time, Flash gets a lot of undeserved crap due to the adware content created on it. We won't get rid of that by changing tech, and platforms should be judged by their capabilities in the hands of good developers, not by mediocrity. And, as far as mobile goes, the trend continues -- iPhone and Android battle it out, now also in courts as well as in consumer markets, while everything else falls under the wagon. If you're creating an app -- do it either with a cross-platform native toolchain, or with HTML5. If you're doing a game, do it with Unity or Flash, and build a native app out of it for mobile.

The interesting thing, to me, is playing out on the Internet. Google+ came out as a very nice product with well-balanced feature set, but (fairly predicably, though I was rooting for it) failed to catch the early adopter fancy for long enough to displace Facebook in any niche. Facebook, on the other hand, scared (or is going to scare) 40% of their audience by announcing Timeline (eek, privacy invasion!). Brilliant move -- you can't succeed today without taking such leaps that nearly half of your audience will be opposed to them, at least initially. Smaller changes simply aren't meaningful enough.

So, I'm betting on Facebook. I'd also guess that once they get Facebook Credits working outside of the Canvas, they're going to demand that any app using Facebook Connect log-ins will accept Credits for payment. I'd hazard a guess they're even going to demand FB Credits exclusivity. They'll fail the latter demand, but that won't stop them from trying it. Having your app's/game's social publishing automatically done by Facebook simply by feeding them events, and not having to think about which ones are useful to publish, is just such a big time saver for a developer, no one will want to miss out on it.

Not even Zynga. They're doing this destination-site, we're-not-gonna-play-inside-Facebook-anymore strategy, but continue to use Facebook Connect for log-ins. That's not because FB Connect is so much more convenient than own username and password (though it is), but because even they can't afford to let go of the "free" access to people's social network. That's the power of Timeline and the new, extended Graph API.

The chips are still in the air. When they fall, I think Facebook will be stronger than ever, but strong enough to displace the "rest of the Internet"? No. As a developer, I want to push Facebook the data for in-game activities, because that saves me time doing the same thing myself. As a publisher, I'm unsure I want Facebook to have all that info, exploiting them for their purposes, risking my own ability to run a business. As a consumer, it makes me uneasy that they have all that info about me, and while I can access and control quite a lot of it, I can't know what they're using it for. I don't think that unease will be enough to stop me or most other consumers from feeding them even more data of our lives, likes and activities. Still, they're only successful doing this as long as they don't try to become a gatekeeper to the net - nor do they need to do that, since they get the data they want without exerting control over my behavior. Trying to fight against that trend is going to be a losing strategy for most of us - possibly even for Google. Apple and Microsoft won't need to fight it, because they're happy enough, for now at least, to simply work with Facebook.

- page 1 of 3