Fishpool

To content | To menu | To search

Tag - conference

Entries feed - Comments feed

Tuesday 28 April 2009

The MySQL community outlook

While I can not consider myself a member of MySQL's community of developers, I've been watching those developments the same way I follow the development of Linux and many of the Java and Apache projects our own services depend on. It was great to meet many of the core members of the development community and get some insight into their thoughts about the future.

Baron Schwartz called in his Percona Performance Conference keynote on Thursday for a new, active MySQL community to take the driver's seat in the development of the database, not just in the incremental improvements way of bug fixing and performance improvement, but also by setting a vision for the next generation MySQL. It's a call to action greatly needed, and an important one despite the active existence of the Drizzle project. This is because while Drizzle already has a vision for the future, it's a radical diversion for the MySQL userbase and one which will not necessarily have smooth upgrade path. Many of the same MySQL users feeling most of the pain of MySQL's current limitations are also those who will not be able to easily upgrade to a radically different architecture due to the amount of data and dependencies in their existing infrastructure.

It's a gap which needs a careful approach of incremental changes to the MySQL base functionality to help users bridge over to a new, brighter future. These changes do not need to be slow. Rapid incremental changes are likely to be easier to digest with a clear upgrade and downgrade path from iteration to iteration leaving the organizations with biggest infrastructures to consider a way to set their own pace through the transition, rather than being forced to take one huge leap and risk a crash to the concrete wall of unexpected incompatibility.

A few such pieces of incremental community improvements I learned a great deal of during the week were the performance and scalability improvements by Google and Percona and their MySQL 5.4 equivalents, the Xtrabackup utility not only as an alternative, but improvement on the Innobackup tool which has significant limitations to its use in large-scale deployments, and the Tungsten Replicator providing useful cross-database replication and rapid failover features helping upgrades and transitions to new database installations while minimizing downtime and impact to users. I'm also curious about the storage engine development by Primebase - I don't think there's ultimately a lot of room for multiple transactional storage engines, but as a competitive research topic, it's certainly good to see alternatives to InnoDB.

[Be sure to check out my earlier posts of the conference learnings as well!]

Monday 27 April 2009

Database innovation on MySQL

If MySQL's core server development and release process has been somewhat of a frustration to the userbase over the past few years, clearly another part of the ecosystem has thrived in ways which brought exciting fruit to the Expo part of this year's conference. MySQL has become a hub of innovation in both transactional and analytics databases in ways which have turned many of my concerns to enthusiasm.

I've already discussed the technologies for data analytics on MySQL, in particular Infobright's storage engine technology. This year I took the opportunity to learn a bit more about their appliance-based competitor Kickfire as well, and it certainly looks like a solid product. I still don't completely understand what the "SQL chip" in their appliance does, but certainly the combination of a special-purpose columnar storage, high-speed memory interface and high-performance indexing should form basis for a great analytics system. How it compares in practice to Infobright's software-only approach, time will tell. I'd be interested in real-world experiences, so if you have some to share, please get in touch. Finally, I missed the Calpont info myself, but once it is released, I'll try to get the time to try it out.

I'm even more excited about the new solutions on the transactional side of things. I've certainly been among the people frustrated by MySQL/InnoDB's scaling issues on modern hardware, and glad to see that the optimization work done by Google, Innobase and Percona is being accepted to the "mainline" MySQL Enterprise Server. However, what I did not expect to see were the solutions shown by Virident and Schooner for accelerated, Flash-based storage appliances. It's interesting how both of these companies have chosen to apply their platforms to accelerate both InnoDB and Memcached, and I'm looking forward to the chance to spend more time with both solutions. While both are Flash-based approaches, they seem to have taken very different architectural choices in the way they're exposing the memory to the software layer, and I'm curious to see the impact those choices have on both IO and storage capacity scaling. In any event, these are unique technologies unlike what I've seen for other platforms at this time. I need to learn how they plan to work with the community and Sun/Oracle in keeping the solutions functionally compatible with standard MySQL server.

The ecosystem doesn't end at the appliances, though. On the software side of things, I was pleasantly surprised by the state of Primebase's PBXT storage engine as well as Continuent's new Tungsten Replicator. While both are still early in their development path, they seem to hold a lot of promise for improving the performance of MySQL's built-in functionality in InnoDB as well as in the replication subsystem. Robert Hodges's demo of Tungsten's set-up and management also looked like it will greatly simplify replication administration, which is a big deal for anyone who has to manage 20+ replicated database systems. What's more, if Robert and his team crack the multi-threaded replication problem, and major scalability concern is lifted.

[Be sure to check out my earlier posts of the conference learnings as well!]

Sunday 26 April 2009

MySQL 2009-2010 roadmap

The development model for MySQL Enterprise took a big step forward with the new community process Karen Padir announced in her Tuesday keynote. This is great for both the open source server as well as enterprise customers, because the closer the tie between the community and the development path, the better the quality and faster the progress towards new functionality. I'm not entirely sure everyone at Sun still completely understands why a working community process is a benefit for the enterprise customer base, but I'm happy steps are made in the right direction, and it seems to me that Karen Padir is going to be a good leader for the product.

A big improvement, for sure, and still there's more to improve here. To borrow the words of Baron Schwartz, MySQL currently "has" a community, while it would really be in everyone's benefit if instead MySQL would "be" a community. I would suggest that the goal should be not monthly "community" releases from Sun, but a completely out-in-the-open development process with the community members being on the driving seat regarding patch acceptance, quality management and releases, much like the Fedora process works. Sure, there's a role for corporate sponsorship and project management, but it's a distinct difference of responsibility. The Drizzle project is another good example of how this can work. An important point to realize here is that there is a difference between the community, an active partner in the process of making the software better, and the unpaid userbase. The latter is an acquisition and conversion vehicle for the former, but they're separate entities.

The announcement of the 5.4 server was at the same time an encouraging as well as confusing example of the changes. I would like to be enthuastic about it, but we've seen MySQL (if not Sun) announce pre-announce releases that didn't appear before, and it's a long way to the promised release time. I asked two questions from many, many MySQL staff members during the week: why is it that 5.4 was announced now, but is slated to be released GA only in December when it clearly demonstrates massive scalability improvements already, and why is it that the feature list for the final 5.4 release is much longer than what's already completed? I did not get a really coherent answer from anyone. Best I could decipher, there is somewhere a faceless "marketing" which decided that a) there should only be one release announced and b) 40% demonstrated improvement is not good enough when it's not the only improvement that can be made. I also learned that it's not unlikely that much of the work which has gone to 5.4.0-beta would be backported to the 5.1 branch and released in a 5.1 point release before the actual 5.4 release, because in fact they can be considered bugfixes.

I consider myself not an entirely unexperienced in the decision processes for release management, and know intimately the clarity hindsight provides to well-intentioned choices made with best available information. I know there are many areas to consider, and every decision made is a compromise. I still can't bring myself to completely understand what exactly led to this particular approach. Lets recap:

  • Improvements already made are announced and made available in beta test form, but beta does not contain everything planned for the release
  • Final release is intentionally delayed by 7 months adding significant project risk to it, despite having no previously committed release schedule
  • Former release version is planned to by improved by making significant performance-altering changes in a point release in order to offset the delay
  • Such a release adds risk to maintenance roadmap and steals away upgrade motivation from the upcoming version

How this plan serves either Sun, the community, the free userbase or the enterprise customers is a mystery to me. It would certainly seem far simpler and clearer to take an aggressive quality assurance and release testing position with the intent to push 5.4 out as a rock-solid replacement upgrade to 5.1 as soon as possible, and only then continue with further updates as a 5.5 release. This would definitely be welcomed by everyone but the class of enterprise customers who like to hear about future versions two years in advance - but keep in mind that such conservative enterprises are not MySQL's primary customer base anyway, and if MySQL is to make inroads there, rapidly improving the quality and performance of the product in the meantime would still be a sensible step.

There is the argument that if I want to get those performance features now, I can use Percona/XtraDB or MySQL 5.1 plus the InnoDB Plugin. While technically that route does work, and clearly is worth pursuing as a user, it does have its drawbacks in terms of requiring multiple sources and it's hard to see how it supports MySQL/Sun's commercial interests, the latter surely having been a consideration in the 5.4 release plans.

Thus far in the argument I have ignored one new component - Oracle. That's because to my understanding the process I've discussed did not consider the acquisition, which was unknown to most people before Monday. Clearly this changes a few points. It's not necessarily in the interests of Oracle for MySQL to continue making inroads to enterprise customers, though if someone's going to be cannibalizing Oracle's database sales, it might as well be Oracle. InnoDB Plugin will also be a product from the same company as MySQL Server in the near future - in fact, in a future likely to be fact before the final GA release of MySQL 5.4. What is the role of a delayed 5.4 release in this equation, then?

Recap of MySQL Conference 2009

This was an interesting week for sure. Of course, we all know it started with a bit of a shock news, but that's not nearly the most interesting bit about the conference. I'm posting a series of cleaned-up notes and opinions about what I saw there as I finish them. Will also try to link to further information where I've seen good notes. Please leave more links in the comments if you have any!

Thursday 23 April 2009

Three domains of data

My MySQL Conference presentation on Tuesday discussed my practical findings on how Infobright's technology works in developing a MySQL-based data warehouse. I also touched on a more high-level question of how to select a technology for a different kinds of data-related problem areas, and this article expands on that discussion.

Continue reading...

Wednesday 22 April 2009

Mining for insight - presentation materials

Completed my MySQL Conference presentation 45 minutes ago. Seemed to go over ok, got some followup questions. Trouble is, I got hit by amazing jetlag half an hour before the session, and almost fell asleep myself during the presentation. Fortunately, survived that anyway, and as far as I could see, was the only one having problems staying awake. Below is an embedded version of the slides, which should also appear on the conference proceedings site later. Now for a beer at the expo. Will blog with more description of the stuff later (update: see this follow-up article).

Read this doc on Scribd: Mining for insight

Wednesday 8 April 2009

Using the Infobright Community Edition for event log storage

Apart from the primary "here's how we ended up using Infobright for data warehousing and how is that working out" topic I'm going to discuss in my MySQL Conf presentation I'll touch on another application, the use of Infobright's open-source Community Edition server for collection and storage of event logs. This is a system we've implemented in the past couple of months to solve a number of data management problems that were gradually becoming problematic for our infrastructure.

We've traditionally stored structured event log data in databases for ease of management. Since Habbo uses MySQL for most everything else, putting the log tables in the same databases was pretty natural. However, there are significant problems to this approach:

  • MyISAM tables suffer from concurrency issues and crash-recovery is very slow due to table consistency check
  • InnoDB tables suffer from I/O bottlenecks and crash-recovery is very slow due to rollback segment processing
  • Both scale badly to hundreds of millions of rows (especially if indexing is required), and mixing them is not a recommended practice
  • Storage becomes an issue over time, especially as indexes can easily require many times as much disk than the data, and an event log is going to have a LOT of rows
  • Partitioning has only recently become available, and before that, managing "archive" tables needed manual effort
  • Perhaps worst of all (as it's very hard to measure), if any of this is happening on the primary DB servers, it's competing for buffer pool memory with transactional tables, thus slowing down everything due to cache misses

Over the years, we've tackled these issues in many ways. However, with our initial experience of scaling an Infobright installation for data warehousing needs, a pretty simple solution became apparent, and we rapidly implemented an asynchronous, buffered mechanism to stream data into an ICE database. We're early with this implementation, but it has turned out to be a satisfactory high-performance solution. Even better, it's a very simple thing to implement, even in a clustered service spanning many hosts, as long as log tables don't need to be guaranteed 100% complete or up-to-date to the last second. Here's a description of the simple solution; extending that to the complex solution providing those guarantees is left as an exercise to the reader.

Rather than running single INSERTs to a log table or writing lines to a text file log, each server buffers a small set of events, eg for the past second in a memory buffer. These are then sent over a message bus or lightweight RPC call to a log server, which writes them to a log file that is closed and switched to a new file after every megarow or every few minutes, whichever is smaller. A second process running on this log server wakes up periodically and loads each of these files (minus the last one, which is still being written to) into the database with LOAD DATA INFILE.

This has multiple general benefits:

  • Buffered messaging requires much less time on the "client" servers compared to managing database connections and executing thousands of small transactions
  • The asynchronous processing ensures database contention can not produce random delays to the normal service operation
  • Batch loading of text files is implemented by every DB server, so there's little in this implementation that is proprietary or dependent on any particular DB solution
Using the Infobright ICE as the backend database provides a number of additional specific benefits:
  • Excellent data load performance
  • No index management, yet capability to run queries on the data without first extracting it to another DB
  • No degradation of performance as deployment size grows, as would happen even to a MyISAM table should it have any indexes
  • Compressed storage, so less spinning iron required
  • Columnar datapack organization should not require table partitioning even over long periods
This works very well for structured events. For unstructured data, a different solution is required, which I will discuss at some later date.

Update: Mark Callaghan asked in the comments for some quantified details. We have not spent the time to produce repeatable benchmarks, so all I can offer on that front is anecdotal data - it's very conclusive for us, given it's addressing our real concerns, but less so for others. That said, ICE does not support inserts, only batch loads, so the solution had to be engineered to use that, which added some complexity, but brought orders of magnitude more performance. A simple benchmark run showed that the end-to-end performance for this exceeded 100,000 events per second when running all parts of the client-logserver-database chain on a single desktop machine.

Query performance depends on the queries made. Summary data is 2-3 orders of magnitude faster to access, the bigger the dataset, the bigger the performance benefit - but expecting that for single row accesses would disappoint badly. Storage compression varies wildly depending on the data in question -- we've seen up to 15:1 compression on some real-world data sets, but others (such as storing email addresses in a varchar column) actually expand on storage. This is why I think of this as a solution for structured, quantified event logs, not for general unstructured log file storage.

Friday 5 September 2008

The sweet spot in free-to-play, pay-for-stuff market

I've been talking recently about a few particularities in the business models based on end-user micropayments that have created lots of followup discussion and questions. So much, in fact, that I decided it's time to try to explain one crucial and somewhat counter-intuitive detail in writing for later reference.

First, a bit of background: this information is based on my work with Habbo over the last 5 years, and is half learned from experience, half based on theoretical models built from that experience. I'm sharing this with the world because while it's been an interesting ride to build an online social game with an end-user business model, breaking pretty much every conventional rule in the process ("games have to have objectives", "there is no profit in micropayments", and so on), it's still better for our business if people understand why it works. If this allows a competitor to fix a problem in their product and get off the ground, so be it - there's plenty of growth to go around here, and failures don't help anyone. As a disclaimer, the numbers I'm discussing here have no relation to Habbo, though the basic observations certainly apply.

Let's start with an obvious statement and follow it up with something less obvious: Everyone wants to maximize revenue per player. However, in a free-to-play environment, where the majority of players do not contribute direct revenue, the right tool for the job is not to try to extract the maximum amount of money from those who do pay - rather, to increase the number of players buying anything at all - even if it's just $1 over their entire lifetime. In other words, it's good to have a lot of very low individual value players.

To explain it in detail, lets look at two assumptions behind a flexible pricing business model: first, that the number of customers grows as the cost of goods drops, and second, that the maximum consumption is unrelated to the minimum. There is no average customer who would spend more than half of others, and less than half of the rest. If there were, the picture of that customer base would look something like the image here, and it's pretty strange looking, wouldn't you say? You've probably seen pictures resembling this one where they don't start from the dominating $0 value point - that's the normal distribution.

The first assumption really is very simple: more people are willing to buy a product at a lower price. This is true for most goods, with some notable exceptions in the luxury goods market, where the perception and desirability of a product goes up with its price. However, it is difficult to create a mass-market luxury item, and those do tend to be cheap (and small).

The second is perhaps slightly more involved especially if one is used to thinking of fixed-price models such as one-time purchase of a boxed product or monthly subscriptions, both of which are difficult to scale up on a revenue per customer basis, so scaling them down is highly undesirable as well. However, it's more clear, if not obvious, by looking at other consumer goods - whether tangible such as drink- and foodstuff or intangible like movies, music and other entertainment. Buying these once certainly does not exclude further sales of the same product to the same customer - rather, it's a strong indicator of sales potential!

The free-to-play, pay-for-stuff model follows both of these assumptions. Cheap purchase price attracts more customers out of the existing free users, and transactional item-based sales allows repeat purchases of theoretically unlimited amount. Those who are willing to buy more will do so, up to some practical maximum of consumable goods and discretionary spending.

In this environment, focusing on higher-paying customers makes sense only if the number of customers drops by less than half when the revenue per customer doubles. Again, with the exception of some luxury goods segments, this rarely happens. Think about it: how many chocolate bars of standard quality would you expect to sell for $1? How about for $2? More or less than half? How about for $10 for the exact same package? I'd wager chocolate bars sell at least 10x better at the price of $1 than at the price of $10 each, and the increase of customer base more than covers the lower per-unit revenue.

This is a simple exhibit of power-law market dynamics, and is easiest observed when looked at through a logarithmic chart. Readers of books like The Long Tail or Critical Mass should not be surprised. There's a twist through - because this starts from zero gains (at the free players), the exponential behaviour follows a different path in the beginning. This model also turns Pareto's Law on its head - due to the (in my experience) relatively high exponent, the highest total value is at the lowest end of the spending.

Now, of course there is a minimum profitable price for a bar of chocolate that does not become near-$0 even at very high volumes, unlike purely digital products, so increasing chocolate-sales revenue by dropping prices does not necessarily increase profits, and I'm completely ignoring the effects of packaging and marketing on the perceived value of items. For digital sales, where packaging is more flexible and material costs are effectively non-existent, we still have to consider not-unsubstantial fixed development costs, a certain amount of costs associated to servers and bandwidth, some transaction-related pricing friction, and so forth, but certainly the minimum value (and price) of one unit of digital sales can be driven much lower than a bar of chocolate.

Monday 11 August 2008

Leipzig GCDC next week

So, I've been working all evening on my presentation for next week's Leipzig Games Convention. Its title is the same as the one I had in ION last spring, but I keep digging up this new interesting stuff that just forces me to rewrite everything. I'd never make a great consultant because I wouldn't be happy reusing what I did for the last client. I guess that's why I love the product world instead :)

Anyhow - I'm bummed because I was planning to stay in Leipzig for the entire conference and first day of the expo to meet people and learn new stuff. Instead, it turns out I need to be back in Helsinki on Tuesday, so it's going to be an in-talk-out type of experience - precisely the kind of thing that does not make any sense for conferencing. Can't help it this time, though. If you're at GCDC on Monday though - let me know, would love to make the best of the time available and maximise the people to meet anyway.

Friday 16 May 2008

ION 08 wrap-up

Finished my own lecture, Reinventing Habbo a couple of hours ago to a half-full audience. Flew through 174 slides in a bit over 30 minutes. Update: I have now made a shorter version of the deck that works better as a reference, below.

Read this doc on Scribd: Reinventing Habbo

The follow-up round table probably was titled very badly, since we got nearly no attendees. Interesting discussion though, small groups get deeper in.

Apart from that, I have a collection of 40-50 business cards from interesting new acquaintances, and several very good discussions I'm looking forward to continuing and/or applying to what we're doing. All in all, good experience. Still, I'm looking forward to getting back home too.

2nd update: Adam Martin has coverage of the event, including notes from one of the panels I was on.

- page 1 of 2