Fishpool

Smartphone platforms comparison - a developer perspective

Osma — Tue, 09 Mar 2010 23:12:00 +0200

Having for years used Nokia phones, lately S60 phones of various generations, with a fair amount of experience of the iPhone/iPod Touch OS and lately having used both the Nokia N900 (Maemo OS) and the Google Nexus One (Android) devices as well, I can't avoid comparing these together. As a developer, I'm not really that interested in what they look like today, because today's devices are not what a developer needs to target for applications - rather, what can one determine of the platforms' future from looking at their past?

I can't make any comparisons to the Palm Pre (WebOS), Windows Mobile or Blackberry devices, since I have no first-hand experience of any of them. However, of the four platforms I know to some degree, not only is iPhone still clearly in the lead, but it looks to have the most predictable future as well. iPhone OS 3.1 no longer misses any significant functionality and has gained all the important bits without giving in on the level of platform polish, and the application market is humongous. Its only real weakness is the draconian control Apple enforces, and the crazy restrictions that results in. Those issues are well documented by a recent EFF post outlining the contents of the iPhone developer program contract.

The imminent launch of iPad is the first time the platform starts to experience any kind of real fragmentation in terms of a application development target. At this point that fragmentation looks like to be minimal - with the iPhone, iPod Touch and iPad all sharing the same OS, same UI, practically same inputs and outputs and differing only by what networks are available for communication and what size the screen is, developers are not going to have a hard time at all in developing for all three devices.

That is quite unlike the situation on the other three platforms (Symbian, Android, Maemo). The fragmentation of the Symbian market is a matter of some notoriety. Basically, the same app will not work on phone models launched 9 months apart, or sometimes even on simultaneously launched devices, due to differences in the OS, let alone differences in the form factor, screen size, input mechanisms, and so on. With already two major revisions announced, this trend is only going to continue, and the base OS is already nearing 15 years old, if traced back to the first 32 bit EPOC it evolved from, though I believe the first S60 UI version came out in 2002.

Android is beginning to suffer from the same disease. Not only are the devices on the market each a running different base OS version with different features available to applications, but nearly all of them are also customized by their manufacturers or network carriers with little regard to compatibility (nor in fact could they have any regard for it, since none of them have any previous experience maintaining a platform). And of course each one has a different form factor. However, the most surprising feature of the platform (as a recent Nexus One user) is that even though Android is barely two years old, it already carries with itself a legacy of inconsistent UI controls. What exactly does one do with an indirect-control pointing device (a trackball) on a device capable of direct control via both a touchscreen and motion sensors? Why are the built-in applications (never mind those available on the Android Market) full of menus, "select an object and execute a function on it via a separate control" type UIs clearly inheriting baggage from the decade before touch screens, and other clunky hacks, when there's a rich base to copy from in the iPhone UI design library of 150,000 applications?

So, what about Maemo? I bought a few years ago the very first Maemo device, the N770 Internet Tablet. I've seen and played with every device since. All of them up to the N900 carried the same "windows and menus" baggage Android is suffering from, but the refreshed UI in N900 got rid of most of that. Not entirely so, but enough that I can state with confidence that the N900 UI is more modern, more designed for the touch screen than Android is. However, Maemo's weakness is that of a platform - there's none. Every version of the OS thus far (five iterations on the market) has broken compatibility with the previous. Now, that's to be expected and somewhat forgivable as long as it's in developers-only mode, essentially being beta tested. It's hard to call N900 a beta test any longer. What's worse, is that Nokia has publicly stated that the next device, whatever it's name, and regardless of whether its OS is called Maemo 6 or MeeGo whatever, is also going to be incompatible with the current one, and applications will require a re-write. This is no way to build a developer base.

So, what do we have to look towards to as application developers, trying to figure out what platform to target when working on our next mobile applications?

iPhone, a consistent, easy to use platform with a stable technical roadmap and little legacy baggage, but saddled with an unpredictable owner who's just as likely to deny you from doing business at all than to support you in it?

Symbian, full of legacy, and with a refreshed, incompatible platform to launch maybe next year?

Android, fast-growing, but already full of clunky hacks, and fragmenting faster than than anyone's seen before?

Or Maemo, approaching a state of polish but unable to maintain direction for the length of one device cycle?

I think we're all going to miss the days of Java mobile games development before this is over.

First thoughts about Balancion

Osma — Sun, 17 Jan 2010 13:15:00 +0200

I got an invite to the Balancion personal finance application beta a week ago, and have played with it somewhat since. I've tried a few similar tools before, ranging from the finance packages of the banks I've been a customer of, to a few desktop applications. Until now, I haven't been sufficiently impressed by them to continue using any for any significant period, but I think Balancion might be one to stick around for a while.

Balancion solves the two issues my previous experiments have failed at: first, it covers the entirety of my personal accounts (or very close thereof), because it isn't limited to just the services offered by one bank (the failing of Nordea's, Sampo's and OP's packages, at least the last time I tried them), and second, it doesn't force me to spend my evenings manually typing in boring details, thanks to its tools for downloading the data from the banks and other institutions. Of course, that's just table stakes for the game, really, but my previous experiences have shown even that much is not a given in a market the size of Finland. I would imagine larger market areas have had more focus on this type of stuff - American banks seem to advertise compatibility with Quicken or MS Money - or now with Mint, the hottest entry in the area. German banks apparently have a standard for transaction data exchange. None of that has been available to individuals in Finland.

What currently lifts Balancion above the table-stakes minimum is how it deals with "uncategorized" expenses. Other tools allow searching for similar historic transactions and categorizing all of them at once. Balancion applies that to the future as well, and learns to recognise more and more stuff as you go. Setting the books up for the first time does require a few hours of clicking around, but it gets less and less manual as time goes. That's what makes it a joy to use (as much as any financial application can be a joy, that is!)

At this point in the beta, it's a bit limited; just tracking income and expenses, plus a few (quite useful and informative) visualizations of the same, which already can be helpful in recognizing big expense areas and saving money. However, I'm looking forward to seeing more of the budgeting, expense management and investing tools in the service. It's pretty clear how this can develop and where the opportunities for the business lie. The crucial question is, how can Balancion add partnerships and cross-sell features while retaining the trust of the users. Thus far, their communication indicates they understand how important this will be to their success.

I'm not terribly happy about the way Balancion authenticates me, though. The email/password login is standard, though I'd prefer to use OpenID to avoid managing one more password. What really bugs me are the mandatory "security questions", which they require to be able to change the password. Such questions, especially since they were limited to two out of half a dozen pre-selected questions only reduce security (seriously, it does not take much investigating to figure out the maiden name of my mother). If this is what their security advisor Nixu truly has recommended to the team, I'm disappointed in Nixu as well. Anyway, I answered the questions with something random - so now I can't change my password at all. This probably was not what they intended.

For anyone interested in this category of services, I would recommend checking out the venture capital pitch presentation of Mint.com, the US equivalent of Balancion. If you want to try out Balancion yourself, ask me for an invite here in the blog comments or by tweeting @osma.

Technology factors to watch during 2010

Osma — Thu, 14 Jan 2010 14:28:00 +0200

Last week I posted a brief review of 2009 here, but didn't go much into predictions for 2010. I won't try to predict anything detailed now either, but here's a few things I think will be interesting to monitor over the year. And no, tablet computing isn't on the list. For fairly obvious reasons, this is focused on areas impacting social games. As a further assist, I've underlined the parts most resembling conclusions or predictions.

Social networks and virtual worlds interoperability

As more and more business transforms to use Internet as a core function, the customers of these businesses are faced with a proliferation of proprietary identification mechanisms that has already gotten out of hand. It is not uncommon today to have to manage 20-30 different userid/password pairs that are in regular use, from banks to e-commerce to social networks. At the same time, identity theft is a growing problem, no doubt in large part because of the minimum-security methods of identification.

Social networks today are a significant contributor to this problem. Each collects and presents information about its users that contribute to the rise of identity theft while having their own authorization mechanisms in a silo of low-trustworthy identification methods. The users, on the other hand, perceive little incentive to manage their passwords in a secure fashion. Account hijacking and impersonation is a major problem area to each vendor. The low trust level of individual account data also leads to a low relative value of owning a large user database.

A technology solution, OpenID is emerging and taking hold in a form of an industry-accepted standard for exchanging identity data between an ID provider and a vendor in need of a verified id for their customer. A few of current backers of the standard in the picture on the right. However, changing the practices of the largest businesses has barely begun and no consumer shift can yet be seen – as is typical for such “undercurrent” trends.

OpenID will allow consumers to use fewer, higher-security ids over the universe of their preferred services, which in turn will allow these services a new level of transparent interoperability in combining data from each other in near-automatic, personalized mash-ups via the APIs each vendor can expose to trusted users with less fear of opening holes for account hijacking.

Browsers vs desktops: what's the target for entertainment software?

Here's a rough sketch of competing technology streams in terms of two primary factors – ease of access versus the rich experience of high-performance software. “Browser wars” are starting again, and with the improved engines behind Safari 4, Firefox 4, IE 8 and Google Chrome, a lot of the kind of functionalitywe're used to thinking belongs to native software or at best browser plugins like Flash, Java or Silverlight will be available straight in the browser. This for sure includes high-performance application code, rich 2D vector and pixel graphics, video streams and access to new information like location-sensing. The plugins will most likely be stronger at 3D graphics and synchronized audio and at advanced input mechanisms like using webcams for gesture-based control. Invariably, especially the new input capabilities will also bring with them new security and privacy concerns which will not be fully resolved within the next 2-3 years.

While 3D as a technology will be available to browser-based applications, this doesn't mean the web will turn to represent everything as a virtual copy of the physical world. Instead, it's best use will be as a tool for accelerating and enhancing other UI and presentation concepts – think iTunes CoverFlow. For social interaction experiences, a 3-degrees-freedom pure 3D representation will remain a confusing solution, and other presentations such as axonometric “camera in the corner” concepts will remain more accessible. Naturally, they can (but don't necessarily need to) be rendered using 3D tech.

Increased computing capabilities will change economies of scale

The history of the “computer revolution” has been about automation changing economies of scale to enable entirely new types of business. Lately we've seen this eg by Google AdWords enabling small businesses to advertise and/or publish ads without marketing departments or involvement of agencies.

The same trend is continuing in the form of computing capacity becoming a utility in Cloud Computing, extreme amounts of storage becoming available in costs which allow terabytes of storage to organizations of almost any size and budget, and most importantly, developing data mining, search and discovery algorithms that enable organizations to utilize data which used to be impossible to analyze as automated business practices. Unfortunately, the same capabilities are available for criminals as well.

Areas in which this is happening as we speak:

further types and spread of self-service advertising, better targeting, availability of media
automated heuristics-based detection of risky customers, automated moderation
computer-vision based user interfaces which require nothing more than a webcam
ever increasing size of botnets, and the use of them for game exploits, money laundering, identity theft and surveillance

The escalation of large-scale threats have raised the need for industry-wide groups for exchanging information and best practices between organizations regarding the security relevant information such as new threats, customer risk rating, identification of targeted and organized crime.

Software development, efficiencies, bottlenecks, resources

Commercial software development tools and methods experience a significant shift roughly once every decade. The last such shift was the mainstreaming of RAD/IDE-based, virtual-machine oriented tools and the rise of Web and open source in the 90s, and now those two rising themes are increasingly mainstream while “convergent”, cross-platform applications which depend on the availability of always-on Internet are emerging. As before, it's not driven by technological possibility, but by the richness and availability of high-quality development tools with which more than just the “rocket-scientist” superstars can create new applications.

The skills which are going to be in short supply are those for designing applications which can smoothly interface to the rest of the cloud of applications in this emerging category. Web-accessible APIs, the security design of those APIs, efficient utilization of services from non-associated, even competing companies, and friction-free interfaces for end users of these web-native applications is the challenge.

In this world, the traditional IT outsourcing houses won't be able to serve as a safety valve for resources as they're necessarily still focused on serving the last and current mainstream. In their place, we must consider the availability of open source solutions not just as a method for reducing licensing cost, but as the “extra developer” used to reduce time-to-market. And as with any such relationship, it must be nurtured. In the case of open source, that requires participation and contribution back to the further development of that enabling infrastructure as the cost of outsourcing the majority of the work to the community.

Mobile internet

With the launch of iPhone, the use of Web content and 3rd party applications on mobile devices has multiplied compared to previous smart phone generations. This is due to two factors: the familiarity and productivity of Apple's developer tools for the iPhone, and the straightforward App Store for the end-users. Moreover, the wide base of the applications is primarily because of the former, as proven by the wide availability of unauthorized applications already before the launch of iPhone 2.0 and the App Store. Nokia's failure to create such an applications market despite the functionality available on S60 phones for years before the iPhone launch proves this – it was not the features of the device, but the development tools and application distribution platform were the primary factor.

The launch of Google's Android will further accelerate this development. Current Android-based devices lack the polish of iPhone, and the stability gained from years of experience of Nokia devices, yet the availability of development tools will supercharge this market, and the next couple of years will see accelerated development and polish cycle from all parties. At the moment, it's impossible to call the winner on this race, though.

Happy 2010 - it's review time

Osma — Mon, 04 Jan 2010 20:05:00 +0200

I was happily snowboarding and skiing (the latter for the first time in two decades) last week, so here comes the year-end review a week late. Last year, I harped on Facebook's closed nature, and over the the year they've tried to open more of the users' data over to the Internet. Still, there are no decent APIs for a user to pull out everything they've posted to Facebook to have their own copy, though. That doesn't seem to stop them from dominating the Internet for the time being, though, so good for them.

I'm trying to think of what would have surprised me over the year, but given I failed to make many accurate predictions myself, things just seemed to happen in pretty natural direction. Oracle's Sun acquisition over in April was a bit of a surprise at the time, but since then, I've grown to appreciate how it might make sense for Oracle. However, what still baffles me is that EC is going along with Monty's campaign of blocking the completion of that acquisition. Look, guys - the entire world does not need to agree on a commercial transaction in order for one to go through! MySQL is not the important thing here overall, Java is.

We managed to complete a few of major transitions for Habbo, most notably replacing the Shockwave client which was getting a bit long in the tooth with an all-new Flash-based Habbo Hotel and integrating Habbo with Facebook and other social networks. I didn't write about either of those launches here at the time, but these are pretty huge things for us because they make approaching Habbo much easier for a new user, and enable us to create all kinds of interesting features that would not have made sense previously.

So, what do I expect from 2010? Well, did the mobile Internet already happen? If not, at least it has a fighting chance this year. I'm having a hard time identifying any people close to me who're not using some Internet services on their phone by now, and some seem to be doing that almost exclusively on a phone. That must mean the rest of the world is close on their heels. As for more predictions, others have taken care of them by now.

One promise I can make is to try to do my part in making the Internet more fun and more social. At least now that even newspapers are beginning to think that asking their readers for money is not just a utopia, we can focus on the apps themselves, not whether they're ad-supportable.

Have a great year MMX!

The balance of great products and rapid evolution

Osma — Tue, 08 Dec 2009 20:03:00 +0200

I wanted to link to Andrew Chen's recent article, Does every startup need a Steve Jobs, because it's a useful discussion about the differences between technical feasibility, design-led desirability, and business viability, a triplet productized by IDEO, and one that we also identified a decade ago at Razorfish (wow, it's really a decade ago!). As for the question in the title - no, I don't think that's the only way to create greatness, though clearly if you have Steve at your disposal, you could do worse than have him run everything. :)

Seriously though, at least in this consumer-targeted software business that I'm familiar with, it's crucially important to have those three principles well represented at every level of the business. Sure, it's possible to create a successful business with just two or perhaps even just one of those viewpoints, especially if you can get away with copying someone else but just doing it with better economics. However, to create something new and be successful, it'd be a mistake to ignore business, technology or design. Sadly, of the three, design is the one most commonly ignored. According to this interview of Ken Auletta of his new book, even Google ignores it, Marissa Mayer notwithstanding.

Anyway, what I'm particularly interested about is how can you marry great design with incremental, rapid iterations on the market. I'm pretty sure I've understood how to iterate out in the open with regards to technical work, and relatively comfortable with iteration regarding business aspects. I've yet to come up with a really convincing argument for iteration and design on a very granular basis - well, I can convince myself, but I'm having less success convincing designers. :) If anyone cares to share their secrets, I'd love to hear more.

Notes about Fedora 12

Osma — Mon, 23 Nov 2009 20:02:00 +0200

Another six months, another Fedora release. Apparently I still couldn't resist the temptation of upgrading, given I got a few days of flu-related downtime. Happy to report it's a pretty smooth release, with most things in the expected places:

GNOME is a tiny bit cleaner than it used to be, which is as expected, given that's what it's been doing for the last 5 releases. Apparently next time it'll be something completely different. I don't know if I should be excited or apprehensive about that..
PulseAudio continues to improve - however, I could swear I've successfully used a Bluetooth headset with Skype earlier, and now audio gets stuck if I pair a headset. That's not the most typical use case, of course, and for the most part, audio no longer sucks on Linux. Too bad my laptop's built-in microphone does suck (don't know if that's with Linux or in general), so I do need a headset to make Skype calls.
Apparently Empathy is approaching a usable IM now that it's made the default. Still slightly prematurely, IMO, and I will continue to use Pidgin with all its warts for the time being.
OpenOffice still works as expected, which is to say, slowly, but reasonably predicably.
I can get rid of many of the hacks I've done to make multihead work as I like without setting it up every time, because now Xorg does that by default. Yippee!
Evolution still continues to gain one or two major regressions per release, and lose none of the earlier. The tally now seems to be: brken live search, fkdup IMAP sync, scrwy calendaring, and, as an additional feature, automatically selecting the wrong recipient address out of several available emails despite being repeatedly told otherwise. Seriously, the thing needs to be taken behind the shed and shot to the head. And I need to find a decent email program. Thunderbird 2 wasn't that - and 3 still isn't done. Sigh.
Google Chromium is about 10x faster than Firefox, and by far the easiest way to install a 32 bit browser (working Flash!) on a 64 bit OS (I should probably reinstall to 32 bits all around, this bits thing doesn't help me do anything better).

That concludes my "yes, I'm a Linux geek" postings for the next six months, I guess. :)

MySQL - could we please move on already?

Osma — Wed, 11 Nov 2009 14:10:00 +0200

I've kept away from this debate since last April, but this eternal dragging-on is getting to me. Could we please move on already regarding the Oracle-Sun-MySQL decision? I'm a customer of MySQL, and I don't really savor the idea of becoming a customer of Oracle. Even so, I'd much rather see Oracle own it, than leave it straggling, let alone see this process drag on and on. This is helping no one.

I'm using a product from a company from which I buy commercial support, but I could switch to using a binary-compatible Open Source tool any day I chose. I am not bound to remaining a customer of the company I'm buying support from for any period longer than the current contract. I can definitely live with that obligation. I can live with the OSS-tool (whether we want to call it MySQL Community, Percona, MariaDB or whatever, I don't care) instead of the commercial product - in fact, I'm getting the understanding that the OSS-tool may in fact be better suited to my requirements than the product. So, I have no issue being bound Oracle, should the merger go through, because I am not bound to them. I can see as much interesting related technology being developed outside the discussed commercial unit as inside it, so I'm certainly not worried about the future of the tech.

At this point in time, I could buy support from at least a couple of different organizations to replace and extend that which I've bought from MySQL/Sun. I have absolutely no reason to think that option would go away should the merger be approved, despite what certain founders now claim. If it's not commercially possible to develop and support a database product without being in full control over its copyright, then how come Percona has a business? If it's possible to provide such support for GPL software on a limited basis, but not on a big-business enterprise level, then how come Red Hat is a successful public company?

I use MySQL as an infrastructure component to run a business which could be described as software-as-a-service. I do not redistribute the code base as part of a licensed product. There are companies who do that, but they've always done it with the full understanding that what they're doing is dependent on having to license something from an independent party over which they have no control. If they don't like licensing from Oracle, then they can choose to re-engineer their solution to work on top of some other database engine. It's not like those don't exist, or like technology, licensed or not, hasn't always carried that risk with it.

I can't avoid thinking that some of the parties keeping this thing from reaching completion are dreaming of Skype -- selling the same business twice. Hey, more power to them if that happens, but frankly, that was dependent on Ebay making a stupid deal at the time. I just do not see what that has to do with anti-trust and why the European Commission needs to be involved. THIS is hurting the market, more so that Oracle is likely to.

I have nothing further on the matter. Thank you for your attention.

Nice going for Bobba Bar -

Osma — Sun, 08 Nov 2009 11:02:00 +0200

What a fun end of the week! Bobba Bar, our virtual world app for mobiles just launched this Thursday on iPhone having been available (in a slightly different form) for Nokia S60 devices for a few months. Four days later it's (at least in Finland where I'm looking at it) the top social networking app (ahead of Facebook, Skype and others) and the 3rd highest ranked free app in general. Great launch performance! Get yours from Bobba.com, or if you don't have a compatible phone, check out the Facebook group for Bobba Bar. Oh, and post here or in Facebook where you see Bobba in your local App Store listings!

Creating your very own Habbo site

Osma — Sun, 01 Nov 2009 15:20:00 +0200

So, I mentioned an idea of relaunching Mobiles Disco in celebration of 10 years of Habbo history. It's actually super-easy to do so. We've created a similar space in Habbo, and with the embed functionality launched last month, setting up a web site with your own rooms is no trouble at all, and the only tools required are just two browser windows while you're working on it. Here's how..

Come up with a theme. I'm just going to use the existing Old Skool rooms, but go wild with your own idea!

Create the rooms in Habbo. Again, I got mine pre-created, but since you can do pretty much anything with your own Habbo spaces, the sky's the limit to what can be done. And with Teleport furni, you can create a complete world if you want to...

Go back to the first room you want start your site from. You'll see a small window at the bottom left corner with instructions for "embed this room". Copy the code from that window, and paste it somewhere for safe-keeping for a moment.

Next, you'll set up the web site for your own Habbo to live on. I'm going to use the Google Sites creator for this particular effort, since it's pretty easy to use. You'll need a name for the site, and decide on, or make your own background theme. I went with a retro theme for this particular one..

The Site will begin with just one page on it. Edit the page, and go into "source code" mode. In Google Sites, this is the "HTML" button at the right end of the toolbar. Paste the embed code you copied from Habbo here and save the page.

Google Sites editor will turn the code into a Google Gadget. Other site creators may work slightly differently. The Gadget is easy to tweak though, so this is pretty nice. Clicking on the gadget pops up a small edit menu, where I can choose to center the gadget, and click on Properties to make it a bit larger (960x720 instead of 800x540 to use my screen better), and turn off the title.

That's it! This simple creation took 10 minutes to do, and 30 to write the instructions for. I'm sure you can do a better one, though. Tweet me about yours, and I'll come and check it out!

Habbo, soon ten years old, still the leader

Osma — Sun, 01 Nov 2009 13:59:00 +0200

According to Nic Mitham at KZero, Habbo was the fastest growing virtual world for another quarter. Thanks, Nic! One thing though:

Although it’s the Grand-Daddy of the sector, Habbo continues to show dominance in the virtual worlds sector from a user acquisition perspective

Grand-daddy? Sheesh, Habbo hasn't even got to its pre-teen years, let alone have kids! Lets talk about that another ten years later :) Seriously though, Mobiles Disco, which preceded Habbo by a few months is now ten years old. I'm thinking we should relaunch the Disco..

Some scaling observations on Infobright

Osma — Sat, 03 Oct 2009 11:12:00 +0300

A couple of days ago, Baron Schwartz posted some simple load and select benchmarking of MyISAM, Infobright and MonetDB, which Vadim Tkachenko followed up with a more realistic dataset and interesting figures where MonetDB beat Infobright in most queries.

Used to the parallel IEE loader, I was surprised by the apparent slow loading speed of Baron's benchmark and decided to try and replicate it. I installed Infobright 3.2 on my laptop (see, this is very unscientific) and wrote a simple perl script to generate and load an arbitrarily large data set resembling Baron's description. I'm not going to post my exact numbers, because this installation is severely resource-constrained below Infobright's recommended smallest installation. However, you can reproduce the results yourself with the attached script, and I will note some observations.

First of all, this was run on a 1.8GHz Core 2 Duo 2GB RAM laptop running a 64-bit kernel and 64-bit ICE. I stopped most other programs for the duration of the test, but was still running Fedora 11's GNOME Desktop, and gave ICE only 400MB main heap, 200MB compressed heap and 300MB loader heap. What I found:

Loading speed is almost a linear function of table width. Every time I doubled the number of (random integer) columns in the loaded table, loading speed approximately halved. At 200 columns, I was seeing approximately the same speed as Baron.
On the other hand, loading speed is NOT affected by the number of rows loaded, from which one could assume that it won't be affected by the size of the pre-existing table either (though I did not test this).
SELECT speed is not affected by the number of columns in the table, unless those columns are being selected or used in the where constraint. The same select executed in the same speed regardless of whether the source table had 10, 100 or 500 columns which were not being inspected.
Loading order, column sorting and select constraints are strongly correlated. Limiting a query with a random-value column causes a query which reads 10%, 50% or 90% of the table to execute in approximately 1x, 2x and 3x time. On the other hand, replacing the random-value column in the where constraint with a sorted-value (in load order) column makes queries accessing the same 10%, 50% or 90% of the rows run in nearly constant time. This is the rough set "knowledge grid" in action, but only works if the aggregations done are sum(), min(), max() or other simple functions supported by the grid.
With large dataset (I stopped at 50x heap space), the constant scale factor in the previous query starts to deteriorate, as the entire knowledge grid no longer fits in heap, and inspecting 90% of it vs 10% will require I/O. At this point the scale is beginning to resemble a realistic production setting, as few people are able to host even 5% of their ADBMS working set in-memory.
So this is where things get really interesting, and performance characteristics shift around. Random-column constrained queries of 10%, 50% and 90% of rows now run at 1x, 1.1x and 1.3x time, while constant-column constraints execute at 1x, 1.3x and 3x their respective performance, but approximately 3000x faster than the random-constraints!
The last point shows why it really matters to Infobright workloads that the most frequent queries are taken into account when deciding the load order for the data set. Even when designing an incremental ETL process, it can pay off immensely to pre-sort the incremental data sets by the most likely constraint or group by columns to allow blocks of data (64k rows each) to be included or excluded for particular query plans.
Why is the random-column constrained queries executing in almost constant time regardless of the rows inspected? Because Infobright's columnar datapack engine accesses 64k rows at a time, it's nearly as expensive to access every tenth row as it is to access nine rows out of every ten, if the distribution of those rows is even across the data set. On the other hand, if the 10% of rows needed are clustered together, then the other 90% of the data set is skipped very early in the query optimization process. A traditional btree-indexed data set would still require a random sweep over most of the index, which would be much larger than Infobright's "knowledge grid" is.

That's it for now. If you decide to run your own test using the script, please post comments. It should run as-is on a machine with the unmodified ICE 3.2 installation and basic Perl packages available. The script takes two arguments: number of columns (each a random integer value), and number of rows, and generates the data set into a named pipe though which it's loaded into a local table created automatically.

A peek under the hood in Infobright 3.2 storage engine

Osma — Mon, 21 Sep 2009 15:50:00 +0300

I've been meaning to post some real-world data on the performance of the Infobright 3.2 release which happened a few weeks ago after an extended release candidate period. We're just preparing our upgrades now, so I don't have any performance notes over significant data sets or complicated queries to post quite yet.

To make up for that, I decided to address a particular annoyance of mine in the community edition, first because it hadn't been addressed in the 3.2 release (and really, I'm hoping doing this would include it into 3.2.1), and second, simply because the engine being open source means I can. I feel being OSS is one of Infobright's biggest strengths, in addition to being a pretty amazing piece of performance for such a simple, undemanding package in general, and not making use of that would be shame. Read on for details.

The annoyance? It's pretty difficult to tell, as a user, what the engine is doing while it's running queries. EXPLAIN isn't hooked up and falls back to the general MySQL code path (which, due to the storage engine not exporting any index information, simply thinks any query will be a full table scan). SHOW PROCESSLIST status data on every query simply says "init" for the entire duration of the queries, which could be minutes at a time. It does write quite a lot of detailed information into an optional debug log, but that's on the database server, inaccessible to the user and application, as well as being rather hard to read.

Fortunately, the existence of those debug statements meant it was very easy to find the places into which I could insert some status instrumentation for the process list. This is certainly not perfect - this doesn't help telling about execution paths before running a query, and the convention for process list status is far more terse than what the debug output of the engine could produce. I could have simply copied the same detail level into the process list, but that doesn't seem to be the norm in MySQL engines, and assuming that Infobright will later include the SHOW PROFILES feature, would not be helpful anyway.

The patch is below (or download it as raw text), and it applies on top of 3.2 src package downloadable at the Infobright.org site. Builds with 'make EDITION=community release' and works for me, but use this at your own risk. Please do post notes and comments, though, I'd be interested to hear about other users. I'm sure the patch could be much improved, too.

Now, what would be really interesting was if the debug log's information of the knowledge grid evaluation could be turned into EXPLAIN output, but that would require more understanding of MySQL internals than what I have...

This was the first time I looked at the source code for Infobright, and the second or third time I did so for MySQL in general. ICE is pretty impressive also in its techniques, not only being the only integrated columnar engine, but also having more join strategies than other engines I've used, and so forth. The code is tough to follow though, and the source package included a huge amount of unused stuff, like a copy of both the InnoDB and NDB storage engines, neither of which is built from the code base. I guess a bit of clean-up would make this somewhat more approachable..

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerGeneral.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerGeneral.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerGeneral.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerGeneral.cpp  
 2009-09-18 12:13:23.001506795 +0300

@@ -26,6 +26,7 @@

         mind->Empty();

         return no_desc;  
         // all done

     }

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"joining");

     //if(desc[0].val1.vc->IsConst() &&
desc[0].val2.vc == NULL) {

     //    // Special case: if there is a
chance for one-dimensional filtering, execute one condition only.

     //    no_desc = 1;

@@ -48,7 +49,8 @@ int JoinerGeneral::ExecuteJoinConditions

        bool loc_result;

        bool stop_execution =
false;    // early stop for LIMIT

        rccontrol.lock(m_conn.GetThreadID())
<< "Starting joiner loop (" << mit.NoTuples() << " rows)."
<< unlock;

-

+      
thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "in joiner loop");

+      

        // The main loop for checking
conditions

        while(mit.IsValid()) {

               
if(mit.PackrowStarted()) {

diff -ur infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerHash.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerHash.cpp

---
infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerHash.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerHash.cpp  
 2009-09-18 15:31:15.263505407 +0300

@@ -55,6 +55,7 @@

     /////////////////// Prepare all descriptor information
/////////////

     // TODO: prepare a common language for both joined
columns, if not compatible

     bool first_found = true;

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "hash
join");

     DimensionVector dims1(mind);  
     // Initial dimension descriptions

     DimensionVector dims2(mind);

     for(int i = 0; i < desc.size(); i++) {

@@ -203,6 +204,7 @@

 

     _int64 hash_row = 0;      
         // hash_row = 0, otherwise deadlock
for null on the first position

     _int64 traversed_rows = 0;

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "hash
join traverse");

     while(mit.IsValid()) {

         if(m_conn.killed())

             throw
KilledRCException();

@@ -247,6 +249,7 @@

     int no_of_matching_rows;

     MIIterator mit(mind, matched_dims);

     MIDummyIterator combined_mit(mind);  
     // a combined iterator for checking non-hashed
conditions, if any

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "hash
join tuples");

     while(mit.IsValid()) {

         if(m_conn.killed())

             throw
KilledRCException();

diff -ur infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerLoop.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerLoop.cpp

---
infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerLoop.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerLoop.cpp  
 2009-09-18 12:12:46.056444175 +0300

@@ -34,6 +34,8 @@

     int cur_dim1, cur_dim2;

     int attr1, attr2;

 

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "loop
join");

+

     ParseDescriptor( desc[0], cur_t1, cur_t2, cur_dim1,
cur_dim2, attr1, attr2, loc_op );

 

   
 //////////////////////////////////////////////////////////////////////////////////


diff -ur infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerSort.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerSort.cpp

---
infobright-3.2-x86_64src/src/storage/brighthouse/core/JoinerSort.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/JoinerSort.cpp  
 2009-09-18 15:32:01.921256944 +0300

@@ -29,6 +29,7 @@

     VirtualColumn *vc2 = desc[0].val1.vc;

     dim1 = vc1->GetDim();

     dim2 = vc2->GetDim();

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "sort
join");

     // The only supported cases (for now):

     if(dim1 == -1 || dim2 == -1 ||  
         // one-dim only

         mind->GetFilter(dim1) == NULL
||

@@ -128,6 +129,8 @@

     s1.Lock();

     s2.Lock();

 

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "sort
join apply");

+

     MINewContents new_mind(mind);

     new_mind.SetDimension(dim1);

     new_mind.SetDimension(dim2);

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/MIRoughSorter.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MIRoughSorter.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/MIRoughSorter.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MIRoughSorter.cpp  
 2009-09-16 23:17:07.487972507 +0300

@@ -77,6 +77,7 @@

 

     ///////////////////////// the main sorting loop
through bigblocks /////////

     if(sorting_needed)    {

+      
 thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(), "sorting
roughly");

       
 rccontrol.lock(mind->m_conn->GetThreadID()) << "Sorting
roughly multiindex..." << unlock;

         _int64 start_tuple = 0;

         _int64 stop_tuple = 0;

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/MIUpdatingIterator.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MIUpdatingIterator.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/MIUpdatingIterator.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MIUpdatingIterator.cpp  
 2009-09-18 15:12:03.293257577 +0300

@@ -116,6 +116,7 @@

 {

     if(!changed)

         return;

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"commit");

     if(one_dim_filter) {

       
 one_dim_filter->Commit();        //
working directly on multiindex filer (special case)

         mind->UpdateNoTuples();

diff -ur infobright-3.2-x86_64src/src/storage/brighthouse/core/MultiSorter.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MultiSorter.cpp

---
infobright-3.2-x86_64src/src/storage/brighthouse/core/MultiSorter.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/MultiSorter.cpp  
 2009-09-18 11:31:50.202579444 +0300

@@ -372,6 +372,7 @@

     }

     else

       
 rccontrol.lock(m_conn->GetThreadID()) << "Sorting " <<
no_obj << " rows..." << unlock;

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"sorting");

     if(max_rate < cur_rate)

         max_rate = cur_rate;

     int byte_ind = 4;      
     // no. of bytes to encode row index (4 or 8)

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/Query_exeq_low.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/Query_exeq_low.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/Query_exeq_low.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/Query_exeq_low.cpp  
 2009-09-18 11:59:48.065253205 +0300

@@ -553,6 +553,7 @@

   
 ////////////////////////////////////////////////////////////////////////


     if(desc.size() < 1)

         return;

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"preparing");

     DelayWhereConditions(desc);

     SyntacticalDescriptorListPreprocessing(desc, mind,
table);

 

@@ -619,6 +620,8 @@

         return;

     }

 

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"executing");

+    

     ///////////////// Apply all one-dimensional filters
(after where, i.e. without outer joins)

     for(uint i = 0; i < desc.size(); i++)

         if(!desc[i].done &&
desc[i].IsInner() && !desc[i].IsType_Join() &&
!desc[i].IsDelayed()) {

@@ -655,6 +658,7 @@

     }

 

   
 /////////////////////////////////////////////////////////////////////////////////////


+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"joining");

     DescriptorJoinOrdering(desc, mind);

 

     ///// descriptor display for joins

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/Query_optimize_RS.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/Query_optimize_RS.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/Query_optimize_RS.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/Query_optimize_RS.cpp  
 2009-09-18 11:27:55.040256644 +0300

@@ -104,6 +104,7 @@

               
               
   MultiIndex &mind,

               
               
   vector<Descriptor> &desc)

 {

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"evaluating P2P");

     bool is_nonempty = true;

     // init by previous values of mind (if any
nontrivial)

     for(int i = 0; i < mind.NoDimensions(); i++)
{

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/RCEngine_results.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/RCEngine_results.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/RCEngine_results.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/RCEngine_results.cpp  
 2009-09-16 23:02:02.200221840 +0300

@@ -37,7 +37,7 @@

 void RCEngine::SendResults(na::DataSource* exectree, THD* thd,
select_result *res, List<Item> &fields)

 {

     int error = 0;

-    thd->proc_info="Sending data";

+    thd_proc_info(thd,"Sending data");

     DBUG_PRINT("info", ("%s", thd->proc_info));

 

     res->send_fields(fields, Protocol::SEND_NUM_ROWS |
Protocol::SEND_EOF);

@@ -136,7 +136,7 @@

 void RCEngine::SendResults(JustATable& results, THD* thd,
select_result *res, List<Item> &fields, ConnectionInfo *conn)

 {

     int error = 0;

-    thd->proc_info="Sending data";

+    thd_proc_info(thd,"Sending data");

     DBUG_PRINT("info", ("%s", thd->proc_info));

 

     res->send_fields(fields, Protocol::SEND_NUM_ROWS |
Protocol::SEND_EOF);

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/RoughJoinWatcher.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/RoughJoinWatcher.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/RoughJoinWatcher.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/RoughJoinWatcher.cpp  
 2009-09-16 23:18:25.821225774 +0300

@@ -184,6 +184,7 @@

     //        - after
checking all the result:

     //          
 - if still potentially_excluded => set as non-intersecting and up to
date

 

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"updating P2P");

     rccontrol.lock(mind.m_conn->GetThreadID()) <<
"Updating P2P..." << unlock;

     anything_to_update = false;

     _int64 pairs_already_updated = 0;

diff -ur
infobright-3.2-x86_64src/src/storage/brighthouse/core/TempTable_aggregate.cpp
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/TempTable_aggregate.cpp


---
infobright-3.2-x86_64src/src/storage/brighthouse/core/TempTable_aggregate.cpp  
 2009-08-26 21:26:43.000000000 +0300

+++
infobright-3.2-x86_64src.new/src/storage/brighthouse/core/TempTable_aggregate.cpp  
 2009-09-18 11:30:24.149255211 +0300

@@ -65,6 +65,7 @@

     ::Filter tuple_left(mit.NoTuples());

     tuple_left.Set();

     gbw.SetDistinctTuples(tuple_left.NoObj());

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"aggregating");

     do {

         if(rccontrol.isOn())  {

           
 if(upper_approx_of_groups == 1)

@@ -222,6 +223,7 @@

 void TempTable::MultiDimensionalDistinctScan(GroupByWrapper& gbw,
DimensionVector &dims)

 {

   
 MEASURE_FET("TempTable::MultiDimensionalDistinctScan(GroupByWrapper&
gbw)");

+    thd_proc_info(&ConnectionInfoOnTLS.Get().Thd(),
"Distinct scan");

     while(gbw.AnyOmittedByDistinct()) {  
 /////////// any distincts omitted? => another pass needed

         ///// Some displays

         _int64 max_size_for_display =
0;

Why mobile computers are a bad idea

Osma — Fri, 28 Aug 2009 11:03:00 +0300

After my last night's posting, I had a small exchange with @moximilian and @jludwig about my claim that calling the N900 a computer is BS and nobody wants a computer. Somehow, Nokia has gone from calling the N-series devices "multimedia computers" a couple of years back to "mobile computers" today, but it's a totally horrible thing to do from a market positioning point of view. I suppose I should clarify my reasoning a bit.

This is how I imagine the thought process has gone: Nokia, an engineering-led manufacturer of fixed-function devices (phones) has had the ambition to "put the Internet in your pocket" for quite some time. So far, so good. Now, an engineer designs a brilliant package of a high-performance programmable microprocessor, significant amount of working memory and storage memory, and a rich set of input and output mechanisms. To an engineer, this fits the definition of a computer, thus it must be a computer.

However, that is not how the world at large sees computers. The general understanding of a computer is a device which requires constant management, is at risk from viruses and other malware, produces incomprehensible error messages and, despite being a window to the wonderful new world of Twitters, Facebooks and all kinds of information and entertainment, is best left alone when at all possible. Yes, the computer industry has made great progress in the last decades in making their produce more approachable and human-friendly, but it's not there yet. Apple, for all its faults, is generally regarded as the gold standard in "computers for the normal people". Yet who hasn't seen a Mac or even an iPhone (once loaded with applications, at least) bug out in the most bizarre of ways?

Computers don't have any built-in value of their own. The value is completely attached to the applications, services and solutions to which they provide access. If it was left at that, and calling something a "mobile computer" would be simply a bad choice of marketing titles, I wouldn't mind. However, as long as the engineers working on the future devices think it's desirable to think of them as computers, they will carry the problems I mentioned along to future devices, because that's what computers do.

The device in your pocket is a terminal, a window onto the services of the Global Computer, and a flexible access point to things no one has yet to invent. It is programmable, it does have memory, and it can compute. Even so, lets not call it a computer.

Reflections on Nokia Maemo

Osma — Thu, 27 Aug 2009 22:14:00 +0300

Earlier today Nokia announced their first handset based on what is likely to be their mobile operating system of the future - the Nokia N900 Maemo. I didn't think I would bother to pay attention, but somehow, I ended up doing so anyway, and this post is a result of that time spent thinking about it. I like quite a few things about it, but can't avoid being deeply bothered by other aspects. I hope by writing this I can make some small contribution to its future.

Why do I care? After the frustrations and disappointments with Nokia devices in the past years, I've tried not to. However, they're impossible to ignore in Finland, I have family reasons to hope this road leads to something good, and it's an attempt to make an open platform -- and I care about open platforms. Why do I feel qualified to comment? Well, because this is not really about devices, it's about software. And software is what I've always done, and managing software organizations is what I think about daily.

There's lot to like about the N900. I haven't seen, let alone played with one, but as far as the specs go, it's a pretty nice set of hardware. Same performance as the iPhone 3GS (which also makes it faster than any Android device announced), 3D acceleration, lots of storage (and a memory card slot), and, as a welcome change from many other Nokia devices, completely standard connectors (3.5mm audio, micro-USB tethering and battery charger). On the hardware side, the only thing not to like about it is the lack of a finger-usable, multi-touch display. This device, like all the other Nokia devices before it, require a stylus or at least long fingernails. It makes up for that by being really high resolution.

It's also based on an open source, Linux-based operating system Nokia has been developing for several years with community participation, Maemo. This makes it more attractive to me on a personal level than iPhone (which is way too closely guarded and controlled by Apple), Palm WebOS (open, but little track record), or even Android (open, but built out of pieces which have far less common with normal Linux than Maemo). It should be fairly clear that all four mentioned are way ahead of things like Symbian S60, which clearly needs to be taken behind the shed and put out of its misery, not matter what Nokia's representatives say about it official capacity.

On a more professional level, the inclusion of Flash 9.4 in the platform is a big deal. I'm anxious to get hold of one and see how much work it is to make Habbo work on it. This could be the first handset capable of technically running it (enough performance, enough resolution, good enough software), though obviously tuning our service to a mobile device would still need work on UI and other pieces.

However, like I wrote above, there's also plenty that bothers me. First of all, unlike most people probably realize, this is actually the 4th Maemo device in about as many years that Nokia releases. First was the Nokia 770 Internet Tablet, essentially an early adopter test device. Then came N800 -- running an updated OS which required applications to be ported to it, but which never was officially released for the 770 (putting the early adopter developers in a rather awkward position). Less than a year later N810 added a physical keyboard and an OS upgrade (which fortunately could be installed, with some difficulty, on the N800). That was quite a long time ago, though.

In the meantime, Maemo has been completely reinvented. The original UI toolkit has been switched to QT, which Nokia bought in the meantime, and all of the (rather limited quantity) applications require significant rework to be compatible with the OS release on the N900. The public reasoning for this compatibility break has been pretty weak -- "to ensure compatibility with S60", which also is moving on to QT framework. Why is this weak? Well, because the transition over on the S60 side also requires all of the (somewhat more numerous) applications developed for that platform to be significantly reworked. In other words, Nokia broke compatibility on both its old smartphone platform and the new platform at the same time, and offered little transitionary compatibility layers to either side. Not for the first time, either. S60 applications have been broken between upgrades several times before, too.

This track record is highly worrying. Despite their years of practice and ambitions to have a lively third party mobile applications market, Nokia has clearly not grasped the importance of a stable platform to the developers they mean to attract. This lack of understanding of one of the most basic requirements is enough to counter pretty much everything I wrote about Maemo versus its closest competitions a few chapters earlier.

Contrast the above to iPhone OS 2.0 to 3.0 transition. Sure, a few things did change. However, developers were given months of notice ahead of time, and the changes, apart from added functionality, were all pretty minor. Of course, Apple has a long history of making major upgrades while retaining forwards compatibility, with the Mac OS 68k to PowerPC, then to OS X, then to Intel CPU transitions.

It's also taken a LONG time for this device to be announced. I don't know, but I get the feeling it's something like a year late. The break in launch schedule between N810 and N900, the amount of changes in the Maemo platform, and the design of the device compared to for instance the N97 all scream "last year" to me. Besides, everyone knew this was coming ages ago. In the time between the launches of N810 and N900, Apple has managed to update the iPhone twice. This lack of predictability in the release cycle doesn't bode well for the next device in the line.

There is nothing more important for progress in software development than cycle time. The only cost-effective, productive way of making software today is to get feedback on it often, and the longer it stays unreleased, the more the feedback is late when it comes. This seems to be another area where Nokia has not been able to shake off their "we make hardware" mentality. Unline hardware, software can be updated with no extra cost. That's an advantage nearly everyone else has learned to make use of, and Nokia, if they truly desire to become a software and services powerhouse, has to finally take to heart.

N900 is not an "iPhone killer". I don't think it's meant to be. When its development started, it's unlikely the iPhone had even been announced. However, it's the best chance for Nokia to ever develop a device better than an iPhone. I hope they will - the world needs competition, and I would like to see Nokia be part in that. However, at this rate they will never catch up - Apple will have released two more major updates before the next Maemo device unless Nokia gets their act together.

I'm still hoping.

Excuse the downtime

Osma — Thu, 16 Jul 2009 23:33:00 +0300

Mea culpa, a small domain to fix email redirection update seems to have gone wrong and taken the blog offline. I didn't notice since I've been mostly offline myself, so thanks to the readers who notified me. If you can read this, the problem has been resolved.

What we're looking for in a data integration tool

Osma — Wed, 27 May 2009 20:05:00 +0300

As our data warehousing process grows and the workflows get more complex, we've revisited the question of what tools to use in this process. Out of curiosity, I had a look at basing such a process on Hadoop/Hive for scalability reasons, but the lack of mature tools and the sacrifices on efficiency that would entail meant we're better off using something else as long as a distributed processing platform is the only thing that can get the job done. I'm also curious about the transition to continuous integration, a model I noticed showing up a couple of years ago and now getting some air under its wings as CEP, IBM's Infosphere Streams, and other similar approaches. Still, I think I'll continue to rely on something else for a while and see how things shake out. Continuous integration clearly is the future, but there are many ways to get there.

So, we had a look at what's going on in the Open Source data integration field. It seems the leaders in that field are Pentaho with Kettle/Pentaho Data Integration, and Talend with Open Studio and Talend Integration Suite. Both seem pretty even in terms of features. Both companies are a bit difficult to approach as a potential customer, so I figured I should also try what would come up from the OSS approach of just posting my thoughts on the Interweb ;)

Besides the technical pilot implementations we've made to compare basic workflow of the various tools, below is a sample of the kind of questions we're considering when evaluating the suitability of the tools.

Product roadmap, release schedule and size of the development team

How often and of what scope of changes should we expect and prepare ourselves for platform upgrades?
Past track record on keeping to a regular updates schedule

Data lineage and dependency, Impact analysis

How to find out which tables are being used to for deriving DWH dimensions and facts?

Logging, auditing, monitoring on row and job level

How to monitor and archive workflows on a row level (amount of rows being inserted/updated/deleted)?
How to maintain, access and query a job execution history (start time/end time/return code)?

Version control

How to track and restore changes in jobs?

Multi-user environment

How can several developers work together?

Change Data Capture

How to assist incremental loads?

Data profiling

How can data source be examined?

Job recovery

How to recover from possible failures in jobs (such as lost database connection)?

Deploy jobs

How to move jobs from one repository to another (development to testing to production)?

Hello, MySQL 6.0, err, something

Osma — Sun, 24 May 2009 13:28:00 +0300

I'm conflicted about the latest twist of the MySQL release saga, ie the announcement of the 6.0.11 alpha version and the accompanying note that it's the last 6.0 release and will be replaced by the already discussed milestone model. From an engineering point of view, I think this is the right step. I'm not sure about that, because I can't really tell exactly what is the engineering model chosen: trunk-first, then backport, or fix-in-releases, then forward port. I also can't tell whether the milestone model is going to be timeboxed or feature-scoped. Personally, I would prefer to see the former of both alternatives.

From a customer point of view, I'm even more confused, though much less concerned. Okay, so 6.0 won't become the marketing version number of any MySQL Enterprise release? Doesn't matter. 5.4 needs to come out first anyway, preferably sooner with a concrete, well-tested feature set, than later with more planned-but-unfinished features stuffed in it. What the release after that is going to be called makes no difference to me, as long as it's also going to contain solid improvements and comes out on predictable schedule that doesn't force me to look for something drastically different in order to deal with scale.

That being said, it's still weird. So if the thought of 6.0 GA release is scrapped, why release anything and still call it 6.0? I guess it's just tying loose ends, but that's an engineering thing, and only the number of existing source branches with stuff to merge together matter, not the version number put to it...

Confusing Sun communication about MySQL 5.4

Osma — Tue, 12 May 2009 17:01:00 +0300

Just received an email newsletter from Sun titled "MySQL 5.4 Preview Release" which states:

Sun Microsystems recently released MySQL 5.4, delivering performance and scalability improvements enabling the InnoDB storage engine to scale up to 16-way x86 servers and 64-way CMT servers.

MySQL 5.4 also includes new subquery optimizations and JOIN improvements, resulting in 90% better response times for certain queries.

Apparently, the confusion about the contents of the release I wrote about earlier continue to reign inside Sun as well. MySQL 5.4 has not been released by any reasonable meaning of the word, since there's "only" a preview available at this time. Compare this to Windows 7: that's already a Release Candidate, but it has not been released. Also, the preview release available does not include new subquery optimizations nor JOIN improvements. Having planned such improvements doesn't count.

As I wrote earlier, the best of the rather bad excuses for the release labeling offered to me was that Sun wanted to avoid confusion by not releasing many versions at once. I think that got replaced (and then some) by plenty of extra confusion about when and what was released, instead. Sorry, no good. Try again, 'kthxbye.

What does Oracle mean for Java?

Osma — Mon, 04 May 2009 22:54:00 +0300

Over the past two weeks I've been mostly focused on MySQL, but the big-ticket item in the Sun/Oracle deal is not databases, it's Java. However, it's also the domain which is far less clear to predict. It was a big deal when Sun decided to open source Java, but the fact of the matter is that the first fully open source release isn't out yet, and Sun has been keeping the testing and certification kit off-limits for open source communities. This means it would still be far too easy for OpenJDK to be killed off.

I've been keeping clear of Oracle for several years, and can't even begin to guess what their position on this is. Oracle has been a pretty active contributor to Linux in particular for several years, and I'm sure their open source strategy and how it works together with their business is pretty well established within at least the engineering parts of the company. At the same time, their notoriously aggressive market tactics make sure that everyone's wary of their next move. Java is a huge part of Oracle's business, and after they purchased BEA, I wouldn't be surprised if Oracle wasn't already the biggest Java company (in terms of revenue) ahead of both Sun and IBM. After completing the Sun acquisition, that'll be guaranteed.

That's a big balance shift for the overall Java community. Now, Oracle is a smart company. My worry is they might emphasize short-term tactical market advantage (owning all of Java, JRockit, Glassfish and WebLogic to compete against other middleware and business applications) over long-term strategic benefit of a unified platform competing with .NET and the host of open source platforms from PHP and Ruby to Python. With such a wide field, following up on, and improving on the open source platform process would be the right thing to do - and it would help me :)

The difference between conversion and retention

Osma — Thu, 30 Apr 2009 11:27:00 +0300

Picked up a piece of analysis today from my newsfeed regarding Twitter audience. Nielsen has posted information about Twitter's month-to-month retention (40%) and compared that to Facebook's and MySpace's. Pete Cashmore over at Mashable promptly misread the basic information and came to an entirely wrong conclusion about the stats, titling his post about it as "60% quit Twitter in the first month". A simple misunderstanding of basic audience analysis like this is the crucial difference between explosively growing traffic and a failure. That's a fail for you, Pete.

What's wrong? Well, retention is a separate matter from conversion. 40% conversion from a trial registration to being a continuing active user to the second month would not be a bad conversion rate. It's not stratospherically great, I've seen better, but I wouldn't be terribly unhappy about such a figure. However, Nielsen didn't say anything at all about first-to-second month conversion. This is what they DID say: "Twitter’s audience retention rate, or the percentage of a given month’s users who come back the following month, is currently about 40 percent."

That's pretty plain English when you take the time to read it. Month to month, regardless of visitor lifetime, not first to second month. On this metric, 40% retention is not good at all, and will definitely be a limiting factor to Twitter's traffic and audience size over time, just the Nielsen article points out (and shows the math for). For any given retention rate, there just is a certain maximum audience reach beyond which any new traffic can't overcome the leaving base, since new traffic is not an inexhaustible supply.

And since today is a busy day, that concludes the free startup advice. Take the time to understand the difference between these metrics, you'll thank yourself for it later.