Fishpool

While trying to make something out of a boring sick-day, I've been searching for a method to synchronise my Evolution calendar with Nokia 6610 phone. It's amazing how bad Linux calendar tools are..

Gammu, the capable but horrible-to-use and terribly documented phone backup/management program DOES in the end turn out to have a way of importing and exporting iCal format. It's just that it's only mentioned in one mailing list archive I already lost the link to. Anyway, the trick is gammu --backup calendar.vcs, which outputs iCalendar format! Same applies to --restore. This puts a new standard to the word esoteric in my book.

However, my phone will not have memory to deal with the complete calendar in Evolution. So, I'm trying to come up with a way of parsing that and extracting only the last week and a couple of weeks into the future. However, as far as I can see, there are approximately zero point zero tools to do that!

Basically, I only found Perl iCal::Parser, which reads a zero-length hash table from all iCalendar files I have. There's libical, with no tools whatsoever. What the hell? No open source developer anywhere keeps a calendar, do they?

Update: I ended up writing a small a Perl script to filter old calendar entries out of the file to synchronise, and in the process of doing so had to look closer than I would have ever wanted at the internals of Gammu. It's a wonderful program from a device compatibility point of view, but its code structure is even more horrible than its user interface. I particularly like the awful file-reading loop which crashes on pretty much any file above 16KB in size.

Anyway, should someone be interested, the Evolution-to-Gammu calendar parser is in behind the next click...

#!/usr/bin/perl
use POSIX;
my $record = undef;
$calstart = strftime "%Y%m%dT%H%M%s", localtime(time()-7*24*3600);
$calend = strftime "%Y%m%dT%H%M%s", localtime(time()+30*24*3600);
print "BEGIN:VCALENDAR\nVERSION: 1.0\n\n";
while (<>) {
  $_ = "$last$1\n", $record = substr($record,0,length($record)-length($last)-1) if (/^ (.*)$/);
  $date = "$3$4", $_ = "$1:$3$4\n" if (/^(DTSTART)(;.*)?:(\d+)(T\d+)?/ || /^(DTEND)(;.*)?:(\d+)(T\d+)?/ || /^(DUE):(\d+)(T\d+)?/);
  $start = $date if (/^DTSTART/);
  if (/^DTEND/) {
    $record .= "CATEGORIES:MEETING\n" if ($start ne $date);
  }
  $record .= $_ if (/^BEGIN:(VEVENT|VTODO)/ || $record);
  if (/^END:(VEVENT|VTODO)/) {
    print "$record\n" if ($date gt $calstart && $date lt $calend);
    $date = $record = $start = undef;
  }
  chomp;
  $last = $_;
}
print "END:VCALENDAR\n";

I run this with

#!/bin/sh
cal=`mktemp /tmp/calXXXXXXXXX.vcs`
filter-ical <$HOME/.evolution/calendar/local/system/calendar.ics >$cal
echo "ALL" | gammu --restore $cal

Last time, I promised to write something about how all of his works together instead of continuing with the installation notes. While I haven't yet figured out every detail of the whole system, this is what I'm going to do today. Read on...

Components

My system consists of a fairly standard Athlon PC equipped with a TechniSat AirStar 2 TV PCI card giving it DVB-T television reception. This card is capable of receiving free-to-air (unencrypted) channels, but not those which require a Conax card to decode. I'm planning to later add a second card with support for this purpose. Another note about the card is that it's of the budget variety with no MPEG-2 decoding functionalities - this actually makes it a better choice for a system like this, given that any modern CPU will be more than capable enough to perform the decoding in software, especially when assisted by a decent video card.

For the display, I'm using the Sanyo PLV-Z1 video projector I bought last spring.

To go with the hardware, I've installed Fedora Core 3 Linux and the MythTV suite, most easily available via ATrpms, a repository of software maintained by Axel Thimm. The other alternative would be to use KnoppMyth, which in fact contains a complete system installation. I did not use this as I repurposed the computer from our workroom PC, and did not want to reinstall it completely.

The DVB card came with a simple remote control, the configuration of which was included in LIRC, which is what all Linux programs use for remote control support.

Viewing and recording TV

The main function of the system is of course viewing and recording TV broadcast. After spending some time going through various deadends trying to make MythTV understand the DVB driver, I stumbled upon the real solution: Zap2Myth. This program takes a list of channels from the tzap program (which is able to find them automatically), and converts that to MythTV's format as an SQL script which you can insert into the MythTV backend database. The process could have been a lot more clear, and in fact I think MythTV should use tzap directly to handle DVB reception, but eventually I did succeed in the installation.

The results are mostly very satisfying, with excellent image quality, comparable to the best digital TV set-top boxes. Not completely so, however. Trying to switch to a channel which is not currently broadcasting can freeze MythTV, requiring me to either wait for a timeout to occur, or to restart the software. A couple of times this has actually crashed the backend, requiring a restart. I have not been able to get subtitles to work either - this is a problem that affects many DVB-T set-top boxes and is due to the Finnish broadcasts being slightly different than in other (larger) markets. I get the feeling MythTV has not been completely tuned for DVB reception, and these glitches would not appear with analog channels.

As an aside: MythTV consists of two linked parts. The backend is in charge of receiving broadcasts, scheduling recordings and storage, while the frontend provides the on-screen menus and viewing capabilities. This split makes it possible to build a solution where the storage is done by a separate computer hidden in another room, and the living-room frontend is a smaller device. In fact, it is a commonplace solution to connect multiple frontends to the same backend, for viewing the same video archive from multiple rooms for instance. In my case, the backend and frontend are on the same box, although I have additionally configured another frontend on my laptop for casual viewing.

Selecting programs to watch or record is done via an easy-to-use program guide, immediately familiar to anyone who's used a Tivo device or most other set-top boxes. This program guide pulls in program information from public web sites via a program called XMLTV. Its coverage of Finnish TV channels is via the Katso! web site. Using channel information from Telkku.com or Ohjelmat.info would however sometimes give better coverage. For the most part, this is a satisfactory solution anyway.

With one DVB-T card, I can watch or record one live broadcast at a time. In fact, due to the way DVB broadcasts are made, it would be possible to do more, as is shown by another software package the name of VDR. MythTV does have patches for similar functionality, but these have not been integrated to the package yet. However, MythTV will be able to utilise as many reception cards you install, and I plan to later install a second DVB card with pay-channel support via a Conax decode card and an analog TV card for receiving a couple of satellite channels from our building's common satellite dish.

DVD

Another goal of this project was to reduce the height of our hifi stack. We've already eliminated the VCR, and eliminating the DVD player as well should be quite easy.

Again, annoying glitches did make this somewhat more work than I expected. MythTV by default utilises a program called MPlayer to provide it with playback capabilities. MPlayer is a wonderfully versatile package, able to display nearly any formats imaginable, with excellent video quality through its postprocessing capabilities. However, DVD playback is lacking one crucially important feature: support for DVD menus. This resulted in most DVD discs showing nothing but a blank screen.

However, the solution to this was to reconfigure MythTV to use another media player for DVD playback. I used Xine, although I could have chosen Ogle as well. Both of these have very robust DVD support. The lack of "boxed" integration does mean that I still have to additionally reconfigure Xine's keyboard commands to match to MythTV's so that the remote control will work as expected.

Music

VCR and DVD player eliminated, lets turn our sights to the CD changer. Since the computer I'm building this on is our old workroom computer, it already had most of our CD collection on it in Ogg Vorbis format. Vorbis is a format competing against MP3, AAC and others, distinctive by providing high quality in low bitrate and at the same time being completely open source. It's gradually getting more widespread market acceptance, with support in some portable players such as the very cool iAudio M3 (christmas present, anyone?).

MythTV supported this with no effort whatsoever via the MythMusic system. Two wishlist items: it could have automatically found the music collection (like for example Rhythmbox and Zinf, but more importantly, it does not have a very good music browsing/playlist management system. Combined with the limitations of the remote control, this makes choosing music a bit of a job. It's much better than the MP3-CD support of our DVD player, but falls very short of the best desktop media players like iTunes or Rhythmbox.

Other functionalities

Now, we are talking about a PC, so the story certainly does not end here. We have the following:

Commercial autoskip: On the second day after the installation of MythTV, it already started to notify us while viewing recorded programs that a commercial break is starting in a couple of seconds, offering to skip it completely. This is a wonderful feature, and I can not really criticise it of not always being 100% correct, as for the most part it does work very well, and perfectly enhances the fast-forward and rewind functions you would expect to have in a digital media center.

Editing recordings: Sometimes you want to archive shows for later viewing, such as recently the last couple of episodes of Sopranos before its Finnish broadcast break, so that we can watch them again before the show returns. This is made perfect by the editing capabilities, allowing us to remove commercials completely from the archived version (which of course can also be re-encoded to DIVX format and burnt to CD-R for offline archiving).

Weather forecast: After telling MythTV that we live in Helsinki, it started to pull weather forecasts complete wth satellite pictures from Weather.com, providing a nicely presented, always up to date weather forecast a couple of remote button presses away.

News: Naturally I can program the latest news broadcast to always be recorded, but in addition MythTV will pull in the web-based news articles from any sites I want. BBC, CNN, Dilbert...

Other video: Whether its home videos or stuff pulled from the net, the system will show it all just like TV programming. I've enjoyed a couple of episodes of MTV's Pimp My Ride this way recently.

Picture gallery: All of our digital pictures are stored on this box, and MythTV includes a nice picture gallery component for watching slideshows on our projector.

Further

The possibilities are endless. There's an extension to MythTV providing classic arcade and console games via the MAME project that I have not yet tried. The built-in web browser is perfect for casually checking a web site from time to time, or regularly accessing MovieLens for the latest movie recommendations. Although Linux doesn't have that many games available for it, some of the best ones are perfect for living room enjoyment..

Bottom line: MythTV provides us with an extremely versatile, and for the most part nicely polished HTPC system, but it still is a bit too difficult to set up, especially when using it for DVB reception in a small country like Finland. Most people will get a more satisfactory experience by buying a ready-made solution, but if you're willing to tinker, and have experience with computers, this approach will give you much more than anything available in a store.

Comment by Sylvain on Tue, 23 Aug 2005 06:16:55:
I also have a Sanyo PLV-Z1 and run MythTV. I was wondering if you were able to drive it in its native resolution (964x544) from X? Thanks, Sylvain

Comment by oa on Thu, 25 Aug 2005 10:30:03:
Check my latest entry, hopefully it'll help you along. http://www.fishpool.org/archives/tech/200508/htpc_project_projector_and_remote_configurations.html

I've been thinking about this project pretty much ever since I replaced our television with a video projector last March. In the meantime, I've just been reading about the matter and thinking about alternative ways of doing it, but yesterday I got on with the job.

Plan: To build a DVB-enabled, Linux and MythTV powered HTPC to act as a DVD player, video recorder, living room entertainment system, and whatever else comes along. In this space, I'll be documenting the process, talking about the gotchas I've ran into, and how I've managed to solve them. Perhaps I'll be asking for a bit of advice as well. The instructions on the web for this are at times quite sparse, often out of date, and occasionally misleading, so hopefully this will help someone. Read on for details...

In September, I bought an Athlon XP 2500+ system with an ASUS A8N7X-VM/400 motherboard. The primary reason I bought that, instead of for example an Athlon64 system, was that I wanted a compact M/B with most things integrated to it, because I was already thinking of building this project on that system. I didn't want to get something totally overpowered for the task, but not something that I couldn't be sure would be up to the task, either. This machine has been running Fedora Core 2 since its installation until about a week ago, when I upgraded it to FC3. It was equipped with a RADEON 9600 PRO card, but in the beginning of this project, I removed that to use the integrated GeForce 4 MX, which is easily good enough for video display, and in fact thanks to NVIDIA's well-optimised drivers, possibly better for that purpose that the ATI, given that we are in a Linux environment.

Yesterday, I bought a budget DVB-T card: TechniSat AirStar 2 TV PCI. I was told it has good reception, and its form factor is one of the smaller cards, which will be good later on when I exchange the case of the PC to a HTPC case. Many of the current crop of DVB-T cards are very high, possibly too high to comfortably fit in a relatively low-profile system. This card is of the latest revision, with an included remote control and IR receiver, which makes the package pretty high value, but a bit worrying considering on Linux it has to be supported by a volunteer effort.

True enough - it turned out that though FC3's default kernel includes the DVB subsystem as modules, it did not include the correct driver for the updated frontend chip (mt352) on the card. Only the earlier revision (mt312) worked out of the box.

Fortunately, this is a problem that can be easily overcome by using the very latest (as of November 2004, anyway) DVB drivers available a the LinuxTV website. Linux kernel module installations have come a long way: it used to be that something like this would require patching, reconfiguring the entire kernel, waiting for the whole kernel compile, and ultimately a bit of anxiety, no matter how used one becomes to the routine, of whether the new kernel would even boot the machine. No more - with FC3, you don't even need the kernel sourcecode. Simply check out the dvb-kernel module, and run make in the build-2.6 subdirectory. The only (small) complication was in deciding how to replace the existing prebuilt modules with the new ones. I decided to simply copy the new ones on top of them. I could have always returned where I started from by reinstalling the kernel RPM. No need to do so - the updated drivers work like a dream.

Comment by ThornyDevil on Sun, 19 Jun 2005 16:17:18:
Thank you for this journal. I couldn't understand why I was having problems with my new AirStar 2 card using KnoppMyth - but now I know! "frontend chip (mt352) on the card. Only the earlier revision (mt312) worked out of the box. "

Comment by Osma on Sun, 19 Jun 2005 20:31:44:
Kernels since 2.6.10 (I think) include the mt352 driver by default. These days I have two of the cards in my HTPC, working very nicely together.

So, I spent today at Linux Summit, since I had a presentation there in the afternoon. Got to hear Jon 'maddog' Hall, a bit of RMS, and some other talks..

My own presentation, co-presented with Kaj Arnö from MySQL, went okay, with a decent enough audience considering it had somehow been left out of the hand-out schedule (it's on the online schedule, though). A couple of questions afterwards, so I guess not everyone was sleeping, either.

Maddog talked about Linux and its enterprise readiness, Mats Wichmann talked about LSB and why developers should care (not that it matters a lot to us, as we don't distribute software), and RMS was his usual self. Apparently he had slipped at the airport and sprained his wrist, or something.

However, the really interesting talk came by surprise to me, as Jarmo Rosenqvist talked about Pupesoft, the financial package developed for Arwidson (Jarmo is the chairman of the board for that company). I had no idea such a project existed. It is both inspiring and rewarding to hear that they have had such a good experience developing the software (starting from a Unisys system, Mac desktops and an open mind) for their company, as I have been involved in similar, if much smaller situations myself (that is, solving the problems of small business owners with open source software).