Laszlo Pandy

Saving data in an XML format has been all the rage for many years now and is the basis for both of the highly debated OOXML and ODF document formats. Part of the reason for its popularity is that many claim it is much easier to understand than the binary file formats of the previous generation. They might even say it is self documenting.

But as Håkon Wium Lie, the CTO of Opera once said:

…I’m no fan of either specification. Both are basically memory dumps with angle brackets around them.

He was talking specifically about the document formats competing for standarization at the time. More generally, he was talking about the whole idea of XML file formats. I remember thinking about Jokosher’s own file format after hearing this quote. It is an XML based format that I largely designed myself.

I will admit that our format is in a lot of ways a memory dump with angle brackes. The main nodes <Project>,<Instrument> and <Event> have the same name and structure as the classes that implement them. Each of these nodes also has a subnode called <Parameters> which is simply a list of the variable name, type and value of all the class variables. This only works well for simple data types; for list and dictionaries we have more complex structures.

In previous versions of Jokosher, I needed a easy way to store the very large list of volume levels used to draw the waveform. I figured that putting each number in its own XML node would take a lot longer to parse and consume much more memory than necessary so I created one simple node that looked like this:

<Levels value="0.911560058594,0.913299560547"/>


Imagine the above except with three thousand floating point levels, as is required for a typical five minute song. At this point it starts to get messy and stops looking like proper XML. It is also harder to decipher, and much more like a memory dump. Luckily the current version of Jokosher does not use this anymore. I stores the level data in a separate file that is just a raw dump of the bytes from the array.

Even though I have just shown Jokosher’s file format to be essentially a memory dump with angle brackets, I don’t feel bad about it. It makes our job as programmers easier, and the whole program is still simple enough that you can figure out most parts of the file format without documentation.

Today I had the chance to look at the file format used by Apple’s GarageBand, a program with a very similar purpose to Jokosher. It was interesting to find that like Jokosher, GarageBand has a folder for the project which contains the project file as well as a separate folder for all the audio files. Also just like Jokosher the project file is XML based.

After a few <dict> nodes obviously representing a dictionary data structure, there is a <data> node containing (wait for it…) 2584 lines of base-64 encoded data, and the entire XML file is only 2794 lines! Talk about a memoy dump with angle brackets. And you though your file format was difficult to interoperate with! If you really want to peer at the impurity, you can download the entire file.

Last weekend it was very hot. So me and my family went up to my uncle’s house on the lake. But of course when we get there, my uncle requests my help to fix his Internet Explorer. Right after you launch IE, it closes itself. But he has no idea what would be causing the crash.

I have seen this before, so I immediately uninstall and extra ‘toolbars’ that the crazy internet has installed in his browser. Turns out that after uninstalling the Yahoo! Toolbar, IE works fine (well actually not fine — but that’s how its supposed to work).

Of course the next thing I did was install Firefox. Then if that ever happened again, he wouldn’t be stuck without internet. But this raises a question in my mind. It wasn’t some dodgy spyware or anything that broke his browser — it was Yahoo!. I think Yahoo! is a respectable company, and they have a reputation to maintain. So why would their plugin for IE be so unstable?

This is relevant to me because we are in the process of creating the Jokosher 0.2 extension API. I would really like it if someone as popular as Yahoo! made an extension for Jokosher. But I would not like it if a user installs an extension and gets rewarded with Jokosher crashing immediately after startup.

We could just use DBUS and keep every extension on a long leash and a separate process, but that would hinder possible integration with the core code. We could also use a lot of exception handling, but that would only work if there was an exception, and not if there was a freeze.

The other possibility is that Yahoo! didn’t have the source code for IE so bugs of that kind could not be found during their testing. Maybe I shouldn’t worry about it at all and just trust the extension writers. Or better yet would be the creation of extensions.jokosher.org where we peer review everything. Of course the only problem with that is that I don’t feel like reviewing every single extension. Any suggestions?

Jokosher Update: Since the launch or 0.1, I have been mostly working on getting translations working. We have all the strings in the POT file with the exception of instrument name (but I’m working on that). If anyone feels like translating please go to: http://launchpad.net/products/jokosher/trunk/+translations
We already have four languages more than 50%!