OurDelta - Builds for MySQL, MariaDB

Last year, Erik Ljungstrom (sensei in the #ourdelta IRC channel on Freenode) created patch for a bug Arjen identified with the handling of (among others) the open-files-limit option; the patch was first included in the OurDelta build of MySQL 5.0.67.

Now, Sun engineer Guilhem Bichot has committed the (modified) patch into the 6.0-maria tree, so it should appear in 6.0 and potentially 5.4. That’s good.

See http://bugs.mysql.com/40368 for details and history.

We had to apply a weird tweak as the default Ubuntu Jaunty packages are named something like 5.1.30really-5.0.xx. Several people have filed bugs on it with Ubuntu on Launchpad.

What I suspect happened (unconfirmed!) is that Canonical was contemplating putting 5.1 into Jaunty, had it in a beta but went back to 5.0  before release. Since downgrading by version number is a manual process in apt-get, the above hack allows a downgrade that looks like an upgrade…

Our original Jaunty build worked fine if you were starting from scratch, however an upgrade from the default MySQL on Jaunty would not work. Peter has built 5.1.30really-5.0.77-d8-ourdelta, which upgrades happily from the default Jaunty install or any other earlier install (such as from Intrepid). If you upgrade from an earlier Ubuntu version, do make sure you fix up your /etc/apt/sources.list.d/ourdelta.list with the release name jaunty. Easy as. Then run apt-get update then apt-get upgrade.

OurDelta builds of MySQL 5.0.77-d8 for Ubuntu 9.04 (Jaunty) builds are done, thanks to some smart and fast work by Peter. The packages are getting copied to the main site right now, and the mirrors should be up-to date within half a day or so.

OurDelta build for MySQL 5.0.77 with patchset d8 is now available, in source and packaged binaries for RHEL/CentOS 4 and 5, Debian 4.0 (Etch) and 5.0 (Lenny), Ubuntu 8.04 LTS (Hardy) and 8.10 (Intrepid).

If you already installed the OurDelta repository information, yum update or apt-get upgrade will install the updated packages for you. Please do review the notes below.

Fixes, Changes & Additions:

• With Percona moving the patches it maintains to using bzr/Launchpad, we now simply pull those in as a subdirectory percona_maintained. The old separate patch dirs are still present in the tree, we’ll remove those while making sure we’re not losing anything along the way. And as you know we do have more patches than just those that are maintained by Percona, also.
• The InnoDB Freeze patch was not included in this build. The Percona XtraDB backup tool is an excellent development. There might still be a use for the freeze trick for other purposes, but we’ll review that.\
• We still need to update the list of patches on the site for this release, and the accompanying docs pages.
• Peter has been working on binary tarballs, as there have been many requests for this. So this is generic Linux binaries, statically linked, which should work on most Linuxes. With the /usr/local/mysql layout. They’re nearly ready, so expect to see them within this week.
• There are no 32-bit builds for Debian 5.0 Lenny yet, the build VM actually the build crash with an internal compiler error, quite interesting. Rebuilding the entire VM doesn’t make a difference, but the same platform outside the VM works fine. We’ll work it out…

Peter has done the prep work for building on Lenny, see our Debian page for info on how to configure apt for easy updating (Lenny comes with 5.0.51).

Arjen & Monty @ LCA2009 in Hobart

I did a couple of sessions on OurDelta at the Linux.conf.au database miniconf: an overview of the project, a short delve into the features, and a “hacking the mysql server for dummies” which was found of particular interest. It’s a pity that session didn’t get accepted into the MySQL conference, it even had MySQL-uberguru Antony Curtis as co-speaker.

In the hacking talk in Hobart, I showed people the basic infrastructure of the source tree, then going through one particular patch and which changes it makes inside the server (and why). This is an excellent way to learn, as patches have a neat limited scope yet they do something significant. The good news is that the sessions were recorded, so when the LCA team finishes transcoding the hundreds of sessions we might be able to put ours up here!

Original author and MySQL co-founders Monty and David were also present, with their families. Monty told me that the last time he’d visited Australia was for an AUUG conference in 1998, that’s a few years before I moved here! David was at Linux.conf.au 2002 in Brisbane, he stayed at my place and my black cat (then tiny kitten) Figaro crawled on his neck. Anyway it was great to catch up, discuss ideas, have some Salmiakki, and share “home cooked” dinner (we had a full kitchen at the accomodation) with a number of other conference attendees. It was a great week!

In case you haven’t seen, Monty left Sun to be independent again. Still working on MySQL though! More interesting times ahead?

In other news, I’ve moved from VirtualBox to VMware Fusion for my desktop development system. While generally very nice (and free), VirtualBox kept destroying my snapshots, thus making work pretty much impossible and causing a lot of extra time getting wasted. VMware Fusion also supports 64-bit guests under OSX, multiple CPU cores, and other fancies. Still, I do hope (and trust) that VirtualBox will be developed and debugged further, it definitely has good potential!

While hard at work, it’s important to be visible so people don’t wonder if there’s anything happening… the open source development model is extremely good for this, things are just public all the time. Some may find this a source of stress or potential embarassment, but I think it’s great and overall makes for better quality code, products, and a nicer work environment.

To see what’s going on with OurDelta right now, we need to take a peek at https://code.launchpad.net/percona-patches where the Percona developers have been busy putting in lots of changes in both the 5.0.67 tree  (mainly little improvements and fixes to existing patches) as well as working on the 5.1 ports of the 5.0 patches. Launchpad lets you subscribe to a particular branch, so you get notified when there’s a new commit, and see the changeset comment. That’s a very handy way to keep up to date.

For OurDelta, the first 5.1 release will very simply be when we have all the 5.0 patches working in 5.1; it’s taking a bit of time and of course it would be great if it took less time, but I think we can all agree that it’s the right thing to do. Or if you feel different, please do speak up!

For 5.0, there’s always little fixes as well as some new patches… so how often to do a build? Right now I think once every two months is decent. Again, if you feel different, please let us know! Simply join the https://launchpad.net/~ourdelta-developers list and put in your thoughts. Community participation (which I believe is the multilateral form of ‘contribution’) is really that simple. Thanks!

OpenSQL Camp 2008 has come and gone, and hooray again for Baron who came up with the idea and made most of it happen (but let’s not forget Sheeri!)  Events such as these are always educational, but the most interesting stuff happens outside of the organised sessions (and this being an un-conf, they weren’t that strictly planned anyway

For me (Arjen) a major chunk of the exercise was acquiring jetlag there and back with no days to spare either side, but I feel it was well worth it. I spent most of the time listening and talking with people rather than coding. It was a great opportunity to catch up with Monty, the Percona crew (Baron, Peter, Vadim, Tom, and more - there’s so many of them now!), Brian, Stewart, Jay, Pat, Eric and other Drizzlers, Sheeri, active OurDelta people like Nick and Rob, ex-Brisbanite Ronald, and of course Jim who thought he might learn something in the MVCC session (no I’m not going to explain that joke!)

The Percona patches have moved to Launchpad, so from now on they’re being developed in plain sight, available earlier for peer review, and more easily integrated into OurDelta builds. Thanks Vadim for making that happen! An excellent example of the Open Source development model - the resulting quality will be even higher! The Percona crew is also working on their end of the 5.1 porting of the patches, with priority given to the performance-related ones. OurDelta contains additional patches and features, so we have extra work anyway - all help is great and much appreciated!

OurDelta is, as stated previously, committed to doing 5.1 builds also, but we’re going to continue doing 5.0 builds for the foreseeable future as well. Most deployments are currently on 5.0, and the various enhancements in OurDelta builds provide breathing space (in terms of performance, and monitoring/tuning instrumentation) while people check out 5.1 and plan for a possible upgrade. And we’ll make sure that anything we put in 5.0 will also be available in 5.1, so that you don’t lose anything when you do upgrade. That’s our promise.

We’re currently looking at a few more platforms to build for, such as Solaris and Windows. The latter is mainly aimed at developers, who will certainly appreciate the extra info they can get out of an OurDelta build and thus make better performing applications!

Next to performance-related patches, instrumentation is and will remain a key focus for OurDelta. We want to get even more information from the server (without increasing disk I/O, contention or CPU load and yes that is possible), as it offers more breadth and depth than any external solution. And we reckon -and that’s us who deal directly with real-world deployments on a daily basis- that is well worth the extra effort!

If you start with the d6 build, you probably have ourdelta.org in your repo files rather than mirror.ourdelta.org. Since we moved to using download mirrors, you need to update your repo config files. There are redirects in place for download users, but yum/apt-get generally don’t like redirects. For details on what your config should now look like, just take a peek at the information for each distro we currently support:

• RHEL/CentOS (4 & 5)
• Debian (4.0 Etch)
• Ubuntu (8.04 LTS Hardy, 8.10 Intrepid)

This week saw the release of OurDelta patchset d7 build of MySQL 5.0.67, basically a cleaned-up update of the earlier (and first) OurDelta d6 build. The number of downloads/fetches within the first few hours surpassed the total number from the previous weeks.

Downloads and yum/apt-get repository fetches now always go via one of our mirrors, as obviously the main server can’t possibly handle all that attention! By default you just get sent to “somewhere on the planet”, although you can tweak your repo setup to only use specific mirrors. If you want to become a mirror for OurDelta, drop us a line and we’ll be happy to add you in; the more the merrier!

Ubuntu 8.10 Intrepid is now also supported. We welcome input on which additional platforms are desirable.

There was a podcast, where interviewer James Purser came up with an good description of what OurDelta is: “a new distro for MySQL”.

Lead of the Drizzle project Brian Aker (Sun Microsystems), briefly hopped through Arjen’s home town of Brisbane Australia, and they had a chat about where and how we can work together on stuff.

OurDelta development in the coming weeks will focus on 5.1. If you would like to get involved with this particular effort, join the ourdelta-developers group on Launchpad, and check out the recent mailing list archive. There’s more to it than just code; but getting started there is not as hard as it seems, and there’s plenty of helpful hands about!