Lemma's blog

Just in case you missed the announcement on the dot (or chose to forget):

This sunday, January the 25th, we'll be having our second KOffice Bug Day. It will all be about finding as many of the remaining bugs in KOffice Beta 5 as possible. To contribute, just read the bug day's techbase page, check the requirements and visit #kde-bugs somewhen on sunday (starting around 10am CET). As always, no programming skills required.

Is it worth it?

Most definitely. Our last KOffice bug day was a what I'd call a big success with the KOffice developers starting to iron out the bug while we were still triaging and krushing. They are really passionate about delivering good quality release.

During the last week I've been working hard on the Bugzilla sources (I've even been learning some Perl ) trying to make searchers faster. It made a rather lengthy blogpost (containing some big pictures though) and I wasn't sure if the general Planet KDE audience was that much interested in it as it's only partly related to KDE. So to avoid spamming the Planet all over I decided to make it a separate post which doesn't get syndicated. It might be interesting to you if you're into Bugzilla, full-text search or semi-fancy benchmarking graphs.

Here you go...

The major tool we use in Bugsquad is our beloved bugtracker, currently running a tailored version of Bugzilla 3.0. It usually works pretty well but has had some problems lately. One of them is that Bugzilla's search performance when searching through the bugs' comments is pretty low, degrading with the total number of comments stored. This total doesn't only encompass open bugs but closed ones as well, so performance is steadily declining (we currently have >180000 bugs).

My journey started when I heard someone asking if switching from MySQL to PostgreSQL as the unerlying storage layer would help the speed. I decided do dive into it and started doing some simple benchmarks. Unfortunately results indicated that PostgreSQL would only marginally increase performance. As there's other sites (eg. techbase) using the same database we'd either have to migrate all of them (if that's even possible) or live with two database servers running at once. While I don't have a problem to have different sites handled by different database systems, memory consumption certainly would have been an issue, so I scrapped the idea.

The Sphinx full-text search engine

On searching for ways to speed things up I discovered a small full-text search engine named Sphinx and decided to give it a try. Sphinx has several features which are pretty useful for our purpose:
(DISCLAIMER: I'm not an expert on full-text search engines. Some of the features described might be pretty common, some might not be. I don't know how Sphinx compares to other fts systems.)

• It's a complete daemon-based search engine with various language bindings ready to use.
• It comes with an indexer that can be used to pull data from MySQL tables into its index.
• It allows substring searches with arbitrary infix length.
• It has builtin word stemming and also provides other text preprocessors like phoneme conversion.

The first results after constructing my first full-text index were pretty promising, so I started with a proof-of-concept implementation of a Bugzilla extension that uses Sphinx.

Extending Bugzilla

Unfortunately developing for Bugzilla means that you have to code Perl - a language I so far always avoided to learn because it's said to be hard to read. While this generally may or may not be true, Bugzilla's code turned out to be pretty understandable and not too hard to work with. It even provides nice hooks to extend its functionality with custom extensions without even having to touch the core code - not for extending or refining the search mechanism though, so I had to devise my own little hook which I hope will be reviewed and make it upstream one day.

Searching using my extension works like this:

• Bugzilla asks the extension to handle a tuple ($field,$operator,$value) where $field is basically the database field to comb through, $operator is the match operator to apply (eg. equals or constains all the words) and $value is the value the user is looking for.
• Additionally the extension is told what Bugzilla itself would do to perform the search (I'll call this the original query).
• The extension then decides if it can handle the search and - if so - queries the Sphinx daemon for bugs matching what the user searches for (else it would just fallback to the original query).
• The original query is then either replaced or enriched with Sphinx' results and sent back to Bugzilla.

Benchmarking

With this basic implementation in place it was time to do some benchmarking. While reading this, please keep the following in mind:

• I'm not an expert on tuning MySQL. I used mysqltuner to get as good and fair results as possible. I ended up giving around 5G of memory to MySQL, distributed betwen innodb_buffer_pool_size, the query cache and the key_buffer_size.
• I know MySQL has its own FULLTEXT index - I didn't use it though.
• I certainly don't want to claim MySQL is slow.
• Tests were performed on my Q6600 desktop machine WHILE I was running the desktop. While I tried to keep my resource usage down, the benchmark's results might not be representative.

The sample set

As I didn't have exact numbers about the amount of data I constructed a sample set consisting of:

• 200 products, 5 components and 5 versions each
• 200.000 bugs
• 2.200.189 comments uniformly distributed among them
• Short descriptions randomly generated from /usr/share/dict/british-english-huge with a uniformly distributed length of 5-20 words
• Comments also randomly generated with a uniformly distributed length of 20-200 words

The benchmark method

I performed tests on the following search types:

• substring - Simple substring search
• anywordsubstr - Substring search trying to match any of several search strings
• allwordssubstr - Substring search trying to match all of several search strings

I performed each test 20 times in a row, using 1-3 random words as search terms (using the same random seed for bare Bugzilla and Bugzilla/Sphinx searches). Searches were performed by sending the queries using HTTP. To strip template processing times I introduced a Bugzilla template that returns nothing but the bare number of hits.

Please keep in mind that the results are only meant to indicate the difference in the overall level of performance. As the search terms were completely random (and usually had a lot of search results) they won't tell you how searches you usually do on Bugzilla will perform. Furthermore even though no list of bugs is generated, search speeds are related to the number of rows found and returned by the database. The benchmark doesn't take this into account.

Results: Comparing plain Bugzilla and Bugzilla/Sphinx

The comparison was done using one single index per table.

This chart shows the times a single search on the bugs' summary took (in ms). med_std shows the average time a regular Bugzilla search took, med_sphinx shows the average time a search using the Sphinx extension took. Overall searching summaries using Sphinx took around half the time of a regular Bugzilla search (47%) with 1 word substring searches showing the smallest drop (71%) and 3 word anywordsubstr searches showing the biggest (35%).

This chart shows the times searching the bugs' comments took (in ms). Not surprisingly the drop is quite big here with Sphinx only taking around 3% of the time regular searches do. Again the smallest drop can be witnessed on 1 word substring searches (7%), the other searches rank between 2% and 4%.

To sum up, Sphinx combined with Bugzilla seems to be a good way to increase performance. Please note that I didn't measure the resources taken. That's probably hard to achieve on a regular desktop PC as Sphinx seems to rely on disk/system caches instead of caching data itself. A quick look suggests that runtime resource consumption is a fair bit lower with Sphinx though.

Results: Scalability

To see how well my extension scales with the number of concurrent searches performed I did another benchmark. I ran the same queries as before but started 1-10 of them at the same time doing 20 testruns per setup. The numbers represent the average time taken per query (please keep in mind that due to the parallelity of the test the time taken to perform all n searches of a run doesn't equal the n ∗ average).

As you can see the search extension seems to scale well. The time increment seems to be linear but I didn't do any tests to confirm this. The small spikes you see now and then are mostly due to one single search taking a little longer - as I said I didn't stay from my machine all of the time.

If running a lot of parallel searches at the same time, the Sphinx search daemon performs best if you partition your index into several parts. It will then run one search thread per index (indexes can also be distributed to several servers to improve performance). That's exactly what I did for these scalability tests, using 4 indexes on 4 CPU cores.

Peculiarities, additional benefits and drawbacks

In addition to being faster, Sphinx has some peculiarities and extra advantages:

It's important to note that the Sphinx extension doesn't always return the same amount of search results a vanilla Bugzilla would. This is due to stripping of certain characters and treating accented characters the same way. In most situation this is beneficial because it ignores common spelling mistakes like inserted apostrophes (´) for plural forms.

Unlike in MySQL there's currently no way to push data into the fulltext index when it changes. Reindexing has to be scheduled. Fortunately the reindexing time can be cut by introducing multi-level delta indexes. However there will always be a lag between the time a summary or comment is entered and its availability for search.

Sphinx indexes (especially if indexed for substring search) take up a considerable amount of disk space. The sample implementation uses one to three indexes (depending on which searches you want to enable):

• bugs.short_desc - 521MB
• longdescs.thetext - 74297MB
• bugs_fulltext.∗ - 6095MB

However trading performance for diskspace might be fair under many circumstances.

To sum up, implementing an external full-text search engine into Bugzilla seems to be quite worthwhile. I hope I'll get this fully working and that our Bugzilla's admins still have a little diskspace left

Albeit having the kwallet category on my blog for quite a while now, I never really bothered to put anything in there. With the advent of KDE 4.2 I thought I'd share what I've been up to.

Bad news first...

Unfortunately I didn't quite get around improving the manager. I actually planned and started this but other things seemed more pressing. Thus KDE 4.2's kwalletmanager will continue to be not-quite-pretty

Good news for everyone

Some things have happened under the hood though. I've mostly been busy fixing bugs, including long withstanding ones like concurrency issues and lockups. Changes and fixes include:

• kwalletd has moved from kdelibs/kio to kdebase/runtime (actually pretty much invisible to anyone but packagers)
• Improved open wallet calls: Open calls can no longer timeout, open Path uses asynchroneous transaction handling.
• Detect exiting/crashing applications: The daemon will now detect applications that exit without closing the wallet or even crash and automatically close the wallet.

Important: If you're using trunk, be sure to svn up both kdelibs and kdebase next time you recompile either. Otherwise your wallet might cease to work.

Good news for some

Apart from that, good news for all KDE Windows users: Starting with KDE 4.1.3, KWallet works fine on Windows. Thanks go to Patrick Spendrin for helping me fix this (and helping me compile when my VirtualBox decided to crash instead of compile) and to Fujisan for providing me with the screenshot to prove it works:

The outlook

One of the major tasks I have in mind for 4.3 is finally redoing the manager UI. Another thing I'm currently on about is making the GNOME Keyring and the KDE Wallet systems interoperable. Unfortunately that's still at a very early stage so I can't really tell you a lot about it yet. I'll keep you posted though.

As you might have read on the dot, Bugsquad is having a KDE PIM party tomorrow, and you're all invited

• Techbase page for the triage
• IRC Channel
• We'll be starting at around 08:00 UTC.

I usually try to be more elaborate but as I'm currently quite busy the bare facts will have to do. I also still want to do a follow-up to our previous KOffice Triage + Krush day so expect more during the next few days.