The major tool we use in Bugsquad is our beloved bugtracker, currently running a tailored version of Bugzilla 3.0. It usually works pretty well but has had some problems lately. One of them is that Bugzilla's search performance when searching through the bugs' comments is pretty low, degrading with the total number of comments stored. This total doesn't only encompass open bugs but closed ones as well, so performance is steadily declining (we currently have >180000 bugs).
My journey started when I heard someone asking if switching from MySQL to PostgreSQL as the unerlying storage layer would help the speed. I decided do dive into it and started doing some simple benchmarks. Unfortunately results indicated that PostgreSQL would only marginally increase performance. As there's other sites (eg. techbase) using the same database we'd either have to migrate all of them (if that's even possible) or live with two database servers running at once. While I don't have a problem to have different sites handled by different database systems, memory consumption certainly would have been an issue, so I scrapped the idea.
The Sphinx full-text search engine
On searching for ways to speed things up I discovered a small full-text search engine named Sphinx and decided to give it a try. Sphinx has several features which are pretty useful for our purpose:
(DISCLAIMER: I'm not an expert on full-text search engines. Some of the features described might be pretty common, some might not be. I don't know how Sphinx compares to other fts systems.)
- It's a complete daemon-based search engine with various language bindings ready to use.
- It comes with an indexer that can be used to pull data from MySQL tables into its index.
- It allows substring searches with arbitrary infix length.
- It has builtin word stemming and also provides other text preprocessors like phoneme conversion.
The first results after constructing my first full-text index were pretty promising, so I started with a proof-of-concept implementation of a Bugzilla extension that uses Sphinx.
Extending Bugzilla
Unfortunately developing for Bugzilla means that you have to code Perl - a language I so far always avoided to learn because it's said to be hard to read. While this generally may or may not be true, Bugzilla's code turned out to be pretty understandable and not too hard to work with. It even provides nice hooks to extend its functionality with custom extensions without even having to touch the core code - not for extending or refining the search mechanism though, so I had to devise my own little hook which I hope will be reviewed and make it upstream one day.
Searching using my extension works like this:
- Bugzilla asks the extension to handle a tuple ($field,$operator,$value) where $field is basically the database field to comb through, $operator is the match operator to apply (eg. equals or constains all the words) and $value is the value the user is looking for.
- Additionally the extension is told what Bugzilla itself would do to perform the search (I'll call this the original query).
- The extension then decides if it can handle the search and - if so - queries the Sphinx daemon for bugs matching what the user searches for (else it would just fallback to the original query).
- The original query is then either replaced or enriched with Sphinx' results and sent back to Bugzilla.
Benchmarking
With this basic implementation in place it was time to do some benchmarking. While reading this, please keep the following in mind:
- I'm not an expert on tuning MySQL. I used mysqltuner to get as good and fair results as possible. I ended up giving around 5G of memory to MySQL, distributed betwen innodb_buffer_pool_size, the query cache and the key_buffer_size.
- I know MySQL has its own FULLTEXT index - I didn't use it though.
- I certainly don't want to claim MySQL is slow.
- Tests were performed on my Q6600 desktop machine WHILE I was running the desktop. While I tried to keep my resource usage down, the benchmark's results might not be representative.
The sample set
As I didn't have exact numbers about the amount of data I constructed a sample set consisting of:
- 200 products, 5 components and 5 versions each
- 200.000 bugs
- 2.200.189 comments uniformly distributed among them
- Short descriptions randomly generated from /usr/share/dict/british-english-huge with a uniformly distributed length of 5-20 words
- Comments also randomly generated with a uniformly distributed length of 20-200 words
The benchmark method
I performed tests on the following search types:
- substring - Simple substring search
- anywordsubstr - Substring search trying to match any of several search strings
- allwordssubstr - Substring search trying to match all of several search strings
I performed each test 20 times in a row, using 1-3 random words as search terms (using the same random seed for bare Bugzilla and Bugzilla/Sphinx searches). Searches were performed by sending the queries using HTTP. To strip template processing times I introduced a Bugzilla template that returns nothing but the bare number of hits.
Please keep in mind that the results are only meant to indicate the difference in the overall level of performance. As the search terms were completely random (and usually had a lot of search results) they won't tell you how searches you usually do on Bugzilla will perform. Furthermore even though no list of bugs is generated, search speeds are related to the number of rows found and returned by the database. The benchmark doesn't take this into account.
Results: Comparing plain Bugzilla and Bugzilla/Sphinx
The comparison was done using one single index per table.
This chart shows the times a single search on the bugs' summary took (in ms). med_std shows the average time a regular Bugzilla search took, med_sphinx shows the average time a search using the Sphinx extension took. Overall searching summaries using Sphinx took around half the time of a regular Bugzilla search (47%) with 1 word substring searches showing the smallest drop (71%) and 3 word anywordsubstr searches showing the biggest (35%).
This chart shows the times searching the bugs' comments took (in ms). Not surprisingly the drop is quite big here with Sphinx only taking around 3% of the time regular searches do. Again the smallest drop can be witnessed on 1 word substring searches (7%), the other searches rank between 2% and 4%.
To sum up, Sphinx combined with Bugzilla seems to be a good way to increase performance. Please note that I didn't measure the resources taken. That's probably hard to achieve on a regular desktop PC as Sphinx seems to rely on disk/system caches instead of caching data itself. A quick look suggests that runtime resource consumption is a fair bit lower with Sphinx though.
Results: Scalability
To see how well my extension scales with the number of concurrent searches performed I did another benchmark. I ran the same queries as before but started 1-10 of them at the same time doing 20 testruns per setup. The numbers represent the average time taken per query (please keep in mind that due to the parallelity of the test the time taken to perform all n searches of a run doesn't equal the n ∗ average).
As you can see the search extension seems to scale well. The time increment seems to be linear but I didn't do any tests to confirm this. The small spikes you see now and then are mostly due to one single search taking a little longer - as I said I didn't stay from my machine all of the time.
If running a lot of parallel searches at the same time, the Sphinx search daemon performs best if you partition your index into several parts. It will then run one search thread per index (indexes can also be distributed to several servers to improve performance). That's exactly what I did for these scalability tests, using 4 indexes on 4 CPU cores.
Peculiarities, additional benefits and drawbacks
In addition to being faster, Sphinx has some peculiarities and extra advantages:
It's important to note that the Sphinx extension doesn't always return the same amount of search results a vanilla Bugzilla would. This is due to stripping of certain characters and treating accented characters the same way. In most situation this is beneficial because it ignores common spelling mistakes like inserted apostrophes (´) for plural forms.
Unlike in MySQL there's currently no way to push data into the fulltext index when it changes. Reindexing has to be scheduled. Fortunately the reindexing time can be cut by introducing multi-level delta indexes. However there will always be a lag between the time a summary or comment is entered and its availability for search.
Sphinx indexes (especially if indexed for substring search) take up a considerable amount of disk space. The sample implementation uses one to three indexes (depending on which searches you want to enable):
- bugs.short_desc - 521MB
- longdescs.thetext - 74297MB
- bugs_fulltext.∗ - 6095MB
However trading performance for diskspace might be fair under many circumstances.
To sum up, implementing an external full-text search engine into Bugzilla seems to be quite worthwhile. I hope I'll get this fully working and that our Bugzilla's admins still have a little diskspace left 
