SMW 1.2 preview: faster more powerful queries

June 9 2008. The upcoming version of Semantic MediaWiki, SMW 1.2, brings various new features, most importantly a new storage engine for faster and more powerful query answering. Users are invited to start testing the development version of the new implementation that can be obtained from Programmer's guide.

From a user perspective, the main changes brought by the new storage backend are as follows:

• most operations, especially queries, are faster,
• full top-level disjunctions in queries will work using the keyword "OR" (this can be disabled by the administrator if desired),
• full equality support (based on redirects) will be the default, and it will not have major performance impacts,
• caching and optimisation strategies for Very Large Wikis to increase performance for applications with large categories,
• some specific performance issues are alleviated (Special pages),
• annotated interwiki links are likely to be supported, and will be available in queries too.

The general performance gain will hopefully let you use larger and more powerful queries. The new store will also require some minor changes in extensions for SMW, but the required adoptions should be very easy to do. The section below describes how to try out the new store to see how it performs on an existing wiki.

In combination with the improved storage design, SMW 1.2 also includes various improvements of the #ask query functionality. In particular, various syntactic simplifications are currently under review, so as to make even complex queries more readable.

Other improvements in SMW 1.2 include automatic storage updates (e.g. if a template was changed, semantic data on affected pages is refreshed now as well), MediaWiki 1.13 support, numerous bugfixes, and extended translations.

Trying out the new store

This section is for administrators who want to try out the new store on their wiki. The new storage implementation is activated by a simple setting in LocalSettings.php, and it is possible to have both new and old store running concurrently. Thus, one can try out the new implementation without being bound to use it for good. On the other hand, it is not recommended to do that on a productively-used wiki that uses several other SMW extensions: Extension "Page Forms" does not yet work with the new SQL store, and other extensions, such as Halo and Semantic Layers, have not yet been tested with the new store (the most recent versions of Semantic Drilldown, Semantic Calendar and Data Transfer, on the other hand, do support it already). However, it is safely possible to switch back to switch back to the old (current) SQL store at any time (no permanent changes are done during the switch).

Users who are interested in testing the new SMW can follow these steps:

(0) Make sure you run on SMW 1.1.1 without problems.

(1) Install the development version of SMW. The preferred method for that is to download SMW from Programmer's guide. No special steps are required after download – the settings of SMW 1.1.1 apply to the new version as well.

(2) To initialise the new store, and to fill it with your current data, the following maintenance scripts must be run (see the README on how to run SMW scripts):

php SMW_setup.php -b SMWSQLStore2
php SMW_refreshData.php -vtpb SMWSQLStore2
php SMW_refreshData.php -vb SMWSQLStore2

The last two (especially the final one) are long-running scripts. Please check your memory use while using them since there are PHP memory leaks in PHP 5 that consume much memory in long-running processes. Use the parameters -s <startindex> and -e <endindex> to refresh your wiki in chunks if needed, and use CTRL-C to stop scripts that eat up too much memory.

All of these updates do not affect the running wiki other than by requiring some computing performance to be executed. The old SMW store is not touched by the updates.

(3) Finally, switch your wiki to use the new store by adding the following to your LocalSettings.php:

$smwgDefaultStore = 'SMWSQLStore2';

As usual, this line should go after the SMW include and before enableSemantics. You can go back to the old store at any time by commenting out this option again. Only the active store is updated while the wiki is used, so if you change a store after having edited the wiki, then you should again invoke SMW_refreshData.php to make sure that the store is up-to-date.

Experiences with the new store can be reported to the SMW Project in the usual way.