ElasticStore

N/A
Provides information about using Elasticsearch
Table of Contents

Search feature information
ElasticStore the component to establish a connection between Semantic MediaWiki and a Elasticsearch cluster.
Collection
Loading... ES active replication monitoring, modification date discrepancy (en) Simple schematic on the matter of ElasticStore/Elasticsearch field mapping (en) ES active replication monitoring, missing entity (en) ES replication monitoring, maintenance mode (en)
Keywords
elasticstore · elasticsearch · search · semantic search · search interface · functional class · full-text search · relevancy scoring

The ElasticStore was introduced as part of Semantic MediaWiki 3.01 to provide a powerful and scalable Query Engine that can serve enterprise users and wiki-farm users better by moving query heavy computation to an external entity (meaning separated from the main DB master/replica) known as Elasticsearch.2.

ElasticStore

Requirements | Features | Setup | Usage | Settings | Technical notes | FAQ

The ElasticStore provides a framework to replicate Semantic MediaWiki related data to an Elasticsearch cluster and enable its QueryEngine to send #ask requests and retrieve information from Elasticsearch (aka ES) instead of the default SQLStore.

The objective is to provide an interface to Elasticsearch to:

• improve structured (and allow unstructured) content searches
• extend and improve full-text query support (including sorting of results by relevancy)
• provide means for a scalability strategy by relying on the ES infrastructure

Requirements

• Elasticsearch: Recommended 7.10.x (SMW 4.1.3), tested with 5.6.6, 6.1+ (SMW 3.x)
• Semantic MediaWiki: 3.0+
• elasticsearch/elasticsearch (PHP ^7.0 ~6.0 or PHP ^5.6.6 ~5.3)

The ElasticStore relies on the elasticsearch php-api to communicate with Elasticsearch directly and is therefore independent of any other vendor or MediaWiki extension that may use Elasticsearch as search backend (e.g. CirrusSearch).

It is recommended to:

• Use Elasticsearch 6+ due to improvements to its sparse field handling
• Consult the official hardware guide on specific Elasticsearch requirements If Elasticsearch has not enough resources, it could defeat this whole feature ElasticStore; check Elasticsearch logs and other Elasticsearch mechanisms in case of issue, and add resources if needed.

Features

The ElasticStore provides the same query features (given those are tested) as the SQLStore making its functionality equivalent to the SQLStore but provides better support for free text matches or for unstructured text (if enabled) when retrieved from an article or file. Furthermore, improved performance and scalability is expected especially when the document count becomes significant larger than 50000 articles or a lot of fulltext queries are executed.

• Active replication monitoring to inform user about the replication state of a document
• Handling of active property type changes without the need to rebuild the entire index itself after it is ensured that all ChangePropagation jobs have been processed
• Support of inverse queries such as [[-Foo::Bar]]
• Support of property chain and path queries (e.g. [[Foo.Bar::Foobar]])
• Support of category and property hierarchies

Setup

Before the ElasticStore (hereby Elasticsearch) can be used as drop-in replacement for the existing SQLStore based QueryEngine the following settings and operations are required:

• Set $GLOBALS['smwgDefaultStore'] = 'SMWElasticStore'; (see $smwgDefaultStore)
• Set $GLOBALS['smwgElasticsearchEndpoints'] = [ ... ]; (see $smwgElasticsearchEndpoints)
• Run php setupStore.php or php update.php
• Rebuild the index using php rebuildElasticIndex.php (see rebuildElasticIndex.php)

A more detailed introduction can be found as part of the usage and settings section.

General notes

Elasticsearch is not expected to be used as data store replacement which means a RDMBS backend (MySQL, SQLite, or Postgres) is still required.

The ElasticStore provides a customized serialization format to transform and transfer the required data to Elasticsearch. The QueryEngine provides a #ask-ES DSL interpreter (see domain language) which ensures that any existing #ask query can be answered by the Elasticsearch cluster without changing its syntax when switching from a SQLStore (given that index process has been completed).

Upgrading ElasticSearch with SMW 4.1.3

If you are using ElasticStore as a backend, then you need to perform the following changes when using SMW 4.1.3 or above3:

• Step 1: Upgrade your version of Elastic Search to 7.10.2 (if you are using docker that should work without any other action than changing the version number)
• Step 2: Make sure your version of the elasticsearch package is right, you might want to add it to the composer.local.json: `COMPOSER=composer.local.json composer require elasticsearch/elasticsearch "7.17.2"`
• Step 3: Add $smwgElasticsearchCredentials after $smwgElasticsearchEndpoints:

$smwgElasticsearchEndpoints = [ [ 'host' => $elasticSearchHost,
                                  'port' => 9200,
                                  'scheme' => 'http',
                                  'user' => $elastic_user,
                                  'pass' => $elastic_password ] ];
$smwgElasticsearchCredentials = [
    'host' => $elasticSearchHost,
    'port' => 9200,
    'scheme' => 'http',
    'user' => $elastic_user,
    'pass' => $elastic_password,
];

See also

Elasticsearch consumes a lot of resources. Make sure that you allocate sufficient resources. If unsure check your system regularly for errors potentially related to the lack of resources.4

Hardware

• www.quora.com: What hardware is best for ElasticSearch?
• www.elastic.co: Hardware