Selenium tests

Table of Contents

System testing of Semantic MediaWiki can be done using Selenium and the Selenium Framework provided by MediaWiki.

Creating Selenium Tests

Selenium IDE can be used well for quick recording of tests, debugging of tests, and learning how to write Selenium tests. Recorded Selenium tests can be exported as PHP to be included in the SMW code. In this HowTo we describe the usage of the IDE.

This section describes the general environment for Selenium Tests. The environment is valid for the whole system testing process, whether you are creating only one or several test cases.

Prerequisites

• Firefox (IDE/XPather not for firefox 4, yet!)
• Recording of tests: Selenium IDE [1] (this howto has been created using Selenium 1.x.)
• Finding targets (only one needed):
  • Firefox Plugin XPather [2]
  • Firefox Plugin DOM Inspector [3]

Selenium preparation

After installing Selenium IDE you can open this tool via the Firefox menu: Tools --> Selenium IDE. Selenium IDE opens in an own window.

• check options
  • Default timeout value for recorded commands (Default is 30.000 ms, 10.000 ms should be adequate)
  • Uncheck "Start recording immediately on open"
• define Base URL: Main Page of your wiki (AUT). Each test case shall start and end on the wikis Main Page.

Wiki (Application under Test)

The created tests shall be used automatically in other wikis afterwards. So, you need to avoid any dependencies to the wiki, in which the test were created. Make sure that:

• LocalSettings.php is on default
• Wiki is using standard skin (vektor)
• Wiki database is empty

Test creation

The aim of the Selenium Tests is to create several modular test cases, which all only check a single functionality or a single use case (for example: Define type of property).

Story

Each test case has one story, containing:

• setUp
• one or more tests
• tearDown

In the setUp you achieve the prerequisites for the respective test case. Mostly you will create and edit pages in the setUp or make some annotations, but you will not make any tests in this section. The general rule is: If a test afterwards will fail in the setUp, then the source of this problem is not in the respective feature or use case which is checked by the whole test case.

The single tests in contrast should check exactly the feature or use case of the whole test case. If you divide them, you may create several tests. Choose significant names for the tests (for example: check semantic search and check displaying).

With the tearDown you bring back the system to its original state before the tests. This means: delete all created pages and undo all of your editings. Finally you have to return to the Main page. The tearDown is very important to not leave any traces in the wiki. So, you can make sure, that your test be used several times in succession without any problems.

Uniform nomination

Please make sure, that you choose a clear nomination. With Selenium you will create a file for the setUp, for the several tests and the tearDown of each test case.

You should use the name of the test case for every file and add an prefix (for example, setUp, test01, etc.). If you have created more than one test for a test case, you need to illustrate the order of the several tests in their nomination.

Example files for one test case:

• setUp_DefineTypeOfProperty
• test01_checksemanticsearch_DefineTypeOfProperty
• test02_checkdisplaying_DefineTypeOfProperty
• tearDown_DefineTypeOfProperty

Record test

• Login manually (will later be done by the Selenium Framework)
• Set Base URL
• Create structure
  • One TestSuite (which will become one Test Case later)
  • Test Cases (which will become functions in the Selenium Framework). Example: setUp_Factboxtest, test01_Factboxtest_factboxpresent, test02_Factboxtest_factboxpropertypresent, ..., tearDown_Factboxtest
• Open Base URL in Browser
• Switch on recording
• Commands are recorded by Selenium automatically. To create and use commands manually, see Selenium Command Reference [4]
• record for setUp -> open, type, clickAndWait (done both together!), clickAndWait, type, clickAndWait (done both together!)
• record for tests -> (see: "How to find targets")
  • Factbox -> e.g., use assertElementPresent
  • Property -> e.g., use assertTextPresent
  • See Selenium Command Reference [5]
• record for tearDown -> clickAndWait...

How to find targets

• Selenium provides commands that need a target, e.g., "assertElemantPresent", see [6]
• There are tools that allow you to find target by simply clicking on an element
• e.g., use XPather [7]
• e.g., use DOM Inspector [8]
  • "Einen Knoten zum Inspezieren durch Draufklicken suchen"
  • Click on element -> Copy&Paste locator -> Put extra slash in front of locator to indicate "XPath" locator, e.g., "//html/body/div[@id='globalWrapper']/div[@id='column-content']/div[@id='content']/div[@id='bodyContent']/div[@id='mw-data-after-content']/div"
    • Note, that this is not a robust way of defining targets.
    • This way, it could easily be that a change to the underlying code leads to a break of the locator
• There are other (and especially more robust to code changes) ways to describe a locator, see [9]
• You can also use the context menu in your browser to add commands. Just right-click the concerning element when Selenium is running (not recording).

Final check and PHP export

After you have finished recording the test, you need to check, if the whole test is running without problems.

• Run the whole test case (Selenium: "Play entire test suite") in the browser, for each Test Case there is indicated whether it run successfully or failed.
• Run the test a second time to verify, that the test is repeatable
• Save Test Suite (made of all Test Cases), save Test Cases (so, now, they can be opened again), use selenium internalt format (HTML). Use the same names as described before (Selenium files don't need a file extension)
• Export test to php: Unfortunately you can not export the whole Test Suite (Selenium terminology). So you need to export each Test Case (Selenium terminology):
  • Select setUp and click File -> Export Test Case As... -> PHP(There are 2 PHP options to choose as export format, when using the Selenium IDE version 1.2; PHP PHPUnit and PHP Selenium_TestCase. Which format do I use?)--24.21.198.189 02:59, 15 August 2011 (CEST)
  • Do the same for the test files and the tearDown
  • Use the same file names. Just add file extension .php

The task of Creating SMW Selenium Tests ends with the creation of the php test files. The next task is Running SMW Selenium Tests.

Examples

• See [10] for examples.

Lessons Learned

This section should give some advices for creating tests in the future. Please add your experiences to reach a better workflow in the test creation process.

• please consider putting lessons learned into WMF efforts, see [11]

Experiences: Selenum IDE

• Browser context menu will help you to quickly find commands
• You may copy and paste (shortcuts are working) commands and edit them manually (for example if its the same command, only using another value)
• Note, that you have to select a test on the left with a double click before editing. A single click is only highlighting a test but you will still edit the last one.
• If you create similar test cases, you may reuse them (load test case, rename and make changes).
• You may also just reuse the setUp and the tearDown and extend them.

Experiences: Selenium with PHP

• In order to have special characters shown correctly in the browser, the TestSuite and TestCase PHP files were saved in UTF-8 format.
• Screenshots can be taken by: $this->captureScreenshot("C:\\Folder1\\Folder2\\..\\mySeleniumScreenshot.png");
• Bear in mind to read Selenium command documentation: e.g., getValue() and getText() are different.

Running Selenium tests

Aim: Running SMW Selenium tests on a local SMW instance.

Scenario

• Here, the setup is the following:
  • We install one instance of MW and SMW.
  • MW contains the Selenium Framework.
  • SMW contains the SMW Selenium Tests.
  • The instance is the AUT (application under test). Could also be another wiki, but here the wiki itself is tested. The AUT is configured to be empty and the Vektor skin is used.
  • Selenium RC (remote control) is used by the Selenium Framwork to run the SMW Selenium Tests on the instance.

What you need

• MediaWiki with Selenium Framework. The WMF Selenium Framework is available for later versions of MediaWiki, only. It also is still under development, especially in regard to Continuous Integration.
• This Documentation uses MW 1.18alpha (r90094).
• Selenium tests for SMW: They will be (not published yet) found in "MediaWiki/extensions/SemanticMediaWiki/tests/selenium/suites"
• Selenium Server remote controlling the browsers: selenium-server.jar [12].

Start Selenium Server on local machine

• E.g., "java -jar .\selenium-server.jar"
• More info at [13].
• Hint: WMF-Selenium-Framework possibly does this.

Setup MediaWiki/Semantic MediaWiki on local machine

• Configure LocalSettings.php of MediaWiki
• Example:

# Selenium 
$wgEnableSelenium = true;
$wgSeleniumConfigFile = $IP . '/tests/selenium/selenium_mysettings_local.ini';

• More information in [14].

• Create configuration file
• Add SMW test suite: testSuite[SMWSeleniumTestSuite] = "extensions/SemanticMediaWiki/tests/selenium/suites/SMWSeleniumTestSuite.php"

• Example:

[SeleniumSettings]

browsers[firefox] 	= "*firefox"
browsers[iexplorer]     = "*iexploreproxy"
browsers[chrome] 	= "*chrome"
host 			= "localhost"
port 			= "4444"
wikiUrl 		= "http://localhost/MediaWiki1_18/"
username 		= "WikiSysop"
userPassword 		= "XXXXXX"
testBrowser 		= "firefox"

startserver	        =
stopserver		=
jUnitLogFile		=
runAgainstGrid	        = false

; To let the test runner start and stop the selenium server, it needs the full
; path to selenium-server.jar from the selenium-remote-control package.
seleniumserverexecpath = "C:\Users\b-kaempgen\Documents\Programme\selenium-server-1.0.3\selenium-server.jar"

testSuite[SMWSeleniumTestSuite] = "extensions/SemanticMediaWiki/tests/selenium/suites/SMWSeleniumTestSuite.php"

• More information at [15]

Run SMW selenium tests

• E.g., "php .\tests\RunSeleniumTests.php"
• See [16] for more information.
• Failure or sucess of tests/assertions is indicated in the console.

Re-running tests after failure

• After a failure, pages might not have been deleted properly from the AUT.
• WMF Selenium Framework provides a way to automatically create a clean wiki instance before running the tests. See more information at [17]. However, this does not seem to work properly, yet.
• As a workaround: In order to re-run tests after a failure, create a backup from the database or use the Extension Nuke to delete all pages.

DocuFramework

We have created a wiki that shall support the creation of system tests. It is a Single-Source-Framework for describing functionality of SMW to be explained in the user/admin manual and tested with Selenium.

• Address: DocuFramework
• Note, this is work in progress.

Related Work

• WMF has some descriptions of tests:
  • http://www.mediawiki.org/wiki/Cite_Extension_Test_Plan
  • http://www.mediawiki.org/wiki/ConfirmEdit_Test_Plan
  • http://www.mediawiki.org/wiki/New_installer/Test_plan
  • http://www.mediawiki.org/wiki/Selenium/Deployment#Automation_work_done_by_the_Calcey_team
• WMF efforts to simplify writing and maintain tests, see [18]
• WMF test style guide: http://www.mediawiki.org/wiki/SeleniumFramework#Test_style_guide

Related Information

• Kick-Off and Vision: Presentation at SMWCon Fall 2010 - SMW System Testing with Selenium - SMWCon Fall 2010
• Status 2011: Presentation at SMWCon 2011 -
• People involved: Benedikt Kämpgen, Jonas Wäckerle