Revision as of 08:22, 16 October 2012

DRAFT as of 15. Oct. 2012

There are in several places infos about how to package tests in Mer for automated testing. This page tries to consolidate the info and define a standard. Also to point form the different places to this page.

The DRAFT shall be discussed at the next QA irc meeting and get it's approval and this note will be removed.

Work on going

this page is at the moment under construction! so please don't mess with it to much :-) consult kontio in IRC #mer

Pages/Sections which will point to this page

Packaging_guidelines#Test_Packages text will be replaced and point to this page

Quality/Development#Guidelines first part till OBS will be replaced with a pointer to this page

Quality/Terminology#Test package text (which even says to use /usr/share/<packagename>!) will be replaced and point to this page

Test Packages

Test package is a RPM package that includes scripts, binaries and/or libraries for testing a feature or function.

Guidelines

rpm packaging

• Test package's name must use <packagename>[-*]-tests format
  • Interfixes shall be used if it makes sense to split up the test package
  • The following interfixes shall be used: unit, benchmark, functional, feature

    eg. bluez-tests or bluez-unit-tests

  • if a binary and/or library is used in more than one <packagename>*-tests package, then it shall go to a package named:

    <packagename>-testutils and the <packagename>*-tests packages shall depend on it

• Doesn't have to depend on testrunner-lite
• Test-definition (tests.xml) is not required, however recommended since without no automated testing is possible
  • Test package's test-definition must be installed to /opt/tests/<packagename>/tests.xml
    • exception if upstream installs it due to historical reasons (e.g. Harmattan) to /usr/share/<packagename>/ it can stay there
• Test package's files should be installed to /opt/tests/<packagename>/
• If test package provides some common test data (audio, video, image) those files should be installed to /opt/tests/<packagename>/{data, audio, video, image, text etc}/,
  use common sense!

tests.xml file definition

• Test definition can have not approved or failing test cases

  in that case the test case must be marked as insignificant:
  <case name="not-so-important-test" insignificant="true">
  see also Test Definition HowTo

• If tests sets require special hardware then the hwiddetect feature of testrunner-lite must be used, see filtering based on Hardware Identifier

  Mer provides a hwiddetect binary: /usr/bin/hwiddetect add the following to your tests.xml to make use of it: TODO: the binary does not yet exist, needs to be created! the following list needs to be updated accordingly
  <hwiddetect>/usr/bin/hwiddetect</hwiddetect>
  The program can identify the following hardware: N9, N900, N950, x86-VM (Virtual Machine), RasperryPi TODO: which other hardware adoptions are available for Mer?

  • If the Mer provided /usr/bin/hwiddetect does not suite you, please provide patches and in the mean time you might use your own tool, but make sure it gets installed with the test package
• It can be expected that tests are run with eat-device package setup, this means that: TODO: link to eat-device description
  • the tests are run as non-root TODO: what shall be done when root permissions are needed for the tests to run?
  • DISPLAY and DBUS_SESSION_BUS_ADDRESS environment variables are exported
  • do not hard code any user paths, use $HOME and $XDG_* environment variables instead
• Testrunner-lite supports host based execution, where testrunner-lite is executed on a PC, and the test steps over ssh on hardware. You should always redirect the stdout, stderr and stdin streams of your background processes, if you don't want your test steps to time out in host-based execution.

  See testrunner-lite wiki and SSH FAQ for more information.

TODO: define values (if needed?!) for some of the key fields like: Level, Type, Domain, Component, Feature, Requirement