Revision as of 09:30, 18 April 2012

Mer QA process

Mer has two testing processes: package and release testing. Check the overall process description.

Package testing

Package testing takes place when a package change is submitted via gerrit. After successful compiling and rebuild of any packages that depend on the package the tests are executed. The test results are then reported back to the gerrit.

Test mapping

One open issue with the package testing is that how to map tests to the package.

If all tests are using test packaging then the mapping could be like

 <package name>:<test package name>[:tr-lite filter]

 bluez:bluez-tests
 bluez:blts-bluetooth-tests:testset=Core

• Another issue is then where the mapping is located and how it is defined. In XML file or database?
• If we want to keep it simple then text or XML file and we can store the mapping file in the git.
  • In Maemo, this information was in the Debian's control file.

Release testing

(Pre) Release testing has two main test sets: core and feature set.

Core set

The Core set contains set of tests that verify overall quality of the Mer Core release. Test set have tests from all architecture domain and API tests. Core is in that way static that it is executed for every release and test content has only minor changes.

Feature set

Feature set includes tests for those packages that have been changed from the previous release. Basically feature set is set of package testing. The feature set content changes to every release.

Reference QA processes

MeeGo

This is from the public side of the MeeGo!

• No CI testing in place
• Only hourly, nightly and release testing
• QA-Reports was used to report test results
• QA Tools were OTS, testrunner-lite, tdriver
• MCTS tests were used

Maemo

Please fix if you see mistakes here.

• Active CI testing in place
  • Test packaging was used, the naming policy was strict
  • Own dashboard for managing test packages
  • Test package had in the Debian control file following parameters
    • XB-Maemo-CI-Packages and XB-Maemo-CI-Stage
    • When test package was updated to the version control, the control file information was moved to database
  • When a package changed, tests that defined the package in the control file were executed
    • Test packages were installed into the testing image and test automation executed all tests found from the image
  • All tests from the test package were executed
    • Caused problems if the test package had long or many test cases
• QA-tools were testrunner-lite, ots
  • OTS had ~couple of hundreds test request per day