Talk:Quality/Test processes

= New test process =

Test definition
Some notes from #mer channel at 23.4.2012 from 12:34 onwards and from the QA meeting 2012-04-26.


 * Separating test definition from the test packaging
 * New test definition defines test cases and test sets
 * Test's steps are in own file, makes test management easier
 * Same information than in the current test-definition
 * Architectural domain and subdomain/feature
 * HW (chroot or device or both)
 * Defines requirements, eg. needs 3G network or test data
 * Defines provides, eg. provides 3G network
 * Defines dependencies to packages 0-*
 * Test definition files are stored in version control


 * This requires lots of changes to existing tools. We could take the current tools in use before developing them further.

New test-definition format: version: 1.0 name: bluetooth tests domain: communications feature: bluetooth test cases: - name: &check_interface timeout: 60 test type: functional dependencies: [bluez] environment: hardware

- name: &create_l2cap_connection timeout: 120 test type: functional dependencies: [bluez] environment: hardware provides: bluetooth-connection-l2cap

- name: &close_l2cap_connection timeout: 120 test type: functional dependencies: [bluez] environment: hardware provides: bluetooth-connection-l2cap-close

- name: &transfer_1Mib_file timeout: 600 test type: functional dependencies: [bluez, mer-testdata-files] environment: hardware requires-before: bluetooth-connection-l2cap requires-after: bluetooth-connection-l2cap-close

- name: &transfer_10Mib_file timeout: 600 test type: functional dependencies: [bluez, mer-testdata-files] environment: hardware requires-before: bluetooth-connection-l2cap requires-after: bluetooth-connection-l2cap-close

test sets: - name: sanity testing - test: &check_interface - test: &create_l2cap_connection - test: &close_l2cap_connection - test: &transfer_1Mib_file

Test selection



 * 1) New test request comes in. The request has parameters such as stage and "changed packages" list
 * 2) Get from the stage's test selection file devices and test sets/cases
 * 3) Get from test definitions requirements, packages and dependencies for test sets/cases
 * 4) Kickstart files are delivered to image builder and test plans for test automation

Test selection: name: Release testing devices: - chroot ots-device: mer-chroot kickstart: chroot-release.ks - virtual ots-device: mer-virtual-i686 kickstart: virtual-image-core.ks - n900 ots-device: mer-n900 kickstart: nemo-n900.ks

default options: - chroot: filter: "hardware=chroot"

default packages: - chroot: - mer-core-chroot-tests - all: - mer-core-sanity-tests

packages: - package: bluez - set: sanity testing - test: transfer_10Mib_file

Description:


 * devices: a list of devices for the tests are executed
 * ots-device: device name in the OTS (test automation)
 * kickstart: mapping kickstart files to devices
 * default options: default options for devices
 * default packages: a list of test packages that are always executed
 * packages: the actual package mapping
 * version: package version to use
 * device: on which devices to execute the test