Talk:Quality/Test processes

New test process

Test definition

Some notes from #mer channel at 23.4.2012 from 12:34 onwards and from the QA meeting 2012-04-26.

• Separating test definition from the test packaging
• New test definition defines test cases and test sets
  • Test's steps are in own file, makes test management easier
  • Same information than in the current test-definition
    • Architectural domain and subdomain/feature
  • HW (chroot or device or both)
  • Defines requirements, eg. needs 3G network or test data
  • Defines provides, eg. provides 3G network
  • Defines dependencies to packages 0-*
• Test definition files are stored in version control

• This requires lots of changes to existing tools. We could take the current tools in use before developing them further.

New test-definition format:

version: 1.0
name: bluetooth tests
domain: communications
feature: bluetooth
test cases:
  - name: &check_interface
    timeout: 60
    test type: functional
    dependencies: [bluez]
    environment: hardware

  - name: &create_l2cap_connection
    timeout: 120
    test type: functional
    dependencies: [bluez]
    environment: hardware
    provides: bluetooth-connection-l2cap

  - name: &close_l2cap_connection
    timeout: 120
    test type: functional
    dependencies: [bluez]
    environment: hardware
    provides: bluetooth-connection-l2cap-close

  - name: &transfer_1Mib_file
    timeout: 600
    test type: functional
    dependencies: [bluez, mer-testdata-files]
    environment: hardware
    requires-before: bluetooth-connection-l2cap
    requires-after: bluetooth-connection-l2cap-close

  - name: &transfer_10Mib_file
    timeout: 600
    test type: functional
    dependencies: [bluez, mer-testdata-files]
    environment: hardware
    requires-before: bluetooth-connection-l2cap
    requires-after: bluetooth-connection-l2cap-close

test sets:
  - name: sanity testing
    - test: &check_interface
    - test: &create_l2cap_connection
    - test: &close_l2cap_connection
    - test: &transfer_1Mib_file

Test mapping / Test plan configuration

1. Test request comes in. The request has parameters such as stage and "changed packages" list
2. Testplanner reads the configuration/mapping file and creates test plan(s)
3. Kickstart files are delivered to image builder and test plans for test automation

Mapping/configuration file:

name: Release testing
devices:
  - chroot
    ots-device: mer-chroot
    kickstart: chroot-release.ks
  - virtual
    ots-device: mer-virtual-i686
    kickstart: virtual-image-core.ks
  - n900
    ots-device: mer-n900
    kickstart: nemo-n900.ks

default options:
  - chroot:
    filter: "hardware=chroot"

default packages:
  - chroot:
    - mer-core-chroot-tests
  - all:
    - mer-core-sanity-tests

packages:
  - package: bluez
    - set: sanity testing
    - test: transfer_10Mib_file

Description:

• devices: a list of devices for the tests are executed
  • ots-device: device name in the OTS (test automation)
  • kickstart: mapping kickstart files to devices
• default options: default options for devices
• default packages: a list of test packages that are always executed
• packages: the actual package mapping
  • version: package version to use
  • device: on which devices to execute the test