Talk:Quality/Test processes

New test process

Some notes from #mer channel at 23.4.2012 from 12:34 onwards:

• Separating test-definition from the test packaging
• New test-definition/metadata defines test case
  • Same information than in the current test-definition
    • Architectural domain and subdomain/feature
  • HW (chroot or device or both)
  • Defines requirements, eg. needs 3G network or test data
  • Defines provides, eg. provides 3G network
  • Defines dependencies to packages 0-*
• Metadata files are stored in version control
  • Can be automatically updated to a database
• Testing changed packages first, then integration testing?

1. This requires lots of changes to existing tools. We could take the current tools in use before developing them further.
2. New testrunner could be written in Python or similar language

New test-definition format:

version: 1.0
name: bluetooth tests
domain: communications
feature: bluetooth
test cases:
  - name: check interface
    timeout: 60
    test type: functional
    dependencies: [bluez]
    environment: hardware

  - name: create l2cap connection
    timeout: 120
    test type: functional
    dependencies: [bluez]
    environment: hardware
    provides: bluetooth-connection-l2cap

  - name: transfer 1Mib file
    timeout: 600
    test type: functional
    dependencies: [bluez, mer-testdata-files]
    environment: hardware
    requires: bluetooth-connection-l2cap

Test mapping / Test plan configuration

1. Test request comes in. The request has parameters such as stage and "changed packages" list
2. Testplanner reads the configuration/mapping file and creates test plan(s)
3. Kickstart files are delivered to image builder and test plans for test automation

Mapping/configuration file:

name: Release testing
devices:
  - chroot
    ots-device: mer-chroot
    kickstart: chroot-release.ks
  - virtual
    ots-device: mer-virtual-i686
    kickstart: virtual-image-core.ks
  - n900
    ots-device: mer-n900
    kickstart: nemo-n900.ks

default options:
  - chroot:
    filter: "hardware=chroot"
  - all:
    test package suffix: "-tests"

default packages:
  - chroot:
    - mer-core-chroot-tests
  - all:
    - mer-core-sanity-tests

packages:
  - alc:
    filter: "name=unittests"
    versio: < 1.5.0
    device: [virtual n900]

  - bluez

  - qt5base:
    test packages:
      - mwts-location-tests 
        filter: "type=functional"
        versio: 2.3.1
        device: [virtual n900]
      - qt5base-unittests

Description:

• devices: a list of devices for the tests are executed
  • ots-device: device name in the OTS (test automation)
  • kickstart: mapping kickstart files to devices
• default options: default options for devices
  • filter: filter options to testrunner-lite
  • test package suffix: if test package(s) are not defined for a package, then trying to use "original package name + suffix" test package. See the bluez package below.
• default packages: a list of test packages that are always executed
• packages: the actual package mapping
  • filter: filter options to testrunner-lite
  • version: package version to use
  • device: on which devices to execute the test
  • test packages: list of test packages and their configurations