Talk:Quality/Test processes

New test process

Some notes from #mer channel at 23.4.2012 from 12:34 onwards:

• Separating test-definition from the test packaging
• New test-definition/metadata defines test case
  • Same information than in the current test-definition
    • Architectural domain and subdomain/feature
  • HW (chroot or device or both)
  • Defines requirements, eg. needs 3G network or test data
  • Defines provides, eg. provides 3G network
  • Defines dependencies to packages 0-*
• Metadata files are stored in version control
  • Can be automatically updated to a database

• Testing changed packages first, then integration testing?

Test mapping / Test plan configuration

1. Test request comes in. The request has parameters such as stage and "changed packages" list
2. Testplanner reads the configuration/mapping file and creates test plan(s)
3. Kickstart files are delivered to image builder and test plans for test automation

Mapping/configuration file:

name: Release testing
devices:
  - chroot
    ots-device: mer-chroot
    kickstart: chroot-release.ks
  - virtual
    ots-device: mer-virtual-i686
    kickstart: virtual-image-core.ks
  - n900
    ots-device: mer-n900
    kickstart: nemo-n900.ks

default options:
  - chroot:
    filter: "hardware=chroot"
  - all:
    test package suffix: "-tests"

default packages:
  - chroot:
    - mer-core-chroot-tests
  - all:
    - mer-core-sanity-tests

packages:
  - alc:
    filter: "name=unittests"
    versio: < 1.5.0
    device: [virtual n900]

  - bluez

  - qt5base:
    test packages:
      - mwts-location-tests 
        filter: "type=functional"
        versio: 2.3.1
        device: [virtual n900]
      - qt5base-unittests

Description:

• devices: a list of devices for the tests are executed
  • ots-device: device name in the OTS (test automation)
  • kickstart: mapping kickstart files to devices
• default options: default options for devices
  • filter: filter options to testrunner-lite
  • test package suffix: if test package(s) are not defined for a package, then trying to use "original package name + suffix" test package. See the bluez package below.
• default packages: a list of test packages that are always executed
• packages: the actual package mapping
  • filter: filter options to testrunner-lite
  • version: package version to use
  • device: on which devices to execute the test
  • test packages: list of test packages and their configurations