The Mer Wiki now uses your Mer user account and password (create account on https://bugs.merproject.org/)

Talk:Quality/Test processes

From Mer Wiki
(Difference between revisions)
Jump to: navigation, search
(→‎Test mapping / Test plan configuration)
(→‎Test selection)
 
(20 intermediate revisions by one user not shown)
Line 1: Line 1:
−
== Test mapping / Test plan configuration ==
+
= New test process =
 +
 
 +
== Test definition ==
 +
Some notes from #mer channel at [http://mer.bfst.de/logs/%23mer/%23mer.2012-04-23.log.html#t2012-04-23T12:34:18 23.4.2012] from 12:34 onwards and from the QA meeting [http://mer.bfst.de/meetings/mer-meeting/2012/mer-meeting.2012-04-26-12.00.html 2012-04-26].
 +
 
 +
* Separating test definition from the test packaging
 +
* New test definition defines test cases and test sets
 +
** Test's steps are in own file, makes test management easier
 +
** Same information than in the current test-definition
 +
*** Architectural domain and subdomain/feature
 +
** HW (chroot or device or both)
 +
** Defines requirements, eg. needs 3G network or test data
 +
** Defines provides, eg. provides 3G network
 +
** Defines dependencies to packages 0-*
 +
* Test definition files are stored in version control
 +
 
 +
<br />
 +
 
 +
* This requires lots of changes to existing tools. We could take the current tools in use before developing them further.
 +
 
 +
New test-definition format:
 +
<pre>
 +
version: 1.0
 +
name: bluetooth tests
 +
domain: communications
 +
feature: bluetooth
 +
test cases:
 +
  - name: &check_interface
 +
    timeout: 60
 +
    test type: functional
 +
    dependencies: [bluez]
 +
    environment: hardware
 +
 
 +
  - name: &create_l2cap_connection
 +
    timeout: 120
 +
    test type: functional
 +
    dependencies: [bluez]
 +
    environment: hardware
 +
    provides: bluetooth-connection-l2cap
 +
 
 +
  - name: &close_l2cap_connection
 +
    timeout: 120
 +
    test type: functional
 +
    dependencies: [bluez]
 +
    environment: hardware
 +
    provides: bluetooth-connection-l2cap-close
 +
 
 +
  - name: &transfer_1Mib_file
 +
    timeout: 600
 +
    test type: functional
 +
    dependencies: [bluez, mer-testdata-files]
 +
    environment: hardware
 +
    requires-before: bluetooth-connection-l2cap
 +
    requires-after: bluetooth-connection-l2cap-close
 +
 
 +
  - name: &transfer_10Mib_file
 +
    timeout: 600
 +
    test type: functional
 +
    dependencies: [bluez, mer-testdata-files]
 +
    environment: hardware
 +
    requires-before: bluetooth-connection-l2cap
 +
    requires-after: bluetooth-connection-l2cap-close
 +
 
 +
test sets:
 +
  - name: sanity testing
 +
    - test: &check_interface
 +
    - test: &create_l2cap_connection
 +
    - test: &close_l2cap_connection
 +
    - test: &transfer_1Mib_file
 +
 
 +
</pre>
 +
 
 +
== Test selection ==
  
 
[[Image:test_mapping.png]]
 
[[Image:test_mapping.png]]
  
−
# Test request comes in. The request has parameter such as stage and changed package list
+
# New test request comes in. The request has parameters such as stage and "changed packages" list
−
# Testplanner reads the configuration/mapping file and creates test plan(s)
+
# Get from the stage's test selection file devices and test sets/cases
 +
# Get from test definitions requirements, packages and dependencies for test sets/cases
 
# Kickstart files are delivered to image builder and test plans for test automation
 
# Kickstart files are delivered to image builder and test plans for test automation
  
−
Mapping/configuration file:
+
Test selection:
 
<pre>
 
<pre>
 
name: Release testing
 
name: Release testing
−
devices: [ chroot virtual n900 ]
+
devices:
−
 
+
  - chroot
−
kickstart:
+
    ots-device: mer-chroot
−
  - chroot: chroot-release.ks
+
    kickstart: chroot-release.ks
−
   - virtual: virtual-image-core.ks
+
   - virtual
−
   - n900: mer-core-n900.ks
+
    ots-device: mer-virtual-i686
 +
    kickstart: virtual-image-core.ks
 +
   - n900
 +
    ots-device: mer-n900
 +
    kickstart: nemo-n900.ks
  
 
default options:
 
default options:
 
   - chroot:
 
   - chroot:
 
     filter: "hardware=chroot"
 
     filter: "hardware=chroot"
−
  - filter: "testset=core"
 
−
  - test package prefix: "-tests"
 
  
 
default packages:
 
default packages:
 
   - chroot:
 
   - chroot:
 
     - mer-core-chroot-tests
 
     - mer-core-chroot-tests
−
   - mer-core-sanity-tests
+
   - all:
 +
    - mer-core-sanity-tests
  
 
packages:
 
packages:
−
   - alc:
+
   - package: bluez
−
     filter: "name=unittests"
+
     - set: sanity testing
−
     versio: < 1.5.0
+
     - test: transfer_10Mib_file
−
    device: [virtual n900]
+
−
 
+
−
  - bluez
+
−
 
+
−
  - qt5base:
+
−
    test packages:
+
−
      - mwts-location-tests
+
−
        filter: "type=functional"
+
−
        versio: 2.3.1
+
−
        device: [virtual n900]
+
−
      - qt5base-unittests
+
  
 
</pre>
 
</pre>
  
−
* devices: list of devices to execute tests
+
'''Description''':
−
* kickstart: map kickstart files to devices
+
 
−
* default options: default options for all packages
+
* devices: a list of devices for the tests are executed
−
** filter: filter options to testrunner-lite
+
** ots-device: device name in the OTS (test automation)
−
** test package prefix: if no test package(s) defined for a package, then test package is the original package name + prefix. See the bluez package below.
+
** kickstart: mapping kickstart files to devices
−
* default packages: list of packages to execute always
+
* default options: default options for devices
−
* packages: package configurations for changed packages
+
* default packages: a list of test packages that are always executed
 +
* packages: the actual package mapping
 +
** version: package version to use
 +
** device: on which devices to execute the test

Latest revision as of 05:52, 2 May 2012

[edit] New test process

[edit] Test definition

Some notes from #mer channel at 23.4.2012 from 12:34 onwards and from the QA meeting 2012-04-26.

  • Separating test definition from the test packaging
  • New test definition defines test cases and test sets
    • Test's steps are in own file, makes test management easier
    • Same information than in the current test-definition
      • Architectural domain and subdomain/feature
    • HW (chroot or device or both)
    • Defines requirements, eg. needs 3G network or test data
    • Defines provides, eg. provides 3G network
    • Defines dependencies to packages 0-*
  • Test definition files are stored in version control


  • This requires lots of changes to existing tools. We could take the current tools in use before developing them further.

New test-definition format: 

version: 1.0
name: bluetooth tests
domain: communications
feature: bluetooth
test cases:
  - name: &check_interface
    timeout: 60
    test type: functional
    dependencies: [bluez]
    environment: hardware

  - name: &create_l2cap_connection
    timeout: 120
    test type: functional
    dependencies: [bluez]
    environment: hardware
    provides: bluetooth-connection-l2cap

  - name: &close_l2cap_connection
    timeout: 120
    test type: functional
    dependencies: [bluez]
    environment: hardware
    provides: bluetooth-connection-l2cap-close

  - name: &transfer_1Mib_file
    timeout: 600
    test type: functional
    dependencies: [bluez, mer-testdata-files]
    environment: hardware
    requires-before: bluetooth-connection-l2cap
    requires-after: bluetooth-connection-l2cap-close

  - name: &transfer_10Mib_file
    timeout: 600
    test type: functional
    dependencies: [bluez, mer-testdata-files]
    environment: hardware
    requires-before: bluetooth-connection-l2cap
    requires-after: bluetooth-connection-l2cap-close

test sets:
  - name: sanity testing
    - test: &check_interface
    - test: &create_l2cap_connection
    - test: &close_l2cap_connection
    - test: &transfer_1Mib_file

[edit] Test selection

Test mapping.png

  1. New test request comes in. The request has parameters such as stage and "changed packages" list
  2. Get from the stage's test selection file devices and test sets/cases
  3. Get from test definitions requirements, packages and dependencies for test sets/cases
  4. Kickstart files are delivered to image builder and test plans for test automation

Test selection: 

name: Release testing
devices:
  - chroot
    ots-device: mer-chroot
    kickstart: chroot-release.ks
  - virtual
    ots-device: mer-virtual-i686
    kickstart: virtual-image-core.ks
  - n900
    ots-device: mer-n900
    kickstart: nemo-n900.ks

default options:
  - chroot:
    filter: "hardware=chroot"

default packages:
  - chroot:
    - mer-core-chroot-tests
  - all:
    - mer-core-sanity-tests

packages:
  - package: bluez
    - set: sanity testing
    - test: transfer_10Mib_file

Description:

  • devices: a list of devices for the tests are executed
    • ots-device: device name in the OTS (test automation)
    • kickstart: mapping kickstart files to devices
  • default options: default options for devices
  • default packages: a list of test packages that are always executed
  • packages: the actual package mapping
    • version: package version to use
    • device: on which devices to execute the test
Retrieved from "https://wiki.merproject.org/wiki/index.php?title=Talk:Quality/Test_processes&oldid=1456"
Personal tools