The Mer Wiki now uses your Mer user account and password (create account on https://bugs.merproject.org/)
Process
(Undo revision 126 by 78.32.229.233 (Talk)) |
(expand process (as me)) |
||
Line 1: | Line 1: | ||
− | == Mer | + | == Mer Process == |
− | + | The objective of the Mer change control processes is to ensure that each change is subjected to sensible QA checks - both automated and manual and to encourage Vendor participation in the QA process. | |
− | + | We will: | |
− | + | ||
− | + | ||
− | + | * Verify every Mer release through continuous integration and QA | |
− | + | ||
− | * | + | |
* Provide a easy contribution model with fast feedback on changes | * Provide a easy contribution model with fast feedback on changes | ||
− | * Have a distributed model of performing | + | * Have a distributed model of performing QA checks, so we can easily scale operations and support a variety of hardware |
− | == | + | == Summary == |
− | + | A contributor makes and submits a change which triggers the process. | |
− | + | When a change is proposed it must pass the following quality gates: | |
+ | (Each of these tests is described in more detail later.) | ||
− | + | === Core Checks === | |
+ | * Packaging checks | ||
+ | * Build check | ||
+ | * ABI Checks | ||
+ | * Trial build | ||
− | + | === Code Review (48hr window) === | |
+ | The change is available for manual review. | ||
− | + | === Vendor Checks: Early and Late (48hr window) === | |
+ | The change is now ready for automated review by vendors; first early notifications are sent, then after a while, late notifications. | ||
+ | * Image creation | ||
+ | * Device flashing | ||
+ | * Functional (OTS) tests | ||
− | == | + | === Manual testing (48hr window) === |
+ | With such a short window there is limited opportunity for manual | ||
+ | testing. | ||
− | + | === Maintainer Review === | |
+ | After 48hrs the maintainer looks at the submitted reviews and makes an accept/decline decision. | ||
− | + | === CI Automation === | |
+ | In order to verify that no obviously conflicting changes have occured the change is queued for automated acceptance. | ||
− | == | + | === Issues === |
+ | Handling conflicts? | ||
− | + | == Change Controlled Code == | |
− | + | We have: | |
− | + | * A large amount of git repositories, with branches "master" (Trunk) and potentially release branches (Mer_1.0, Mer_2.0) | |
+ | * Each of these repositories checks out to be a .spec package, with tarball included, including _meta (OBS meta information) and _attribute (attributes) | ||
+ | * "Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core | ||
− | + | == Contributor Process (abstracted) == | |
− | + | When a contributor identifies that a change is required in a package they should: | |
− | + | # Clone the git repository for the package | |
+ | # Make changes to the package | ||
+ | # Commit with a signed-off-by in the commit message | ||
+ | # Submit the change to the review process | ||
+ | |||
+ | They should now observe reviews and correct problems. This process is repeated until the change is approved and merged to the master git repository or until the proposal is rejected. | ||
+ | |||
+ | == Reviewer process (abstracted) == | ||
+ | |||
+ | # Read reviews of code changes on review site | ||
+ | # Perform own review and comment on review site | ||
+ | |||
+ | == Maintainer process (abstracted) == | ||
+ | |||
+ | # Read manual and automatic reviews | ||
+ | # Optionally do own review | ||
+ | # Approve/decline contribution on site | ||
+ | # Merge code change (manually or automatically) | ||
+ | |||
+ | == Core Checks process (Automated) == | ||
+ | |||
+ | When a proposed change comes in the following automated core checks are carried out and reported as 'reviews' of the change. | ||
=== Packaging checks === | === Packaging checks === | ||
− | Check if the proposed change is acceptable according to packaging guidelines | + | Check if the proposed change is acceptable according to packaging guidelines. |
=== Single-build test === | === Single-build test === | ||
− | Test if the proposed change builds on it's own | + | Test if the proposed change builds on it's own or if the appropriate sha1 has been built on a public OBS. |
=== Dependency-build test === | === Dependency-build test === | ||
− | Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly) | + | Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly). |
− | + | Following a dependency build there will be fresh packages to permit additional checks: | |
+ | |||
+ | ==== ABI checks ==== | ||
Test if the changed package causes any kind of breakage in ABI, utilizing http://forge.ispras.ru/projects/abi-compliance-checker | Test if the changed package causes any kind of breakage in ABI, utilizing http://forge.ispras.ru/projects/abi-compliance-checker | ||
− | == | + | == Vendor notification process (Automated) == |
− | + | A notice is sent over various channels (eg email, AMQP, rss) to notify the vendor community that a change is available to be checked. The results of the dependency-build test are provided and the vendor can carry out vendor specific checks. | |
− | + | The objective of this notification step is to ensure that changes do not cause regressions in vendor devices; or that, if applied to Core, vendors will be aware of problems that arise) | |
− | + | This is a 2-stage process; certain early notification vendors are told about the change right away, if there are no bad reviews within 12 hours, we notify late vendors. Reviews must be provided within the 48 hr review window to be considered. | |
− | == | + | It is possible that Mer will operate a sample reference vendor using a 'virtual device' such as kvm. |
+ | |||
+ | [Question: What's the criteria for *not* sending late notifications? Is this a veto?] | ||
+ | [Question: What kinds of checks should be allowed in the early notification review? Veto is quite powerful so it should probably be limited to "Doesn't build/boot"] | ||
+ | |||
+ | Typical gates here are: | ||
+ | |||
+ | ==== Image creation test ==== | ||
+ | |||
+ | Test if the repository result of the dependency-build test will create an 'image' succesfully, based on the vendor kickstart files. | ||
=== Upgradeability check === | === Upgradeability check === | ||
− | Take packages from dependency-build test and attempt an upgrade, report back as code reviews | + | Take packages from dependency-build test and attempt an upgrade on a suitable image (eg last official release), report back as code reviews. |
+ | |||
+ | === Device flash and boot === | ||
+ | Verify that any image is flashable and actually boots on a device to a certain level. | ||
+ | |||
+ | |||
+ | === Functional (OTS) tests === | ||
+ | If any functional or regression tests can be run then any failures may be provided. | ||
+ | |||
== Implementation DRAFT == | == Implementation DRAFT == | ||
Line 133: | Line 186: | ||
patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id> | patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id> | ||
</pre> | </pre> | ||
+ | |||
+ | ==== Things to make life easier ==== | ||
+ | |||
+ | If the OBS can identify the sha1 of a package then a sha1 built in a | ||
+ | public OBS against the correct repos is acceptable |
Revision as of 22:46, 22 October 2011
Contents |
Mer Process
The objective of the Mer change control processes is to ensure that each change is subjected to sensible QA checks - both automated and manual and to encourage Vendor participation in the QA process.
We will:
- Verify every Mer release through continuous integration and QA
- Provide a easy contribution model with fast feedback on changes
- Have a distributed model of performing QA checks, so we can easily scale operations and support a variety of hardware
Summary
A contributor makes and submits a change which triggers the process.
When a change is proposed it must pass the following quality gates: (Each of these tests is described in more detail later.)
Core Checks
- Packaging checks
- Build check
- ABI Checks
- Trial build
Code Review (48hr window)
The change is available for manual review.
Vendor Checks: Early and Late (48hr window)
The change is now ready for automated review by vendors; first early notifications are sent, then after a while, late notifications.
- Image creation
- Device flashing
- Functional (OTS) tests
Manual testing (48hr window)
With such a short window there is limited opportunity for manual testing.
Maintainer Review
After 48hrs the maintainer looks at the submitted reviews and makes an accept/decline decision.
CI Automation
In order to verify that no obviously conflicting changes have occured the change is queued for automated acceptance.
Issues
Handling conflicts?
Change Controlled Code
We have:
- A large amount of git repositories, with branches "master" (Trunk) and potentially release branches (Mer_1.0, Mer_2.0)
- Each of these repositories checks out to be a .spec package, with tarball included, including _meta (OBS meta information) and _attribute (attributes)
- "Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core
Contributor Process (abstracted)
When a contributor identifies that a change is required in a package they should:
- Clone the git repository for the package
- Make changes to the package
- Commit with a signed-off-by in the commit message
- Submit the change to the review process
They should now observe reviews and correct problems. This process is repeated until the change is approved and merged to the master git repository or until the proposal is rejected.
Reviewer process (abstracted)
- Read reviews of code changes on review site
- Perform own review and comment on review site
Maintainer process (abstracted)
- Read manual and automatic reviews
- Optionally do own review
- Approve/decline contribution on site
- Merge code change (manually or automatically)
Core Checks process (Automated)
When a proposed change comes in the following automated core checks are carried out and reported as 'reviews' of the change.
Packaging checks
Check if the proposed change is acceptable according to packaging guidelines.
Single-build test
Test if the proposed change builds on it's own or if the appropriate sha1 has been built on a public OBS.
Dependency-build test
Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly).
Following a dependency build there will be fresh packages to permit additional checks:
ABI checks
Test if the changed package causes any kind of breakage in ABI, utilizing http://forge.ispras.ru/projects/abi-compliance-checker
Vendor notification process (Automated)
A notice is sent over various channels (eg email, AMQP, rss) to notify the vendor community that a change is available to be checked. The results of the dependency-build test are provided and the vendor can carry out vendor specific checks.
The objective of this notification step is to ensure that changes do not cause regressions in vendor devices; or that, if applied to Core, vendors will be aware of problems that arise)
This is a 2-stage process; certain early notification vendors are told about the change right away, if there are no bad reviews within 12 hours, we notify late vendors. Reviews must be provided within the 48 hr review window to be considered.
It is possible that Mer will operate a sample reference vendor using a 'virtual device' such as kvm.
[Question: What's the criteria for *not* sending late notifications? Is this a veto?] [Question: What kinds of checks should be allowed in the early notification review? Veto is quite powerful so it should probably be limited to "Doesn't build/boot"]
Typical gates here are:
Image creation test
Test if the repository result of the dependency-build test will create an 'image' succesfully, based on the vendor kickstart files.
Upgradeability check
Take packages from dependency-build test and attempt an upgrade on a suitable image (eg last official release), report back as code reviews.
Device flash and boot
Verify that any image is flashable and actually boots on a device to a certain level.
Functional (OTS) tests
If any functional or regression tests can be run then any failures may be provided.
Implementation DRAFT
Current implementation, OBS, Gerrit, BOSS
Contributor setup
1. Register account on http://review.merproject.org 2. Add SSH keys and set user name, real name
Contribution model (gerrit)
1. Clone git repository for package from git://username@review.merproject.org:29418/packagepath
2. Do changes to package
3. Commit with signed-off-by in commit message, git commit -s -m "Message"
4. Submit patch to gerrit: git push origin HEAD:refs/for/master
5. Receive reviews and correct problems at http://review.merproject.org, until approved and merged to git repository
Reviewer model (gerrit)
1. Read automatic reviews of code change on http://review.merproject.org
2. Perform own review and comment on review site, use grading:
-1 I would prefer that you didn't submit this 0 No score +1 Looks good to me, but someone else must approve
Maintainer model (gerrit)
1. Read manual and automatic reviews
2. Optionally do own review, use +2 / -2 rating:
-2 Do not submit +2 Looks good to me, approved
3. Approve/decline contribution on site (Submit vs ..?)
4. Code change gets merged automatically
Automation
Documentation for gerrit hooks: http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.0/config-hooks.html
Gerrit hook: patchset-created
patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id>
Things to make life easier
If the OBS can identify the sha1 of a package then a sha1 built in a public OBS against the correct repos is acceptable