The Mer Wiki now uses your Mer user account and password (create account on https://bugs.merproject.org/)

Process

From Mer Wiki
(Difference between revisions)
Jump to: navigation, search
(→‎ABI checks: updated link to acc tool)
 
(32 intermediate revisions by 4 users not shown)
Line 1: Line 1:
−
== Mer process ==
+
The objective of the Mer change control processes is to ensure that each change is subjected to sensible QA checks - both automated and manual and to encourage Vendor participation in the QA process.
 +
 
 +
We will:
 +
 
 +
* Verify every Mer release through continuous integration and QA
 +
* Provide a easy contribution model with fast feedback on changes
 +
* Have a distributed model of performing QA checks, so we can easily scale operations and support a variety of hardware
 +
 
 +
== Summary ==
 +
 
 +
A contributor makes and submits a change which triggers the process.
 +
 
 +
When a change is proposed it must pass the following quality gates:
 +
(Each of these tests is described in more detail later.)
 +
 
 +
=== Core Checks ===
 +
* Packaging checks
 +
* Build check
 +
* ABI Checks
 +
* Trial build
 +
 
 +
=== Code Review (48hr window) ===
 +
The change is available for manual review.
 +
 
 +
=== Vendor Checks: Early and Late (48hr window) ===
 +
The change is now ready for automated review by vendors; first early notifications are sent, then after a while, late notifications.
 +
 
 +
Vendors run whatever tests they like; typically:
 +
* Image creation
 +
* Device flashing
 +
* Functional (OTS) tests
 +
 
 +
=== Manual testing (48hr window) ===
 +
With such a short window there is limited opportunity for manual
 +
testing.
 +
 
 +
=== Maintainer Review ===
 +
After 48hrs the maintainer looks at the submitted reviews and makes an accept/decline decision.
 +
 
 +
=== CI Automation ===
 +
In order to verify that no obviously conflicting changes have occured the change is queued for automated acceptance.
 +
 
 +
=== Issues ===
 +
Handling conflicts?
 +
 
 +
== Change Controlled Code ==
  
 
We have:
 
We have:
Line 7: Line 52:
 
* "Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core
 
* "Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core
  
−
We want to:
+
== Contributor Process (abstracted) ==
  
−
* Make sure that every Mer release is release-able through continuous integration and QA
+
When a contributor identifies that a change is required in a package they should:
−
* Provide a easy contribution model with fast feedback on changes
+
−
* Have a distributed model of performing these QA checks, so we can easily scale operations and include donated hardware
+
  
−
== Contribution model (abstracted) ==
+
# Clone the git repository for the package
 +
# Make changes to the package
 +
# Commit with a signed-off-by in the commit message
 +
# Submit the change to the review process
  
−
1. Clone git repository
+
They should now observe reviews and correct problems. This process is repeated until the change is approved and merged to the master git repository or until the proposal is rejected.
 +
 
 +
== Reviewer process (abstracted) ==
 +
 
 +
# Read reviews of code changes on review site
 +
# Perform own review and comment on review site
 +
 
 +
== Maintainer process (abstracted) ==
 +
 
 +
# Read manual and automatic reviews
 +
# Optionally do own review
 +
# Approve/decline contribution on site
 +
# Merge code change (manually or automatically)
 +
 
 +
== Core Checks process (Automated) ==
 +
 
 +
When a proposed change comes in the following automated core checks are carried out and reported as 'reviews' of the change.
 +
 
 +
=== Packaging checks ===
 +
 
 +
Check if the proposed change is acceptable according to packaging guidelines.
 +
 
 +
=== Single-build test ===
 +
 
 +
Test if the proposed change builds on it's own or if the appropriate sha1 has been built on a public OBS.
 +
 
 +
=== Dependency-build test ===
 +
 
 +
Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly).
 +
 
 +
Following a dependency build there will be fresh packages to permit additional checks:
 +
 
 +
==== ABI checks ====
 +
 
 +
Test if the changed package causes any kind of breakage in ABI, utilizing [https://github.com/lvc/abi-compliance-checker ABI Compliance Checker].
 +
 
 +
== Vendor notification process (Automated) ==
 +
 
 +
A notice is sent over various channels (eg email, AMQP, rss) to notify the vendor community that a change is available to be checked. The results of the dependency-build test are provided and the vendor can carry out vendor specific checks.
 +
 
 +
The objective of this notification step is to ensure that changes do not cause regressions in vendor devices; or that, if applied to Core, vendors will be aware of problems that arise)
 +
 
 +
This is a 2-stage process; certain early notification vendors are told about the change right away, if there are no bad reviews within 12 hours, we notify late vendors. Reviews must be provided within the 48 hr review window to be considered.
 +
 
 +
It is possible that Mer will operate a sample reference vendor using a 'virtual device' such as kvm.
 +
 
 +
[Question: What's the criteria for *not* sending late notifications? Is this a veto?]
 +
[Question: What kinds of checks should be allowed in the early notification review? Veto is quite powerful so it should probably be limited to "Doesn't build/boot"]
 +
 
 +
Typical gates here are:
 +
 
 +
==== Image creation test ====
 +
 
 +
Test if the repository result of the dependency-build test will create an 'image' succesfully, based on the vendor kickstart files.
 +
 
 +
=== Upgradeability check ===
 +
 
 +
Take packages from dependency-build test and attempt an upgrade on a suitable image (eg last official release), report back as code reviews.
 +
 
 +
=== Device flash and boot ===
 +
Verify that any image is flashable and actually boots on a device to a certain level.
 +
 
 +
 
 +
=== Functional (OTS) tests ===
 +
If any functional or regression tests can be run then any failures may be provided.
 +
 
 +
 
 +
== Implementation DRAFT ==
 +
 
 +
Current implementation, OBS, Gerrit, BOSS
 +
 
 +
=== Contributor setup ===
 +
 
 +
1. Register account on http://review.merproject.org
 +
2. Add SSH keys and set user name, real name
 +
 
 +
=== Contribution model (gerrit) ===
 +
 
 +
1. Clone git repository for package from ssh://username@review.merproject.org:29418/projectpath/packagepath
  
 
2. Do changes to package
 
2. Do changes to package
  
−
3. Commit with signed-off-by in commit message
+
3. Commit with signed-off-by in commit message, git commit -s -m "Message"
  
−
4. Submit patch to review site
+
4. Submit patch to gerrit: git push origin HEAD:refs/for/master
  
−
5. Receive reviews and correct problems, until approved and merged to git repository
+
5. Receive reviews and correct problems at http://review.merproject.org, until approved and merged to git repository
  
−
== Reviewer model (abstracted) ==
+
=== Reviewer model (gerrit) ===
  
−
1. Read automatic reviews of code change on review site
+
1. Read automatic reviews of code change on http://review.merproject.org
  
−
2. Perform own review and comment on review site
+
2. Perform own review and comment on review site, use grading:
  
−
== Maintainer model (abstracted) ==
+
<pre>
 +
                              -1 I would prefer that you didn't submit this
 +
                              0 No score
 +
                              +1 Looks good to me, but someone else must approve
 +
</pre>
 +
 
 +
=== Maintainer model (gerrit) ===
  
 
1. Read manual and automatic reviews
 
1. Read manual and automatic reviews
  
−
2. Optionally do own review
+
2. Optionally do own review, use +2 / -2 rating:
 +
 
 +
<pre>
 +
                              -2 Do not submit
 +
                              +2 Looks good to me, approved
 +
</pre>
 +
 
 +
 
 +
3. Approve/decline contribution on site (Submit vs ..?)
 +
 
 +
4. Code change gets merged automatically
 +
 
 +
=== Automation ===
 +
 
 +
Documentation for gerrit hooks: http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.0/config-hooks.html
 +
 
 +
==== Gerrit hook: patchset-created ====
 +
 
 +
<pre>
 +
patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id>
 +
</pre>
 +
 
 +
==== Things to make life easier ====
 +
 
 +
If the OBS can identify the sha1 of a package then a sha1 built in a
 +
public OBS against the correct repos is acceptable
 +
 
 +
=== Release Process ===
 +
 
 +
Mer has weekly releases and goes through three phases: Pickup, Prerelease, Release.
 +
 
 +
==== Pickup ====
 +
 
 +
Pickup happens on Friday 00:01 UTC. A pickup is basically to grab the most recent commit SHA1 from each package git, update the Core with this new commit ID and initiate a release rebuild, which should take at most 48 hours, that is, finished on Saturday 23:59 UTC
 +
 
 +
==== Prerelease ====
 +
 
 +
Prerelease happens on Sunday 00:01 UTC. Prerelease consists of generating a release and providing this to vendors for QA purposes in order to catch issues before release, in order to see what changes ought to be pulled out before release.
 +
This allows vendors to do automated builds during sunday and be ready for monday.
 +
 
 +
==== Release ====
 +
 
 +
Release happens on Thursday 00:01 UTC. Release consists of the prerelease with eventual fixes and pulled out changes.
 +
 
 +
=== Draft new release process model ===
 +
 
 +
==== project-core branches and tags ====
 +
 
 +
mer/project-core.git is a description of the contents in the Mer Core, that is, xml files indicating what packages exist, what commit ID corresponds to their current contents, OBS project configurations, etc.
 +
 
 +
During development, the master branch is being updated continuously as new changes are merged in packages on their respective master branches.
 +
 
 +
When a pre-release is being prepared, it will create a branch off master for stabilization towards weekly release. Stabilisation fixes can then be pushed through gerrit.
 +
 
 +
The Mer CI OBS will have these ongoing builds:
 +
 
 +
  Core:ARCH -> master. At completion of each verified build, it will trigger a publish and increment the Version
 +
 
 +
  Core:ARCH:next -> current prerelease branch. At completion of each verified build, it will trigger a publish and increment the Version
 +
 
 +
The fakeobs will have these projects available:
 +
 
 +
  Core:ARCH -> master, binaries will be the latest complete Core:ARCH build from Mer CI OBS.
 +
 
 +
  Core:ARCH:next -> current prerelease branch, binaries will be latest completed Core:ARCH:next build from Mer CI
 +
 
 +
  Core:ARCH:<Version> -> Specific version; not all Versions will always be available.
 +
 
 +
CI testing is being done against Core:ARCH in Core always.
 +
 
 +
To propose changes to project configurations, etc, submit changes to refs/for/master.
 +
 
 +
==== Version numbers ====
 +
 
 +
RELEASEDATE = Always a Thursday
 +
 
 +
<pre>
 +
0.RELEASEDATE.0.0.0 -> First 'master' completed build after latest release, next release being RELEASEDATE
 +
 
 +
0.RELEASEDATE.0.1.0 -> Prerelease branch for release that'll happen on RELEASEDATE, first release
 +
 
 +
0.RELEASEDATE.0.1.1 -> Prerelease branch for release that'll happen on RELEASEDATE, second release
 +
 
 +
                .2 -> etc
 +
 
 +
0.RELEASEDATE.1    -> Release on RELEASEDATE
 +
 
 +
If a new release on 0.RELEASEDATE stabilization branch:
 +
 
 +
0.RELEASEDATE.1.0.1 -> First 0.RELEASEDATE 'master' build after 0.RELEASEDATE.1 release
 +
 
 +
0.RELEASEDATE.1.1.0 -> First Prerelease for new release on 0.RELEASEDATE stabilization branch
 +
 
 +
0.RELEASEDATE.2    -> Release for 0.RELEASEDATE stabilization branch
 +
 
 +
0.LATERRELEASEDATE.1 -> New release on 0 branch
 +
 
 +
If invasive change/major release new work begins:
 +
 
 +
1.RELEASEDATE.0.0.0 -> Invasive change, drop in quality OK, first 'master' completed build after 1.RELEASEDATE
 +
 
 +
.. etc
 +
 
 +
</pre>
 +
 
 +
=== New process 2 ===
 +
 
 +
We have a new method in OBS that allows us to full-copy a project with content to a new project. This enables us to do an abstraction of 'builds'.
  
−
3. Approve/decline contribution on site
+
A release build is described as:
 +
* A project core, along with source, hosted on a MDS instance. In practice, this is a <link project=> to a OBS project hosted within MDS.
 +
* A set of binaries that has been built which corresponds to the source
  
−
4. Code change gets merged (manually or automatically)
+
A new release build is made by (i586 example):
 +
* Running "copyprj -m "why" -H -b -n -p Core:PREVIOUS:i586 Core:NEWVERSION:i586
 +
* This results in a disabled publish/build project with the contents of Core:PREVIOUS:i586 in Core:NEWVERSION:i586
 +
* Change <link> in project meta to point to new MDS Core:NEWVERSION:i586
 +
* Enable builds first
 +
* Let build finish and wrap up
 +
* If build failed or anything unresolvable
 +
* Enable publishing
 +
* Disable builds
 +
* Disable publishing
 +
* Execute release script to do new release repository based on published OBS repo

Latest revision as of 16:40, 27 October 2012

The objective of the Mer change control processes is to ensure that each change is subjected to sensible QA checks - both automated and manual and to encourage Vendor participation in the QA process.

We will:

  • Verify every Mer release through continuous integration and QA
  • Provide a easy contribution model with fast feedback on changes
  • Have a distributed model of performing QA checks, so we can easily scale operations and support a variety of hardware

Contents

[edit] Summary

A contributor makes and submits a change which triggers the process.

When a change is proposed it must pass the following quality gates: (Each of these tests is described in more detail later.)

[edit] Core Checks

  • Packaging checks
  • Build check
  • ABI Checks
  • Trial build

[edit] Code Review (48hr window)

The change is available for manual review.

[edit] Vendor Checks: Early and Late (48hr window)

The change is now ready for automated review by vendors; first early notifications are sent, then after a while, late notifications.

Vendors run whatever tests they like; typically:

  • Image creation
  • Device flashing
  • Functional (OTS) tests

[edit] Manual testing (48hr window)

With such a short window there is limited opportunity for manual testing.

[edit] Maintainer Review

After 48hrs the maintainer looks at the submitted reviews and makes an accept/decline decision.

[edit] CI Automation

In order to verify that no obviously conflicting changes have occured the change is queued for automated acceptance.

[edit] Issues

Handling conflicts?

[edit] Change Controlled Code

We have:

  • A large amount of git repositories, with branches "master" (Trunk) and potentially release branches (Mer_1.0, Mer_2.0)
  • Each of these repositories checks out to be a .spec package, with tarball included, including _meta (OBS meta information) and _attribute (attributes)
  • "Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core

[edit] Contributor Process (abstracted)

When a contributor identifies that a change is required in a package they should:

  1. Clone the git repository for the package
  2. Make changes to the package
  3. Commit with a signed-off-by in the commit message
  4. Submit the change to the review process

They should now observe reviews and correct problems. This process is repeated until the change is approved and merged to the master git repository or until the proposal is rejected.

[edit] Reviewer process (abstracted)

  1. Read reviews of code changes on review site
  2. Perform own review and comment on review site

[edit] Maintainer process (abstracted)

  1. Read manual and automatic reviews
  2. Optionally do own review
  3. Approve/decline contribution on site
  4. Merge code change (manually or automatically)

[edit] Core Checks process (Automated)

When a proposed change comes in the following automated core checks are carried out and reported as 'reviews' of the change.

[edit] Packaging checks

Check if the proposed change is acceptable according to packaging guidelines.

[edit] Single-build test

Test if the proposed change builds on it's own or if the appropriate sha1 has been built on a public OBS.

[edit] Dependency-build test

Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly).

Following a dependency build there will be fresh packages to permit additional checks:

[edit] ABI checks

Test if the changed package causes any kind of breakage in ABI, utilizing ABI Compliance Checker.

[edit] Vendor notification process (Automated)

A notice is sent over various channels (eg email, AMQP, rss) to notify the vendor community that a change is available to be checked. The results of the dependency-build test are provided and the vendor can carry out vendor specific checks.

The objective of this notification step is to ensure that changes do not cause regressions in vendor devices; or that, if applied to Core, vendors will be aware of problems that arise)

This is a 2-stage process; certain early notification vendors are told about the change right away, if there are no bad reviews within 12 hours, we notify late vendors. Reviews must be provided within the 48 hr review window to be considered.

It is possible that Mer will operate a sample reference vendor using a 'virtual device' such as kvm.

[Question: What's the criteria for *not* sending late notifications? Is this a veto?] [Question: What kinds of checks should be allowed in the early notification review? Veto is quite powerful so it should probably be limited to "Doesn't build/boot"]

Typical gates here are:

[edit] Image creation test

Test if the repository result of the dependency-build test will create an 'image' succesfully, based on the vendor kickstart files.

[edit] Upgradeability check

Take packages from dependency-build test and attempt an upgrade on a suitable image (eg last official release), report back as code reviews.

[edit] Device flash and boot

Verify that any image is flashable and actually boots on a device to a certain level.


[edit] Functional (OTS) tests

If any functional or regression tests can be run then any failures may be provided.


[edit] Implementation DRAFT

Current implementation, OBS, Gerrit, BOSS

[edit] Contributor setup

1. Register account on http://review.merproject.org 2. Add SSH keys and set user name, real name

[edit] Contribution model (gerrit)

1. Clone git repository for package from ssh://username@review.merproject.org:29418/projectpath/packagepath

2. Do changes to package

3. Commit with signed-off-by in commit message, git commit -s -m "Message"

4. Submit patch to gerrit: git push origin HEAD:refs/for/master

5. Receive reviews and correct problems at http://review.merproject.org, until approved and merged to git repository

[edit] Reviewer model (gerrit)

1. Read automatic reviews of code change on http://review.merproject.org

2. Perform own review and comment on review site, use grading: 

                              -1 I would prefer that you didn't submit this
                               0 No score
                              +1 Looks good to me, but someone else must approve

[edit] Maintainer model (gerrit)

1. Read manual and automatic reviews

2. Optionally do own review, use +2 / -2 rating: 

                              -2 Do not submit
                              +2 Looks good to me, approved


3. Approve/decline contribution on site (Submit vs ..?)

4. Code change gets merged automatically

[edit] Automation

Documentation for gerrit hooks: http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.0/config-hooks.html

[edit] Gerrit hook: patchset-created 

patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id>

[edit] Things to make life easier

If the OBS can identify the sha1 of a package then a sha1 built in a public OBS against the correct repos is acceptable

[edit] Release Process

Mer has weekly releases and goes through three phases: Pickup, Prerelease, Release.

[edit] Pickup

Pickup happens on Friday 00:01 UTC. A pickup is basically to grab the most recent commit SHA1 from each package git, update the Core with this new commit ID and initiate a release rebuild, which should take at most 48 hours, that is, finished on Saturday 23:59 UTC

[edit] Prerelease

Prerelease happens on Sunday 00:01 UTC. Prerelease consists of generating a release and providing this to vendors for QA purposes in order to catch issues before release, in order to see what changes ought to be pulled out before release. This allows vendors to do automated builds during sunday and be ready for monday.

[edit] Release

Release happens on Thursday 00:01 UTC. Release consists of the prerelease with eventual fixes and pulled out changes.

[edit] Draft new release process model

[edit] project-core branches and tags

mer/project-core.git is a description of the contents in the Mer Core, that is, xml files indicating what packages exist, what commit ID corresponds to their current contents, OBS project configurations, etc.

During development, the master branch is being updated continuously as new changes are merged in packages on their respective master branches.

When a pre-release is being prepared, it will create a branch off master for stabilization towards weekly release. Stabilisation fixes can then be pushed through gerrit.

The Mer CI OBS will have these ongoing builds: 

 Core:ARCH -> master. At completion of each verified build, it will trigger a publish and increment the Version

 Core:ARCH:next -> current prerelease branch. At completion of each verified build, it will trigger a publish and increment the Version

The fakeobs will have these projects available: 

 Core:ARCH -> master, binaries will be the latest complete Core:ARCH build from Mer CI OBS.

 Core:ARCH:next -> current prerelease branch, binaries will be latest completed Core:ARCH:next build from Mer CI

 Core:ARCH:<Version> -> Specific version; not all Versions will always be available.

CI testing is being done against Core:ARCH in Core always.

To propose changes to project configurations, etc, submit changes to refs/for/master.

[edit] Version numbers

RELEASEDATE = Always a Thursday 

0.RELEASEDATE.0.0.0 -> First 'master' completed build after latest release, next release being RELEASEDATE

0.RELEASEDATE.0.1.0 -> Prerelease branch for release that'll happen on RELEASEDATE, first release

0.RELEASEDATE.0.1.1 -> Prerelease branch for release that'll happen on RELEASEDATE, second release

                 .2 -> etc

0.RELEASEDATE.1     -> Release on RELEASEDATE

If a new release on 0.RELEASEDATE stabilization branch:

0.RELEASEDATE.1.0.1 -> First 0.RELEASEDATE 'master' build after 0.RELEASEDATE.1 release

0.RELEASEDATE.1.1.0 -> First Prerelease for new release on 0.RELEASEDATE stabilization branch

0.RELEASEDATE.2     -> Release for 0.RELEASEDATE stabilization branch

0.LATERRELEASEDATE.1 -> New release on 0 branch

If invasive change/major release new work begins:

1.RELEASEDATE.0.0.0 -> Invasive change, drop in quality OK, first 'master' completed build after 1.RELEASEDATE

.. etc

[edit] New process 2

We have a new method in OBS that allows us to full-copy a project with content to a new project. This enables us to do an abstraction of 'builds'.

A release build is described as:

  • A project core, along with source, hosted on a MDS instance. In practice, this is a <link project=> to a OBS project hosted within MDS.
  • A set of binaries that has been built which corresponds to the source

A new release build is made by (i586 example):

  • Running "copyprj -m "why" -H -b -n -p Core:PREVIOUS:i586 Core:NEWVERSION:i586
  • This results in a disabled publish/build project with the contents of Core:PREVIOUS:i586 in Core:NEWVERSION:i586
  • Change <link> in project meta to point to new MDS Core:NEWVERSION:i586
  • Enable builds first
  • Let build finish and wrap up
  • If build failed or anything unresolvable
  • Enable publishing
  • Disable builds
  • Disable publishing
  • Execute release script to do new release repository based on published OBS repo
Retrieved from "https://wiki.merproject.org/wiki/index.php?title=Process&oldid=2661"
Personal tools