Process

Mer Process

The objective of the Mer change control processes is to ensure that each change is subjected to sensible QA checks - both automated and manual and to encourage Vendor participation in the QA process.

We will:

Verify every Mer release through continuous integration and QA
Provide a easy contribution model with fast feedback on changes
Have a distributed model of performing QA checks, so we can easily scale operations and support a variety of hardware

Summary

A contributor makes and submits a change which triggers the process.

When a change is proposed it must pass the following quality gates: (Each of these tests is described in more detail later.)

Core Checks

Packaging checks
Build check
ABI Checks
Trial build

Code Review (48hr window)

The change is available for manual review.

Vendor Checks: Early and Late (48hr window)

The change is now ready for automated review by vendors; first early notifications are sent, then after a while, late notifications.

Vendors run whatever tests they like; typically:

Image creation
Device flashing
Functional (OTS) tests

Manual testing (48hr window)

With such a short window there is limited opportunity for manual testing.

Maintainer Review

After 48hrs the maintainer looks at the submitted reviews and makes an accept/decline decision.

CI Automation

In order to verify that no obviously conflicting changes have occured the change is queued for automated acceptance.

Issues

Handling conflicts?

Change Controlled Code

We have:

A large amount of git repositories, with branches "master" (Trunk) and potentially release branches (Mer_1.0, Mer_2.0)
Each of these repositories checks out to be a .spec package, with tarball included, including _meta (OBS meta information) and _attribute (attributes)
"Core" git repository, which contains .xml files and other OBS configuration along with pointers to specific git repositories and commit IDs representing current state of Mer Core

Contributor Process (abstracted)

When a contributor identifies that a change is required in a package they should:

Clone the git repository for the package
Make changes to the package
Commit with a signed-off-by in the commit message
Submit the change to the review process

They should now observe reviews and correct problems. This process is repeated until the change is approved and merged to the master git repository or until the proposal is rejected.

Reviewer process (abstracted)

Read reviews of code changes on review site
Perform own review and comment on review site

Maintainer process (abstracted)

Read manual and automatic reviews
Optionally do own review
Approve/decline contribution on site
Merge code change (manually or automatically)

Core Checks process (Automated)

When a proposed change comes in the following automated core checks are carried out and reported as 'reviews' of the change.

Packaging checks

Check if the proposed change is acceptable according to packaging guidelines.

Single-build test

Test if the proposed change builds on it's own or if the appropriate sha1 has been built on a public OBS.

Dependency-build test

Test if the proposed change will break other packages in the Core by compiling the package and everything that depends on it (directly and indirectly).

Following a dependency build there will be fresh packages to permit additional checks:

ABI checks

Test if the changed package causes any kind of breakage in ABI, utilizing http://forge.ispras.ru/projects/abi-compliance-checker

Vendor notification process (Automated)

A notice is sent over various channels (eg email, AMQP, rss) to notify the vendor community that a change is available to be checked. The results of the dependency-build test are provided and the vendor can carry out vendor specific checks.

The objective of this notification step is to ensure that changes do not cause regressions in vendor devices; or that, if applied to Core, vendors will be aware of problems that arise)

This is a 2-stage process; certain early notification vendors are told about the change right away, if there are no bad reviews within 12 hours, we notify late vendors. Reviews must be provided within the 48 hr review window to be considered.

It is possible that Mer will operate a sample reference vendor using a 'virtual device' such as kvm.

[Question: What's the criteria for *not* sending late notifications? Is this a veto?] [Question: What kinds of checks should be allowed in the early notification review? Veto is quite powerful so it should probably be limited to "Doesn't build/boot"]

Typical gates here are:

Image creation test

Test if the repository result of the dependency-build test will create an 'image' succesfully, based on the vendor kickstart files.

Upgradeability check

Take packages from dependency-build test and attempt an upgrade on a suitable image (eg last official release), report back as code reviews.

Device flash and boot

Verify that any image is flashable and actually boots on a device to a certain level.

Functional (OTS) tests

If any functional or regression tests can be run then any failures may be provided.

Implementation DRAFT

Current implementation, OBS, Gerrit, BOSS

Contributor setup

1. Register account on http://review.merproject.org 2. Add SSH keys and set user name, real name

Contribution model (gerrit)

1. Clone git repository for package from ssh://username@review.merproject.org:29418/projectpath/packagepath

2. Do changes to package

3. Commit with signed-off-by in commit message, git commit -s -m "Message"

4. Submit patch to gerrit: git push origin HEAD:refs/for/master

5. Receive reviews and correct problems at http://review.merproject.org, until approved and merged to git repository

Reviewer model (gerrit)

1. Read automatic reviews of code change on http://review.merproject.org

2. Perform own review and comment on review site, use grading:

                              -1 I would prefer that you didn't submit this
                               0 No score
                              +1 Looks good to me, but someone else must approve

Maintainer model (gerrit)

1. Read manual and automatic reviews

2. Optionally do own review, use +2 / -2 rating:

                              -2 Do not submit
                              +2 Looks good to me, approved

3. Approve/decline contribution on site (Submit vs ..?)

4. Code change gets merged automatically

Automation

Documentation for gerrit hooks: http://gerrit-documentation.googlecode.com/svn/Documentation/2.2.0/config-hooks.html

Gerrit hook: patchset-created

patchset-created --change <change id> --change-url <change url> --project <project name> --branch <branch> --uploader <uploader> --commit <sha1> --patchset <patchset id>

Things to make life easier

If the OBS can identify the sha1 of a package then a sha1 built in a public OBS against the correct repos is acceptable

Release Process

Mer has weekly releases and goes through three phases: Pickup, Prerelease, Release.

Pickup

Pickup happens on Friday 00:00 UTC. A pickup is basically to grab the most recent commit SHA1 from each package git, update the Core with this new commit ID and initiate a release rebuild, which should take at most 48 hours, that is, finished on Saturday 23:59 UTC

Prerelease

Prerelease happens on Sunday 00:00 UTC. Prerelease consists of generating a release and providing this to vendors for QA purposes in order to catch issues before release, in order to see what changes ought to be pulled out before release.

Release

Release happens on Wednesday 00:00 UTC. Release consists of the prerelease with eventual fixes and pulled out changes.

Draft new release process model

project-core branches and tags

mer/project-core.git is a description of the contents in the Mer Core, that is, xml files indicating what packages exist, what commit ID corresponds to their current contents, OBS project configurations, etc.

During development, the master branch is being updated continuously as new changes are merged in packages on their respective master branches.

When a pre-release is being prepared, it will create a branch off master for stabilization towards weekly release. Stabilisation fixes can then be pushed through gerrit.

The Mer CI OBS will have these ongoing builds:

Core:ARCH -> master. At completion of each build, it will trigger a publish

Core:ARCH:next -> current prerelease branch. At completion of each build, it will trigger a publish.

The fakeobs will have these projects available:

Core:ARCH -> master, binaries will be the latest complete Core:ARCH build from Mer CI OBS. Core:ARCH:next -> current prerelease branch, binaries will be latest completed Core:ARCH:next build from Mer CI

CI testing is being done against Core:ARCH in Core always.

To propose changes to project configurations, etc, submit changes to refs/for/master.

Version numbers

RELEASEDATE = Always a Thursday

0.RELEASEDATE.0.0.0 -> First 'master' completed build after latest release

0.RELEASEDATE.0.1.0 -> Prerelease branch for RELEASEDATE, first release

0.RELEASEDATE.0.1.1 -> Prerelease branch for RELEASEDATE, second release

                 .2 -> etc

0.RELEASEDATE.1     -> Release for RELEASEDATE

If a new release on 0.RELEASEDATE stabilization branch:

0.RELEASEDATE.1.0.1 -> First 0.RELEASEDATE 'master' build after 0.RELEASEDATE.1 release

0.RELEASEDATE.1.1.0 -> First Prerelease for new release on 0.RELEASEDATE stabilization branch

0.RELEASEDATE.2     -> Release for 0.RELEASEDATE stabilization branch

0.LATERRELEASEDATE.1 -> New release on 0 branch

If invasive change/major release new work begins:

1.RELEASEDATE.0.0.0 -> Invasive change, drop in quality OK, first 'master' completed build after 1.RELEASEDATE

.. etc