Revision as of 20:58, 24 January 2012 (view source) Lbt (Talk | contribs) m (FakeOBS moved to Mer Delivery System: Rename FakeOBS to Mer Delivery System) ← Older edit		Latest revision as of 01:08, 14 June 2012 (view source) Sonach (Talk | contribs) (→‎Installation)
(20 intermediate revisions by 4 users not shown)
Line 1:		Line 1:
−	To set up your own Mer build:	+	Mer provides the Mer Delivery System (MDS) to make it easy for vendors to track upstream Mer releases.
−	1. Install:	+	= Overview =
		+
		+	MDS is a small collection of tools that both supports distributing Mer
		+	binary releases and OBS build targets to vendors and accessing the
		+	local copy of the release from a private vendor-managed OBS.
		+
		+	= Operations =
		+
		+	== MDS tools ==
		+
		+	=== Installation ===
		+
		+	The MDS is installed in the root of the data area which will store the
		+	Mer releases (several Gb). This may be a dedicated VM.
		+
		+	Ensure the system has the following installed:
	* git		* git
Line 13:		Line 28:
	* cpio		* cpio
−	2. Get the release tools	+	On Debian this can be done as follows (although pip is not a preferred Debian solution):
		+
		+	aptitude install python-dev build-essential python-pip ca-certificates git git-core libcurl3-gnutls liberror-perl libidn11
		+	pip install smmap
		+	pip install async
		+	pip install gitdb
		+	pip install GitPython
		+
		+	Under OpenSUSE11.4, it is a better way to install smmap/async/gitdb/GitPython as following:
		+	zypper install python-pip
		+	pip install smmap
		+	pip install async
		+	pip install gitdb
		+	pip install GitPython
		+
		+
		+	Get the release tools
		+	mkdir -p /srv/mer/mds/
		+	cd /srv/mer/mds/
	git clone http://review.merproject.org/p/mer/release-tools		git clone http://review.merproject.org/p/mer/release-tools
−	3. Build the tools	+	=== Update ===
−	Note: You might need to set rsync proxy:	+
		+	The tools are updated periodically - simply run
		+	git pull
		+
		+	Eventually:
		+	git remote update
		+	to update and then
		+	git checkout vXXX
		+	to obtain the version required.
		+
		+	== Mer release(s) ==
		+
		+	=== Setup ===
		+
		+	This step will pull (using rsync) all the git repositories for the Mer
		+	packages (nb: this is a huge download).
		+
		+	It will then build a list of the packages, create a set of mappings
		+	and prepare the OBS event stream.
		+
		+	Finally the OBS projects are cloned and extracted ready for the API
		+	service.
		+
		+	Note: You might need to set rsync proxy:
	export RSYNC_PROXY=proxy.company.com:8080		export RSYNC_PROXY=proxy.company.com:8080
−	This will rsync all the package git repos and clone Core git and check it out	+	cd release-tools; make; make update
−	and make a mappings cache of git repos (will md5sum all sources)	+
−	cd release-tools; make	+	Note: after doing "make", you "make update" grabs the rpm binaries.
−	4. Pull down the latest Mer release binaries	+	=== Update ===
−	make update	+	An update simply requires:
−	5. Start the fakeobs server	+	cd release-tools; make update
		+
		+	This will re-synchnronise the package git repos, the OBS projects and
		+	reset the 'latest' link.
		+
		+	It can be run periodically to ensure that up-to-date changes are
		+	available.
		+
		+	There is no mechanism at the moment to specify that certain releases
		+	should be made available.
		+
		+	You might have to, after 'make update' to go into obs-project/Core and do 'git checkout db08d70258b5a53689fcee3d3deb94926015ff26', as to get the right version of core.
		+
		+	== Setup an API service for Mer releases ==
		+
		+	MDS runs a simple http server that emulates the OBS api for a static release.
		+
		+	To start this 'fake obs server' (hence the original name for the packages):
	python tools/fakeobs.py 8001 &		python tools/fakeobs.py 8001 &
−	6. Setup OBS locally on your build environment of choice:	+	Note: You should do this under "release-tools" directory, since fakeobs.py will read mappings.xml under "release-tools" directory.
−	* As VMWare image	+
−	** download -vmx- image [http://download.opensuse.org/repositories/openSUSE:/Tools/images/ from here]	+	=== Add the MDS project as a remote link ===
−	** load into VMWare Player	+
−	** add Network Adapter (Bridged mode!)	+	A local OBS can now add a 'remote link' to the MDS API service
−	** add extra min 12GB HDD and mount it under /var/cache	+	(FakeOBS) port 8001, /public, note, http, not-https. So assuming the
−	** continue to setup [[Mer_OBS_Setup_Guide#Setting_Up_the_OBS_Install|here, stop before step 5]]	+	MDS is running on a machine called 'mer' then this would work.
−	* On an existing openSUSE installation ([[OBS_Setup|OBS Setup]])	+
−	* Building only on VMs: [[Mer_OBS_VM_Setup_Guide|Mer OBS VM Setup Guide]].	+	<project name="MerDS">
		+	<title>Mer Delivery System</title>
		+	<description>A 'remote link' to the MDS API service running against a local copy of Mer
		+	</description>
		+	<remoteurl>http://mer:8001/public</remoteurl>
		+	<person userid="Admin" role="maintainer"/>
		+	<person userid="Admin" role="bugowner"/>
		+	</project>
		+
		+	=== MDS Build Targets ===
		+
		+	Mer supports a number of build targets for various release points and architectures.
		+
		+	These are organised as:
		+	<link>:<area>:<arch>:<release>
		+
		+	So the build target:
		+	MerDS:Core:armv6l:0.20120120.1
		+	would build against the collection of packages that forms 'Core' built
		+	for the [[arch|armv6l architecture]].
		+
		+	=== Use MDS repos as a build target ===
		+
		+	Once your project is set-up, you can add Mer build targets to the OBS
		+	and they should start building automatically. Given a setup as above
		+	then a project meta would contain repositories similar to this:
		+
		+	<project name="MyMerUx">
		+	<title>MyMerUx</title>
		+	<description>
		+	A UX building against Mer
		+	</description>
		+	<person userid="Admin" role="maintainer"/>
		+	<person userid="Admin" role="bugowner"/>
		+	<repository name="MyMerUX_Mer_Core_i586">
		+	<path repository="Core_i586" project="MerDS:Core:i586"/>
		+	<arch>i586</arch>
		+	</repository>
		+	</project>
		+
		+	== Running under Systemd ==
		+
		+	MDS can (and should) be run as a service. This sample systemd service file may be useful:
		+
		+	in /etc/systemd/system/mds.service
		+
		+	[Unit]
		+	Description=Mer Delivery System
		+	After=multi-user.target
		+
		+	[Service]
		+	User=mds
		+	Group=mds
		+	WorkingDirectory=/srv/mer/mds/release-tools/
		+	ExecStart=/usr/bin/python /srv/mer/mds/release-tools/tools/fakeobs.py 8001
		+
		+	[Install]
		+	WantedBy=multi-user.target
		+
		+
		+	Then:
		+	systemctl enable mds.service
		+	systemctl start mds.service
		+	systemctl status mds.service
		+
		+	= Limitations =
		+
		+	There is no way to limit MDS to only certain architectures.
		+
		+	There is no way to see the list of build target releases or architectures.
		+
		+
		+	You can [https://bugs.merproject.org/enter_bug.cgi?product=Mer%20Core log a bug] (against the .Release-tools component) or show up in #mer and
		+	poke Stskeeps if there's any questions.
		+
		+	[[Category:mds]][[Category:obs]]
		+
		+	= MDS2 =
−	7. Add a remote link for your FakeOBS's server, port 8001, /public, http, not-https ([[Mer_OBS_Setup_Guide#Add_a_remote_link|see how]])	+	'''DRAFT'''
−	8. Follow the rest of [[Mer_OBS_Setup_Guide#Setting_Up_the_OBS_Install|OBS Install]] until the end	+	Mer Delivery System 2.0 is a new approach to the MDS idea. Instead of utilizing checked out (in the git sense) project cores to know what OBS projects & packages it has to present, it now has the ability to access any commit/tag/git reference inside a git repository.
−	9. Once your project is set-up, OBS should start building it automatically (use Status Monitor to track the progress)	+	== mappings.xml ==
		+	Instead of a
		+	<mapping project="Core:i586" path="obs-projects/Core/i586" binaries="obs-repos/Core:i586:latest" reponame="Core_i586" />
		+	a different form is used:
		+	<mapping project="Core" path="packages-git/mer/mer-core" />
−	Show up in #mer and poke Stskeeps if there's any questions	+	When querying MDS through it's OBS API, it will now respond to requests for Core:GIT_REFERENCE:SUBDIR. This means, to find which packages to respond with, it will look in mappings.xml for "Core", find the GIT_REFERENCE commit in packages-git/mer/mer-core and extract packages.xml from subdir/ in the tree.

---

Latest revision as of 01:08, 14 June 2012

Mer provides the Mer Delivery System (MDS) to make it easy for vendors to track upstream Mer releases.

Contents 1 Overview 2 Operations 2.1 MDS tools 2.1.1 Installation 2.1.2 Update 2.2 Mer release(s) 2.2.1 Setup 2.2.2 Update 2.3 Setup an API service for Mer releases 2.3.1 Add the MDS project as a remote link 2.3.2 MDS Build Targets 2.3.3 Use MDS repos as a build target 2.4 Running under Systemd 3 Limitations 4 MDS2 4.1 mappings.xml

[edit] Overview

MDS is a small collection of tools that both supports distributing Mer binary releases and OBS build targets to vendors and accessing the local copy of the release from a private vendor-managed OBS.

[edit] Operations

[edit] MDS tools

[edit] Installation

The MDS is installed in the root of the data area which will store the Mer releases (several Gb). This may be a dedicated VM.

Ensure the system has the following installed:

• git
• python
• make
• smmap: http://pypi.python.org/pypi/smmap
• async: http://pypi.python.org/pypi/async
• gitdb: http://pypi.python.org/pypi/gitdb
• gitpython homepage: http://pypi.python.org/pypi/GitPython/0.3.2.RC1
• gitpython install guide: http://packages.python.org/GitPython/0.3.2/intro.html#installing-gitpython
• cpio

On Debian this can be done as follows (although pip is not a preferred Debian solution):

aptitude install python-dev build-essential python-pip ca-certificates git git-core libcurl3-gnutls liberror-perl libidn11
pip install smmap
pip install async
pip install gitdb
pip install GitPython

Under OpenSUSE11.4, it is a better way to install smmap/async/gitdb/GitPython as following:

zypper install python-pip
pip install smmap
pip install async
pip install gitdb
pip install GitPython

Get the release tools

mkdir -p /srv/mer/mds/
cd /srv/mer/mds/
git clone http://review.merproject.org/p/mer/release-tools

[edit] Update

The tools are updated periodically - simply run

git pull

Eventually:

git remote update

to update and then

git checkout vXXX

to obtain the version required.

[edit] Mer release(s)

[edit] Setup

This step will pull (using rsync) all the git repositories for the Mer packages (nb: this is a huge download).

It will then build a list of the packages, create a set of mappings and prepare the OBS event stream.

Finally the OBS projects are cloned and extracted ready for the API service.

Note: You might need to set rsync proxy:

export RSYNC_PROXY=proxy.company.com:8080

cd release-tools; make; make update

Note: after doing "make", you "make update" grabs the rpm binaries.

[edit] Update

An update simply requires:

cd release-tools; make update

This will re-synchnronise the package git repos, the OBS projects and reset the 'latest' link.

It can be run periodically to ensure that up-to-date changes are available.

There is no mechanism at the moment to specify that certain releases should be made available.

You might have to, after 'make update' to go into obs-project/Core and do 'git checkout db08d70258b5a53689fcee3d3deb94926015ff26', as to get the right version of core.

[edit] Setup an API service for Mer releases

MDS runs a simple http server that emulates the OBS api for a static release.

To start this 'fake obs server' (hence the original name for the packages):

python tools/fakeobs.py 8001 &

Note: You should do this under "release-tools" directory, since fakeobs.py will read mappings.xml under "release-tools" directory.

[edit] Add the MDS project as a remote link

A local OBS can now add a 'remote link' to the MDS API service (FakeOBS) port 8001, /public, note, http, not-https. So assuming the MDS is running on a machine called 'mer' then this would work.

<project name="MerDS">
 <title>Mer Delivery System</title>
 <description>A 'remote link' to the MDS API service running against a local copy of Mer
 </description>
 <remoteurl>http://mer:8001/public</remoteurl>
 <person userid="Admin" role="maintainer"/>
 <person userid="Admin" role="bugowner"/>
</project>

[edit] MDS Build Targets

Mer supports a number of build targets for various release points and architectures.

These are organised as:

<link>:<area>:<arch>:<release>

So the build target:

MerDS:Core:armv6l:0.20120120.1

would build against the collection of packages that forms 'Core' built for the armv6l architecture.

[edit] Use MDS repos as a build target

Once your project is set-up, you can add Mer build targets to the OBS and they should start building automatically. Given a setup as above then a project meta would contain repositories similar to this:

<project name="MyMerUx">
  <title>MyMerUx</title>
  <description>
  A UX building against Mer
  </description>
  <person userid="Admin" role="maintainer"/>
  <person userid="Admin" role="bugowner"/>
  <repository name="MyMerUX_Mer_Core_i586">
    <path repository="Core_i586" project="MerDS:Core:i586"/>
    <arch>i586</arch>
  </repository>
</project>

[edit] Running under Systemd

MDS can (and should) be run as a service. This sample systemd service file may be useful:

in /etc/systemd/system/mds.service

[Unit]
Description=Mer Delivery System
After=multi-user.target

[Service]
User=mds
Group=mds
WorkingDirectory=/srv/mer/mds/release-tools/
ExecStart=/usr/bin/python /srv/mer/mds/release-tools/tools/fakeobs.py 8001

[Install]
WantedBy=multi-user.target

Then:

systemctl enable mds.service
systemctl start mds.service
systemctl status mds.service

[edit] Limitations

There is no way to limit MDS to only certain architectures.

There is no way to see the list of build target releases or architectures.

You can log a bug (against the .Release-tools component) or show up in #mer and poke Stskeeps if there's any questions.

[edit] MDS2

DRAFT

Mer Delivery System 2.0 is a new approach to the MDS idea. Instead of utilizing checked out (in the git sense) project cores to know what OBS projects & packages it has to present, it now has the ability to access any commit/tag/git reference inside a git repository.

[edit] mappings.xml

Instead of a

 <mapping project="Core:i586" path="obs-projects/Core/i586" binaries="obs-repos/Core:i586:latest" reponame="Core_i586" /> 

a different form is used:

 <mapping project="Core" path="packages-git/mer/mer-core" />

When querying MDS through it's OBS API, it will now respond to requests for Core:GIT_REFERENCE:SUBDIR. This means, to find which packages to respond with, it will look in mappings.xml for "Core", find the GIT_REFERENCE commit in packages-git/mer/mer-core and extract packages.xml from subdir/ in the tree.