The Mer Wiki now uses your Mer user account and password (create account on https://bugs.merproject.org/)


Image Creation

From Mer Wiki
Revision as of 21:36, 30 September 2012 by Lbt (Talk | contribs)

Jump to: navigation, search

This page has not been fully updated for Mer. Please edit to remove meego specific things and convert to mic terminology

This is the main Image Creator developer's guide. For a more simplistic, step-by-step document, go to: Image Creation For Beginners.

Contents

Overview

The tool used to create Mer images is called "MIC" (to distinguish from obsolete MIC - Moblin Image Creator and the equally obsolete MIC2). MIC is composed of a series of tools to create images, convert images, chroot, etc. MIC is primarily based on Fedora livecd-tools and appliance-tools.

With MIC tools, users can create different types of images for different purposes, including live CD images, live USB images, raw images for KVM, VMDK images for Vmware, vdi images for VirtualBox, loop images for IVI platforms, NAND images for Moorestown platforms, ubi images for N900, fs image for MeeGo developers. Also, users can use MIC tools to manipulate images, like transforming an image from a virtual machine to a live image, and providing a chroot environment based on an existing live image. With these features, developers can do development work on a host virtual machine running MeeGo or a Meego chroot environment, and transfer the resulting new live image to a target device for final debug/verification.

Features

MIC offers these major tools:

  • mic-image-creator: create images.
  • mic-image-convertor: convert a raw/vmdk/vdi/live image into a live image.
  • mic-chroot: provide a MeeGo environment from a live/loop image for development, it also can translate that chroot file system into a live image.
  • mic-image-writer: write a MeeGo image to a USB disk. This is a safe alternative to dd.

The following support is provided:

  • Supports various types of images: livecd, liveusb, loop, raw, vmdk, nandmrst, vdi, fs, ubi
  • Supports image coversion from virtual machine/live images to live images.
  • Uses kickstart (.ks) files for image creation. Through this, users can specify which software repositories to use, which packages to install, and basic system configuration directives. Please refer to http://fedoraproject.org/wiki/Anaconda/Kickstart for more information about kickstart configuration files.

Usage

Installation

mic is preinstalled in the Mer SDK

You should load these modules into the host if they're not loaded automatically by the kernel:
(Use lsmod to list modules.)

  • squashfs
  • squashfs-tools
  • dm_snapshot
  • loop

Add gpg key as user root:

# gpg2 --keyserver subkeys.pgp.net --recv 0BC7BEC479FC1F8A
# gpg2 --export --armor 0BC7BEC479FC1F8A > /etc/pki/rpm-gpg/RPM-GPG-KEY-meego

Proxy Setting

If you need to use proxy to access Internet, you must set proxy for mic correctly. Generally, you should export these environment variables:

$ export http_proxy="http://proxy.yourcompany.com:8888"
$ export https_proxy="http://proxy.yourcompany.com:8888"
$ export no_proxy="127.0.0.0/8, .yourcompany.com"

You also can set proxy in /etc/mic/mic.conf or in ~/.mic.conf by adding the below two lines:

proxy=http://proxy.yourcompany.com:8888/
no_proxy=localhost,127.0.0.0/8,.yourcompany.com

Of course, you need to change proxy.yourcompany.com:8888 and .yourcompany.com according to your network.

Running mic-image-creator

Configuration of images is based on kickstart, the format used for unattended installation in Fedora and Redhat.

Super user privileges are needed. The tool is more or less self-documented, use the --help option to see options.

sudo mic-image-creator --help

Creating Supported Image Types

KickStart (.ks) configuration files are passed to MIC to create tailored images. KickStart files specify what repos to pull from, what packages to include, what post-scripts to run and what type of images to create.

creating arm images needs 'qemu-arm' os 'qemu-arm-static' command available, and creating vdi or vmdk images needs 'vboxmanager' available, so be sure your system have installed them before creating these types of images.

To obtain the official Meego .ks files, go here: http://wiki.meego.com/Image_Creation#Official_Meego_.ks_files

Create ARMv7 Images

When creating images for ARMv7 hardware, pass --arch=armv7hl to mic-image-creator commands.

Create Livecd Image

sudo mic-image-creator --config=default.ks --format=livecd --cache=mycache

This tells image-creator to use the kickstart file default.ks to obtain info about which packages to download and include in the image, and --cache is the directory on your local machine which will host a cache of these packages. The cache is very useful if you are remote to the server and reduces the amount of downloaded packages next time you create an image. Next time the command is run, this cache will simply be 'updated' if there are changes, rather than re-downloading from the repositories again.

The output of this command will be a file named meego-1.2-default-XX.iso created. This ISO image is a hybrid image and can be either written to a disk device or burned onto a cd.

To burn it onto a USB stick, just run the following command, assuming the USB stick is /dev/sdb in your system:

sudo mic-image-writer meego-1.2-default-XX.iso

Create Moorestown NAND Image

sudo mic-image-creator --config=default.ks --format=mrstnand

remove the first 512 bytes from the image to remove the partition information.
sudo dd bs=512 skip=1 if=meego-1.2-default-XX-sda.bin of= meego-1.2-default-XX-nand.img

Create Liveusb Image

sudo mic-image-creator --config=default.ks --format=liveusb --cache=mycache

A file named meego-1.0-default-XX.usbimg will be created. To burn it onto a USB stick, run the following command, assuming the USB stick is /dev/sdb in your system:

sudo mic-image-writer meego-1.2-default-XX.usbimg

This image has a FAT file system and can be mounted easily on Windows and other OSes.

Create Liveusb Image Interactively

sudo mic-image-creator --config=default.ks --format=liveusb --interactive --cache=mycache

It directly creates a MeeGo live usb stick.

Liveusb image can be created in two different ways, interactive and non-interactive (default). When running MIC in a non-interactive mode, MIC creates a USB images file that can be copied directly into a USB stick. The non-interactive mode can be used for daily build or automated testing. The interactive mode will write the images onto a USB disk, it will detect if a USB stick is available and detects and verifies a partition is available. If two or more USB devices are present, you will be asked to select the target device. This way the contents of the existing USB stick won't be destroyed unless no appropriate partition is found, end user should use it to create the live USB.

Create Loop Image

This is the simplest image format available. Such images can be loop mounted and chrooted into, for example, to build applications or for debugging purposes.

sudo mic-image-creator --config=default.ks --format=loop --cache=mycache

A file named meego-1.2-default-XX.img is created. You can use below to mount it and chroot into it.

sudo mount -o loop meego-1.2-default-XX.img /mnt
sudo chroot /mnt su -

Create KVM Image

Can be used with QEMU or other VMM applications to launch MeeGo as a virtualized instance.

sudo mic-image-creator --config=default.ks --format=raw --cache=mycache

A file named meego-1.2-default-XX folder with a meego-1.2-default-XX-sda.raw image is created. For optimal results, use this feature in a machine with VT support and enable it in the BIOS.

If you use Fedora or openSUSE, run the following command to launch the image into MeeGo KVM virtual machine:

sudo qemu-kvm -m 512 -boot c -hda meego-1.2-default-XX-sda.raw -std-vga

If you use Ubuntu, run the following command launch image:

sudo kvm -m 512 -boot c -hda meego-1.2-default-XX-sda.raw

Create VMDK Image

VMDK images can be loaded into Vmware or Vmware player. MIC also generates a VMX file that has image configuration and can be used to launch the image in Vmware by just clicking the file. If you have vmplayer, it should open automatically.

sudo mic-image-creator --config=default.ks --format=vmdk --cache=mycache

A file named meego-1.2-default-XX folder with both meego-1.2-default-XX-sda.vmdk image and meego-1.2-default-XX-sda.vmx is created. Just run vmware or vmware player, and select the generated vmx file.

KickStart Files (configuration files used for image creation)

MIC Configuration file

MIC options can be specified in a global configuration file (/etc/mic/mic.conf) and local configuration file ($HOME/.mic.conf), where you can specify:

  • Repository configuration
  • proxy settings
  • cache directories and
  • other variables

that you normally would have to re-type in your command-line.

The two configuration files have the same format, /etc/mic/mic.conf is global, no matter whoever is running MIC, this configuration file will be read first, $HOME/.mic.conf is just valid for current user, every user can have their own settings. The final configuration is a combination of two configuration files, options in $HOME/.mic.conf will override those from /etc/mic/mic.conf. Options from command line have the highest priority. If an option is specified on the command line, the same option in both configuration files, global and local will be ignored.

Below is an example you can cut-and-paste, and reuse for your needs:

[main]
cachedir=/home/user1/mycache
tmpdir=/home/user1/mystorage/tmp
outdir=/home/user1/mystorage
proxy=http://my.proxy.com:911/
no_proxy=localhost,127.0.0.0/8,.mysite.com,172.16.0.0/16

cachedir = directory where the cached repo(s) will reside. With this variable set, you do not need to pass the --cache flag in the command-line.

tmpdir = temporary directory used by MIC when creating images. With this variable set, you do not need to pass the --tmpdir flag in the command-line.

outdir = where your images will reside once they are created. With this variable set, you do not need to pass the --outdir flag in the command-line.

proxy = specify your proxy if you're behind a behind a firewall.

no_proxy = specify what domains should not sure the proxy setting.

image_format = specify image format.

default_ks = specify a default kickstart file, there are several kickstart files provided in repository http://repo.meego.com/MeeGo/devel/trunk/repo/ia32/os/

Note: When specifying proxy and no_proxy, you do not need to use the --proxy flag in your .ks files when referring to repos.


In configuration files, you also can specify one or multiple repositories, for example:

[trunk]
name=trunk
baseurl=http://repo.meego.com/MeeGo/devel/trunk/repo/ia32/os/
enabled=1

Once there is this repository info in your configuration files, and both image_format and default_ks are set, you can just run "sudo mic-image-creator" to create a image, there isn't any extra argument needed.

Use Bootstrap

MIC highly depends on yum and librpm, but different Linux distributions used diffrent yum and librpm versions, so MIC has some compatibility issues on OpenSUSE, Ubuntu and Debian, bootstrap is just for fixing these compatibility issues.

bootstrap is a minimal MeeGo file system, MIC can run very smoothly on it using chroot mode. You can use the below command to create a bootstrap:

sudo mic-create-bootstrap -n trunk -k /your/repo/cache/path -r http://repo.meego.com/MeeGo/builds/trunk/latest/repos/oss/ia32/packages/ -o /your/final/bootstrap

NOTE: The repo-url specific for your needs may differ

-n is used to specify repository name, if you have its cache before, it can reduce runtime dramatically to use it with -k option, -k is used to specify your repository cache dir, -r is used to specify MeeGo main repository.

Once you created your bootstrap, you can use this bootstrap to run MIC as follows

sudo mic-image-creator --bootstrap=/your/final/bootstrap --format=livecd --config=default.ks --cache=/your/repo/cache/path


Alternatively, you can create and use bootstrap in one mic-image-creator run, as follows

sudo mic-image-creator --build-bootstrap --bootstrap=/your/final/bootstrap --format=livecd --config=default.ks --cache=/your/repo/cache/path

Running mic-image-writer

mic-image-writer writes a Meego image to a USB disk as an alternative to dd.

mic-image-writer can run in both console mode and GUI mode. It can decide which mode to run, according to current system environment. You can also use a given option to force it to run in some other mode. Run 'mic-image-writer --help' to get usage information:

Usage: mic-image-writer [options] [image file]

Options:
  -h, --help     Show this help message and exit
  -c, --console  Run in console mode
  -g, --gui      Run in GUI mod

Here is a run example in console mode:

$ sudo mic-image-writer meego-xxx.img
Available usb disk:
        [1] /dev/sdc: SanDisk USB Flash Drive
        [2] /dev/sdb: SanDisk U3 Cruzer Micro
Please choice [1..2] ? 2
Source: /myhome/meego-xxx.img
Target: /dev/sdb
Image size: 559 MB
Estimated time: 55 seconds
Elapsed time: 64, progress: 100% 8944+0 records in
8944+0 records out
586153984 bytes (586 MB) copied, 63.0486 s, 9.3 MB/s

mic-image-writer can estimate how long it will take to write the given image to your USB disk and reports current writing progress. It automatically unmounts your USB disk, if it is mounted. If your USB can't be unmounted, it will terminate.

If you love UI mode, you can run it in X enviroment. The command with the added -g flag:

sudo mic-image-writer -g meego-xxx.img

Image name is optional.

Running mic-image-convertor

It's very easy to use:

sudo mic-image-convertor --source-image=InputImage --target-format=Targegformat

sudo mic-image-convertor -I InputImage -T Targegformat

For example, to translate a KVM raw image to livecd image, just type:

sudo mic-image-convertor --source-image=meego-core-200902200545/meego-core-200902200545-sda.raw --target-format=livecd

A new livecd image of meego-converted-from-raw-200902201804.iso is generated. The tool can detect the type of input image, either raw or vmdk.

This tool is very useful. For example, the developer can launch a virtual machine running MeeGo v2 and make whatever changes on the virtual system like yum install/remove a package, scp a source tarball to VM and build and try, and etc. Then with a simple "sync" or shutdown of VM, the VM image now contains his changes. With this tool, the VM image can be transformed to a live image and burned to a USB flash disk. Now developers can have final verifications in a target device, with changes in the live system.

Running mic-chroot

There are two major usage models for mic-chroot for developers.

chroot directly into the image to use it as a development environment, then optionally create a new image based off of your changes

sudo mic-chroot  -c livecd meego-core-200903131337.iso

The above command would present a chroot, using the livecd image, to developers with some bind mounts like /proc /sys /dev/pts and /parentroot. With those bind mounts, developers now can easily exchange files in chroot env with host /parentroot, and conduct yum install, yum remove and any other network related operations. After done and typing "exit", a new live ISO image is created from the chroot env with the changes developers made.

Unpack and modify the image's filesystem (which you can save), and create a new image based off of your changes

Sometimes, developers want to keep the chroot env, so that they can do multiple changes in that root file system at different times. So they want to execute following:

sudo mic-chroot -s my-chroot-fs --unpack-only  meego-core-200903131337.iso --bind-mounts=/proc:/proc;/:/parentroot;/sys:/sys;/dev/pts:/dev/pts

The above command creates a folder called 'my-chroot-fs' in the current directory, using the chroot environment from the livecd image. Of course, if one forgets to add the --bind-mounts options, you'll need to manually do the bind mounting yourself. Don't forget copying /etc/reslov.conf into my-chroot-fs if you need to use DNS. Should you wish to create an image from the chroot, just execute following to create a livecd image

sudo mic-chroot -c livecd --convert-only my-chroot-fs/

Enabling Autoinstallation

You need to copy a kickstart file into /root/mic-ks.cfg which is used for automated installation of the image. To activate autoinstall mode, boot with autoinst on the command line or add --menus=autoinst to the bootloader directive in the kickstart file you use to create the image.

Development Release Processes

http://wiki.meego.com/MIC_Development_Release_Processes

Known issues

  • MIC is not compatible with Ubuntu 8.04 (Yum package compatibility)
  • MIC is not compatible with the initial Ubuntu 9.04. Make sure you have package libsqlite3-0 of 3.6.10-1ubuntu0.2 or later.
  • MIC has some issues on Debian, please refer to MIC on Debian to get installation instructions

Troubleshooting

You can file a bug on http://bugzilla.meego.com/enter_bug.cgi?product=Development%20Tools (Note: you should select MIC for component) if you run into any MIC-related issue, bugzilla can track your issue faster and more efficiently.

libgcc related issues

libgcc_s.so.1 must be installed for pthread_cancel to work"

"Error: failed to create image : '/sbin/mksquashfs /var/tmp/imgcreate-FKCSsk/iso-u8xCPh/LiveOS/osmin /var/tmp/imgcreate-FKCSsk/iso-u8xCPh/LiveOS/osmin.img' exited with error (-6)

This error occurs when MIC is unable to find the proper version of libgcc. This usually happens on x86_64 systems - MIC is looking for the 32 bit version of libgcc, but only the 64 bit version is installed. Installing the 32 bit version of libgcc should resolve this.

General squashfs issues

"Error: failed to create image : '/sbin/mksquashfs /var/tmp/imgcreate-FKCSsk/iso-u8xCPh/LiveOS/osmin /var/tmp/imgcreate-FKCSsk/iso-u8xCPh/LiveOS/osmin.img' exited with error (-6)

This error can also be an issue with the syslinux package. Try updating to the latest version for your distro.

device-mapper issues

unable to remove open device

This is a race condition with device-mapper in Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=506644. Usually rerunning the MIC command will resolve this.

python related issues

from urlgrabber.grabber import URLGrabber
ImportError: No module named urlgrabber.grabber

Some people are using python they built themselves, so some python modules aren't under their python library path. For such error, only one way is not to use your own python but to use python your system installed.

debian-based distros related issues

On debian-based distros, we used pycentral to package mic, but mic source installtion used distutils, so two kinds of installation have different installation path, if they exist there at the same time, it will result in some unpredictable errors to run mic. To run it correctly, you can keep only one installation, source or binary.

Remove binary installation

sudo apt-get remove mic

Remove source installation

Get python lib path

$ python
Python 2.5.2 (r252:60911, Sep 30 2008, 15:41:38)
[GCC 4.3.2 20080917 (Red Hat 4.3.2-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.sysconfig
>>> print distutils.sysconfig.get_python_lib()
<PYTHONLIBPATH>
>>>
$

Remove mic files

$ sudo rm -rf <PYTHONLIBPATH>/mic
$ sudo rm -rf <PYTHONLIBPATH>/mic-0.*
$ sudo rm -f /usr/bin/mic
$ sudo rm -f /usr/bin/mic-*
$ sudo rm -f /usr/bin/moblin-*

Note: You must replace <PYTHONLIBPATH> with the above python lib path

If program jams after mkfs.vfat, it might be due missing vol_id program. vol_id is replaced with blkid in new ubuntu versions. Try run /lib/udev/vol_id, if it's missing you have troubles.

ARM related issues

Traceback (most recent call last):
  File "/usr/bin/mic-image-creator", line 856, in <module>
    ret = main()
  File "/usr/bin/mic-image-creator", line 682, in main
    run_in_bootstrap(options.bootstrap, argv, bindmounts, arch = options.arch)
  File "/usr/bin/mic-image-creator", line 261, in run_in_bootstrap
    ret = subprocess.call(args, preexec_fn = chroot_bootstrap)
  File "/usr/lib/python2.5/subprocess.py", line 444, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.5/subprocess.py", line 594, in __init__
    errread, errwrite)
  File "/usr/lib/python2.5/subprocess.py", line 1149, in _execute_child
    raise child_exception
OSError: [Errno 8] Exec format error

This error occurs when trying to build ARM images without passing --arch=armv7l to meego-image-creator.

Error: Requires qemu version >= 0.13 for armv7hl

you should update the version of 'qemu-arm' or 'qemu-arm-static' to 0.13.0 or above.

Tips: if you encounter arm images creating failed, please try to run bootstrap mode, for armv7hl and armv7nhl is not supported in legacy mode.

Proxy cache

See http://bugs.meego.com/show_bug.cgi?id=2343

Some servers (such as with squid as a proxy) may cache more aggressively than others, and as a result repomd.xml is invalid (old copy) and references out of date metadata files (primary, comps, filelists, etc). This is because the server is returning a cached/old copy of repomd.xml.

Proxy cache should be disabled in mic and yum.

Advanced Hacking Tips

replace kernel in live image

$sudo mic-image-convertor --source-image=<your-netbook.img> --target-format=livecd --shell

This command will give you a chroot environment of MeeGo, you can follow the below steps to replace the default kernel with a new one.

1). remove the old kernel

# rpm -e kernel-netbook

NOTE: execute the above command in chroot environment.

2). copy new kernel to chroot environment of MeeGo

$sudo cp /path/to/kernel-*.rpm /var/tmp/imgcreate-*/install_root/

NOTE: execute the above command in your host system.

/path/to/kernel-*.rpm is your new kernel to install, /var/tmp/imgcreate-*/install_root/ is the directory where chroot MeeGo mounted on.

3). install the new kernel

# rpm -ivh /kernel-*.rpm
# rm /kernel-*.rpm

NOTE: execute the above command in chroot environment.

After that, you can find out the new kernel in directory /boot/.

4). update contents of isolinux

$sudo cp /var/tmp/imgcreate-*/install_root/boot/initrd-*.img  /var/tmp/imgcreate-*/iso-*/isolinux/initrd0.img
$sudo cp /var/tmp/imgcreate-*/install_root/boot/vmlinuz-*  /var/tmp/imgcreate-*/iso-*/isolinux/vmlinuz0

NOTE: execute the above command in your host system.

5. create a new image

#exit

NOTE: execute the above command in chroot environment.

After following all the above steps, you'll have a new image.

build btrfs image

From version 0.20.1 on, mic can support btrfs, you can create btrfs image by change fstype=ext3 in your kickstart file to fstype=btrfs. A successful creation will depend on btrfs-progs (btrfs-tools on debian-based distros, btrfsprogs on openSUSE) and btrfs kernel module in your local system, the known issue is we can't create btrfs image on Fedora 11, the issue is the default btrfs kernel module has some issue which will make kernel panic, so for this case, you must make sure you have a stable and latest kernel and btrfs-progs 0.19 on your system. You'll need to build your kernel by yourself and use it to boot your system.

Btrfs-progs is a prerequisite tool, it includes /sbin/mkfs.btrfs, but its version must match btrfs kernel module, so you must ensure this, mic repo http://repo.meego.com/MeeGo/tools/repos/ has an issue which will install btrfs-progs 0.19 when you install mic, this is an repo sync error, so please make sure you have your distro's btrfs-progs after mic is installed, if your distro's btrfs-progs version is 0.19, this isn't an issue, otherwise you must remove btrfs-progs installation introduced by mic and reinstall your distro's btrfs-progs.

How to build btrfs image on Fedora 11 or lower version?

  1. Download kernel-2.6.34 or kernel-2.6.35
  2. Configure btrfs as module
  3. make; make modules; make install; make modules_install
  4. Reboot your system using this built-newly kernel
  5. Install btrfs-progs 0.19

Done, you're ready for creating btrfs image.

Metapackages, Patterns, and Repository Creation

Meta-packages are discouraged in MeeGo. The preferred method is to use patterns in the repository (supported by both zypper and mic). This is done by creating a patterns.xml file, which is a fairly simple XML file:

<?xml version="1.0" encoding="UTF-8"?>
<pattern xmlns:rpm="http://linux.duke.edu/metadata/rpm"
         xmlns="http://novell.com/package/metadata/suse/pattern">
   <name>meego-core</name>
   <summary>MeeGo Core</summary>
   <description>Packages needed for Compliance</description>
   <uservisible/>
   <category lang="en">Base Group</category>
   <rpm:requires>
      <rpm:entry name="pam"/>
      <rpm:entry name="rootfiles"/>
      <rpm:entry name="bash"/>
      <rpm:entry name="sysvinit"/>
      <!-- ...etc... -->
   </rpm:requires>
</pattern>

The detailed documentation for this XML schema may be found on the OpenSuSe Wiki. There is also a Relax NG schema found in the source code for libzypp (buried in the folder zypp/parser/yum/schema).

In MeeGo's package-groups, you can find several utilities, style sheets, and examples for how to create patterns and convert them to groups (comps).

Once you have both a patterns.xml and comps.xml file, you can create your repository like this. (For brevity, assuming a simple 'packages' repository.)

   ## Copy RPM packages to a folder, repos/
   $ mkdir -p repos/repodata
   $ cp -f patterns.xml comps.xml repos/repodata
   $ createrepo -g repos/repodata/comps.xml repos/
   $ modifyrepo repos/repodata/patterns.xml repos/repodata

Now, GPG sign your repos/repodata/repomd.xml file. Note that `modifyrepo` is a part of the `createrepo` package.

Personal tools