gehrcke.de » ATLAS Software

new system successfully tested: “Distribution of High Performance Computing Jobs among Multiple Computing Clouds”

Jan-Philip Gehrcke — Tue, 18 Aug 2009 12:14:26 +0000

Hello you out there!

I just started running the first serious test of the system I’ve developed during this year’s Google Summer of Code. If I wanted to put it in sensational words, the test could be called “Distribution of Particle Physics High Performance Computing Jobs among Multiple Computing Clouds”; just to get some readers . During the test, there will be some time I just sit around and watch my monitor, so I decided to share my experience about the new system with you and keep record of the test progress within this blog post.

Content (don’t worry, the sections are short):

0 Introduction
1 Preparation
2 VM startup
3 Session monitoring (number of Job Agents)
4 Job submission
5 Session monitoring (number of jobs)
6 Job monitoring
7 Job completion, output receipt
8 Examine output
9 VM shutdown
10 Appendix

0 Introduction

First of all, I’ve to introduce the system. This is the longest section.

It’s a job scheduling system supporting Virtual Machines (VMs) in multiple Infrastructure-as-a-Service computing clouds. Because of this, I found the name “Clobi”, which somehow comes from “cloud” and “combination”. If you know a better name, then please let me know . Currently, the system is using and supporting Nimbus (and it’s prepared for Cumulus, Nimbus’ storage service) and Amazon Web Services (more precisely EC2, SQS, SimpleDB, S3).

It’s an “elastic” and “scalable” job system that can set up a huge computing resource pool almost instantly. VMs are added to or removed from the pool dynamically; based on need and demand. Jobs are submitted and processed using an asynchronous queueing system. An arbitrary number of clients is allowed to submit jobs to an existing resource pool. The basical realiability is inherited from the reliability of the core messaging components Amazon SQS (for queueing) and Amazon SimpleDB (for bookkeeping): to prevent data from being lost or becoming unavailable, it is stored redundantly and geographically dispersed across multiple datacenters.

Furthermore, the job system’s components are highly decoupled, which allows single components to fail or to get re-initialized without affecting the others.

The motivating application for this system is ATLAS Computing (for the ATLAS experiment at LHC, CERN (Geneva)): a common ATLAS Computing application (the so-called “full chain”) will be run during this test.

But: the basic system is totally generic and can be used in any case whenever it’s convenient to distribute jobs among different clouds. This is always the case when one tries to satisfy the basic computing power needs for a low price by e.g. operating an own Nimbus cloud, but wants to able to instantly balance out peaks of desired computing power by simply adding Amazon’s EC2 to the resource pool for a certain amout of time. By using Clobi, combining different clouds to one big resource pool becomes very easy.

These are the main components used during this test:

a special ATLAS Software Virtual Machine image based on CernVM. I placed it on Nimbus Teraport Cloud and (as Amazon Machine Image) on S3 for EC2

the Clobi Resource Manager (observing job queues, starting/killing VMs, …)

the Clobi Job Agent (running on VMs, polling & running jobs, bookkeeping, …)

the Clobi Job Management Interface (providing methods to submit / remove / kill / monitor / … jobs)

a Ganga Clobi Backend (integrates Clobi into Ganga, which is “an easy-to-use frontend for job definition and management”)

The meaning of these components will become clearer in the following parts.

Let me show step by step — but only very roughly — how I use Clobi within the first serious test. Many details are left out, but you will get it in principle. After reading this blog post, you’ve an overview about what the system does and what I’ve actually done during the summer.

1 Preparation

I’ve prepared session/cloud configuration files. Using them, I started a new Clobi session (a resource pool) with the Clobi Resource Manager (It’s a Python application and I use it locally; here on my desktop machine). At first, it does much configuration and initialization stuff, including setting up SQS queues and SimpleDB domains. Interaction with Amazon Web Services is done via the boto module for Python. After initialization, the Resource Manager offers an interactive mode (it’s a multi-threaded console application with user interface, built using the urwid module for Python).

2 VM startup

Using the Resource Manager, I started one VM on EC2 and one on Nimbus Teraport Cloud, both based on “the special ATLAS Software VM”, containing ATLAS Software 15.2.0 and the Clobi Job Agent. Starting VMs manually is done with a very simple command. The driving forces in the background are boto in case of EC2 and the Nimbus cloud reference client, which I’ve wrapped and controlled via Python’s subprocess module. The main loop thread of the Resource Manager periodically polls the states of just started EC2 instances and the states of Nimbus client subprocesses to figure out if the instructed actions result in success or not.

The following screenshot shows the Resource Manager in action (you basically see a terminal window). Follow a bit of the log. As you can see, it’s very easy to run VMs and — after a certain amout of time — the Resource Manager detects that both VMs have successfully started booting:

Clobi Resource Manager showing two started Virtual Machines

The EC2 VM needed around 10 minutes to start up, while the Nimbus VM needed 20 minutes. Reason: the AMI is ~10 GB big; the image on Nimbus ~20 GB (wasted space & time, but it’s just a test..).

You might say: “Only two VMs? Boring..”. I say: I could have taken several hundred. The point is: it would not make any difference, except in cost and in the amount of space used for log files. Clobi uses technology / is designed to always work reliably; even in different orders of magnitude. This is often called “scalable” or “elastic”. Basically, this positive characteristic is inherited from Amazon’s SQS, SDB and S3, which are used by Clobi to do management and control of the system.

3 Session monitoring (number of Job Agents)

I conceal almost all the details of how the components exchange information. But I’ve to tell the following to you, to not completely confuse you:

the Resource Manager gave some bootstrap information to the VMs.

the Job Agent is automatically invoked on VM operating system startup.

Using the bootstrap information, each VM’s Job Agent “registers with SimpleDB”. The Resource Manager has a monitoring functionality to check SimpleDB for running Job Agents:

Clobi Resource Manager showing two started Clobi Job Agents

Voilà, now it’s definitely known that both VMs successfully started their Job Agent’s. These start in a “watching/lurking” state, periodically polling SQS for jobs.

4 Job submission

The SimpleDB / SQS / S3 data structure together with Clobi’s Job Management Interface allows to submit jobs with different priorities, to remove jobs, to kill running jobs and to monitor jobs. Furthermore, transmission and receipt of an input/output sandbox archive is possible. This is needed to deliver executables and small input data and to receive small output data as well as stdout/err and other logs.

I’ve downloaded Ganga and installed it to my local machine. Then, I’ve started developing a new “Clobi backend” to integrate Clobi’s Job Management Interface into Ganga. Using this new backend, it’s possible to submit/kill/monitor/.. jobs right away from the Ganga interface, using Ganga’s common job description and management commands.

To test the system, I’ve prepared some shellscripts that invoke running “The Full Chain” on the worker nodes. This is a very good test to validate the whole system: it needs some very small input files, only works if the ATLAS Software was set up properly (uoooh.. not trivial!), stresses the VM (the simulation step consumes much CPU power) and leaves some small output files for the output sandbox.

At Ganga startup, I provided a configuration file containing few but essential information about the Clobi session that I’ve set up before via Resource Manager. From this configuration file Ganga’s Clobi backend e.g. knows which SimpleDB domain to query and to which SQS queues the job messages must be submitted. Using this bootstrap information, an arbitrary number of Gangas could be used from anywhere to submit and manage jobs within this special Clobi session.

I will now submit the same job (the “full chain” thing) three times: the Nimbus VM has two virtual cores and its Job Agent will try to receive and run two jobs at the same time. The EC2 VM (m1.small) only has one virtual core. Hence, three jobs are needed to use the VMs to full capacity . This is a screenshot from the Ganga terminal session where I submitted the jobs:

Ganga: job submission via Clobi backend

The Clobi backend successfully did its job: it created three Clobi job IDs, submitted three SQS messages and uploaded three input sandbox archives.

5 Session monitoring (number of jobs)

The Resource Manager is able to observe the queues and to determine the number of jobs submitted to them. It recognizes two jobs in the queue for priority 2:

Clobi Resource Manager detected two jobs in the queues

Only two? Maybe one Job Agent polled a job right away after submission, or maybe the SQS measurement was not exact (this is possible, too). Anyway, few time later there is only one job left in the queues and then they are empty. This means that the Job Agent on the Nimbus VM successfully grabbed two jobs and the EC2 VM grabbed one:

Clobi Resource Manager detects zero jobs in the queues

While me and Ganga are waiting for the jobs to finish (this takes some time and Ganga periodically polls the state of the jobs via Clobi backend / Clobi Job Management Interface), I use the time to advise you of an important fact: it’s the objective to automate the observe-queues-and-start/kill-VMs-as-required process in the future. The current Resource Manager is very prepared for this. Let me show the monitoring loop to you:

Clobi Resource Manager showing its monitoring loop

It observes the number of jobs in the queues and the number of running Job Agents periodically. Based on this information the Resource Manager easily could start / kill virtual machines (I’ve already demonstrated how easy starting is; killing is described later). I did not implement this algorithm until now, because a) I had no time and b) I could have done it quick and dirty, but I really did not need this feature to develop and test the rest of the system. But this feature will come, because if it’s implemented properly with intelligent policies, it’s just great.

6 Job monitoring

As I’ve already mentioned, Ganga periodically checks the jobs’ states. Therefore, the Ganga Clobi backend provides a special method that Ganga calls from time to time from one of its monitoring threads. Normally this happens quietly, but I’ve put some debug output into this method. Let’s check it out:

Ganga receives monitoring information via the Clobi backend

7 Job completion, output receipt

After some more time, Ganga discovered that one of the three jobs finished successfully. This means that the Job Agent detected a returncode of 0 of the job shellscript and could successfully store the output sandbox archive to S3. At this point, the Clobi backend triggers to download and extract the output sandbox archive. This looks like:

Clobi                              : INFO     status for job-090818044445-3585-3088: completed_success
Ganga.GPIDev.Lib.Job               : INFO     job 60 status changed to "completed"
Clobi                              : INFO     download atlassessions/0907210728-testsess-0c7e/jobs/out_sndbx_job-090818044445-3585-3088.tar.bz2 from S3
Clobi                              : INFO     store key 0907210728-testsess-0c7e/jobs/out_sndbx_job-090818044445-3585-3088.tar.bz2 as file /home/gurke/gangadir/workspace/gurke/LocalAMGA/60/output/out_sndbx_job-090818044445-3585-3088.tar.bz2 from bucket atlassessions
Clobi                              : INFO     Download of output sandbox archive successfull.

Did you have doubts that this is my first serious test and everything worked until now? Some parts of the system are already tested very much, of course. But the Clobi backend for Ganga made the transition from vitally-important-features-missing to just-scratch-along-usability only a few hours ago. I’m really very happy that everything worked until now, but the output sandbox archive extraction could be improved:

Clobi                              : CRITICAL Error while extracting output sandbox
Clobi                              : CRITICAL Traceback:
Traceback (most recent call last):
  File "/mnt/hgfs/E/gsoc_code_repo/ganga_clobi_backend/Clobi/Clobi.py", line 243, in clobi_dl_extrct_outsandbox_arc
    sp = subprocess.Popen(
NameError: global name 'subprocess' is not defined

Yoooah, I (want to) use Python’s subprocess module to extract the tar.bz2 archive with system’s tar, but I forgot to import subprocess . This is forgotten and fixed easily . Btw: of course I got three downloaded output archives and three extraction errors .

8 Examine output

The output sandbox archive files were stored on my local machine by Ganga’s Clobi backend. I take a look into one of it by extracting it manually:

$ ls
out_sndbx_job-090818044451-3585-01c2.tar.bz2
$ tar xjf out_sndbx_job-090818044451-3585-01c2.tar.bz2
$ ls
AOD_007410_00001.pool.root  evgen.log  joblog_job-090818044451-3585-01c2  recoAOD.log
EVGEN_007410_00001.pool.root  jobagent_log  out_sndbx_job-090818044451-3585-01c2.tar.bz2

Great! The AOD file is there. This means that 1) Clobi did perfect work to control and manage the job and 2) the ATLAS Computing part (“The Full Chain”) worked perfectly:

the interaction between a certain particle (which was defined within the input sandbox) and the ATLAS detector was successfully simulated.

the ATLAS detector output (basically times and voltages) was calculated successfully.

particle tracks and energy deposits were successfully reconstructed from times and voltages.

an event summary was successfully built from tracks and energy deposits.

“The summary” is saved within the AOD file, which successfully returned to my local machine. Cool. Every single part of the system worked as it should (psss, don’t think of the extraction…).

9 VM shutdown

You perhaps asked yourself how to dynamically kill VMs. Besides the “hard kill” (invoking Nimbus/EC2 API calls to shut down a VM), I’ve implemented a mechanism that I call “soft kill”: the Resource Manager sets the “soft kill flag” for a specific VM (in SimpleDB) and the corresponding Job Agent checks it from time to time. When it is set, it waits until all currently running jobs are done and then the Job Agent shuts down the VM. Let’s watch it in action (I had to look up the command of my own application, too few sleep recently!):

Clobi Resource Manager setting up the softkill flag for both VMs

After waiting some time we see the number of running Job Agents decrease to zero..

Clobi Resource Manager detected that both Job Agents / VMs have shut down

10 Appendix

If you are very interested, you can find some additional material:

My earlier work on this topic (from last year), “Amazon Web Services for ATLAS Computing” (AWSAC) can be found here: http://gehrcke.de/awsac.

The aboriginal GSoC project description can be found here.

I’ve already written some blog posts about this project during Google Summer of Code.

The last visualizations of the system (from the planning period) are these two schemes: one, two.

A detailled, up-to-date and exact description of the system (“Clobi“) is planned for the future.

I will need the last days of GSoC to implement some missing and important features, to search and fix bugs and to clean everything up to make it presentable. I will definitely work on this project after GSoC (as the time allows it, of course). Currently I think about pushing the project to either bitbucket or Google code. Both support mercurial repositories and this is what I used for my code until now (locally).

If you like this, spread it! Every question and/or comment is much appreciated!

Thanks for listening

CernVM: local ATLAS Software — the clean solution

Jan-Philip Gehrcke — Tue, 30 Jun 2009 14:54:01 +0000

In my blog post CernVM: how to set up a local ATLAS Software Release, I presented a brutal approach how to override CVMFS (CernVM‘s filesystem with HTTP backend) to install a local ATLAS Software release. Now I worked out a very clean and smooth solution. This approach allows:

to use the local ATLAS Software without “hacking” anything
to use local software and software provided by CVMFS at the same time.

The introduction and motivation from CernVM: how to set up a local ATLAS Software Release is still valid. The rest can be considered as deprecated.

The objective is to install ATLAS Software 15.2.0 to the local filesystem of CernVM 1.2.0 using pacman. Therefore we need some room in the filesystem (around 10 GB). Regarding this, the blog post Resize ext3 file system in loopback file could be interesting for you. This step is necessary on e.g. Nimbus. If you work on Amazon’s EC2, you perhaps would like to embed an EBS volume to install the software there. Useful hints for this are given in this blog post: EC2: Install ATLAS Software to an EBS volume

Prepare installation — start off from a fresh CernVM

Assume a fresh (maybe with a bigger file system than by default) CernVM 1.2.0 just booted up. Bootstrap it to the following configuration (via web interface):

create the new linux user atlasuser
under Virtual Organization Configuration choose ATLAS — keep everything else as No

Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.

[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive

You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

Now switch to root via sudo su. Create the directory /opt/atlas-local, which will be the place to put all the local ATLAS related software like CMT and pacman and the ATLAS Software itself. When using an EBS volume, /opt/atlas-local would be a convenient mount point. Give all rights to this directory.

[root@bla opt]# mkdir /opt/atlas-local
[root@bla opt]# chmod 777 /opt/atlas-local

Now we can already start installing an ATLAS Software release using pacman. I do this stuff so often, so I’ve created a little script for this. It checks if pacman’s already there. If not, it downloads & extracts it. pacman then is set up and the ATLAS Software installing command gets invoked. To use this script, create a file /opt/atlas-local/install_ATLAS-15-2-0.sh and fill it up with this content:

PACMANDIR=/opt/atlas-local/pacman
ATLASINSTALLDIR=/opt/atlas-local/15.2.0
if [ -e ${PACMANDIR} ]; then
    echo --info: ${PACMANDIR} exists
    echo --info: cd ${PACMANDIR}/pacman-*
    cd ${PACMANDIR}/pacman-*
    echo --info: pwd: `pwd`
    echo --info: pacman setup
    source setup.sh
else
    echo --info: mkdir -p and cd to ${PACMANDIR}
    mkdir -p ${PACMANDIR}
    cd ${PACMANDIR}
    echo --info: download latest pacman
    wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
    echo --info: extract..
    tar xzf pacman-latest.tar.gz
    echo --info: cd pacman-*
    cd pacman-*
    echo --info: pwd: `pwd`
    echo --info: setup pacman..
    source setup.sh
fi
 
echo --info: mkdir -p, cd to ${ATLASINSTALLDIR}
mkdir -p ${ATLASINSTALLDIR}
cd ${ATLASINSTALLDIR}
echo --info: pwd: `pwd`
 
echo --info: start installing ATLAS:
echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. I use Indiana University mirror (am-IU). You perhaps want to modify this. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.

If you don’t like using this script at all, then just grab out the important commands. But please think twice about the path where to install ATLAS Software. Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under /opt/atlas-local allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.

Start ATLAS Software installation using pacman

Be sure to work as atlasuser. It’s just more clean to install the software not as root. Then, source the /opt/atlas-local/install_ATLAS-15-2-0.sh script from above.

[atlasuser@bla atlas-local]$ source install_atlas_15-2-0.sh
--info: mkdir -p and cd to /opt/atlas-local/pacman
--info: download latest pacman
--07:58:25--  http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
[...]
pacman-latest.tar.gz' saved [856237/856237]
--info: extract..
--info: cd pacman-3.28 pacman-latest.tar.gz
--info: pwd: /opt/atlas-local/pacman/pacman-3.28
--info: setup pacman..
--info: mkdir -p, cd to /opt/atlas-local/15.2.0
--info: pwd: /opt/atlas-local/15.2.0
--info: start installing ATLAS:
--info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

This will take a while (something around an hour for installing from am-IU to EC2 in US). Inbetween, you can log in with a second shell and proceed with the next important steps.

Make useful CVMFS content local

While working on the dirty solution, some problems came up. In particular, I had to hack around the CMTCONFIG problem. The CernVM collaborators have spent considerable amount of time to sort out these problems. So let’s profit from their work. I figured out that “taking over” their version of the Configuration Management Tool CMT and their cmthome directory is enough to get everything working properly.

Copy CernVM’s CMT from CVMFS to the local filesystem

Work as root. At first, copy the whole CMT directory from CVMFS (which has /opt/atlas/ as mountpoint) to /opt/atlas-local:

[root@bla ~]# cp -R /opt/atlas/software/sw/CMT /opt/atlas-local

Now CMT’s path changed, which means that it has to be re-installed. By doint this, /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh gets refreshed. This is important to set up CMT’s environment later on.

[root@bla atlas-local]# cd /opt/atlas-local/CMT/v1r20p20090520/mgr/
[root@bla mgr]# ./INSTALL
============================================
       CMT installation terminated.
           --------------------
 cmt.exe is available on this site for:
    Darwin-PowerMacintosh
    Darwin-i386
    Linux-i686
    Linux-x86_64
    VisualC
============================================

CMT has successfully found its new location. Btw, let’s look at the currently existing directory structure of /opt/atlas-local:

[root@bla ~]# cd /opt/atlas-local
[root@bla atlas-local]# ls
15.2.0  CMT  install_atlas_15-2-0.sh  pacman

It should look like that

Create `cmthome` and modify the `requirements` file

The essential part of this step is to grab&modify CernVM’s requirements file for ATLAS Software releases. Work as atlasuser and copy the cmthome directory to your local ATLAS Software:

[atlasuser@bla atlas-local]$ cp -R /opt/atlas/software/15.2.0/15.2.0/cmthome /opt/atlas-local/15.2.0/

Modify the requirements file within that new cmthome directory so that SITEROOT points to the new local ATLAS Software:

[atlasuser@bla atlas-local]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ ls
Makefile  cleanup.csh  cleanup.sh  requirements  setup.csh  setup.sh
[atlasuser@bla cmthome]$ vi requirements
[... modify SITEROOT path ...]
[atlasuser@bla cmthome]$ cat requirements
#---------------------------------------------------------------------
set CMTSITE STANDALONE
set SITEROOT /opt/atlas-local/15.2.0
macro ATLAS_TEST_AREA ${HOME}/testarea
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag projectArea
macro SITE_PROJECT_AREA ${SITEROOT}
macro EXTERNAL_PROJECT_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc4-gcc34-opt

In the next step, this new requirements file will be parsed by CMT to create a setup script which will be necessary to set up the correct runtime environment for ATLAS Software.

Set up CMT environment, create the ATLAS Software setup script

Work as atlasuser. To set up the CMT environment, use the /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh, which was created while installing CMT.

[atlasuser@bla ~]$ source /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh

Now, cmt and cmt.exe should be in the path. You can check it by typing cm *TAB*.

Switch to the cmthome directory and use cmt config to create the setup script for the ATLAS Software runtime environment:

[atlasuser@bla ~]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ cmt config
sh: manpath: command not found
------------------------------------------
Configuring environment for standalone package.
CMT version v1r20p20090520.
System is Linux-i686
------------------------------------------
Creating setup scripts.
Creating cleanup scripts.

Interim report: installation is done!

Now we’re done. This new ATLAS Software installation is prepared for usage. Each of the previous steps is unique for this installation and does not have to be repeated. To use this ATLAS Software, only /opt/atlas-local/15.2.0/cmthome/setup.sh has to be invoked.

In case that /opt/atlas-local was the mountpoint to an EBS volume, this volume now contains a fully working ATLAS Software release. It may get attached to any VM running an operating system that supports ATLAS Software (Like CernVM + group-atlas, Scientific Linux 3/4, …) to start production runs with this specific installation.

Test the new local ATLAS Software: Run the Full Chain

To test & validate the new ATLAS Software installation, let’s run the Full Chain. An easy explanation of what this is and how to realize it with a shell script is given in this blog post: ATLAS Software: How to run The Full Chain

Create working directory, setup runtime environment

Work as atlasuser. Create a working directory. Many files will be produced while running the full chain.

[atlasuser@bla ~]$ mkdir atlasrun_15-2-0___1/
[atlasuser@bla ~]$ cd atlasrun_15-2-0___1

When the pacman installation from above has finished, you can set up the runtime environment for the new ATLAS Software installation.

This step always has to be executed before using the software after a new log in.

[atlasuser@bla atlasrun_15-2-0___1]$ source /opt/atlas-local/15.2.0/cmthome/setup.sh -tag=15.2.0
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
AtlasLogin: WARNING - test directory [/home/atlasuser/testarea/15.2.0] doesn't exist - the runtime environment won't reflect it

Don’t forget the -tag parameter!
The manpath problem isn’t an acutal problem. In this blog post you can read how to make manpath available. But it’s not needed. The warning about the test directory is correct, but for our full chain test we don’t need the testarea. If you need it for your production run, then create the stated directory.

Run the full chain!

I prepared a little script to run it. It’s explained in ATLAS Software: How to run The Full Chain

Download the script an run it:

[atlasuser@bla atlasrun_15-2-0___1]$ wget gehrcke.de/gsoc/athena_whole_chain_1event.sh
[...]
`athena_whole_chain_1event.sh' saved [1229/1229]
[atlasuser@bla atlasrun_15-2-0___1]$ source athena_whole_chain_1event.sh
[...]
`singlepart_singlepi' saved [961/961]
 
read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
 
real    0m42.774s
user    0m29.074s
sys     0m1.292s
 
read EVGEN file.. simulate.. create HITS file..-> simul.log
 
real    9m30.879s
user    9m2.227s
sys     0m4.936s
 
read HITS file.. digitize.. produce RDO file..-> digi.log
 
real    2m35.875s
user    2m12.147s
sys     0m5.116s
 
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
 
real    3m54.783s
user    3m17.422s
sys     0m6.427s
 
read ESD file.. convert.. produce AOD file..-> recoAOD.log
 
real    1m44.373s
user    1m31.754s
sys     0m3.687s

Success!

The full chain performed successfully with the local ATLAS Software installation. At the same time, we could have used the “remote ATLAS Software” via CVMFS. I verified this by

logging in with an additional shell to the same machine
creating another working directory
setting up the runtime environment for the remote ATLAS Software 15.2.0 by sourcing /opt/atlas/software/setupScripts/setupAtlasProduction_15.2.0.sh
running my full chain script

It resulted in

read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
 
real    6m13.287s
user    0m31.648s
sys     0m1.441s
 
read EVGEN file.. simulate.. create HITS file..-> simul.log
 
real    25m53.893s
user    9m2.742s
sys     0m5.593s
 
read HITS file.. digitize.. produce RDO file..-> digi.log
 
real    15m2.761s
user    2m12.836s
sys     0m6.455s
 
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
 
real    22m21.185s
user    3m19.384s
sys     0m6.891s
 
read ESD file.. convert.. produce AOD file..-> recoAOD.log
 
real    3m10.793s
user    1m33.584s
sys     0m4.454s

Pay attention to the long real timings. The reason is clear: The files had to be transferred via HTTP from a proxy server providing the content of CVMFS.

ATLAS Software: How to run The Full Chain

Jan-Philip Gehrcke — Tue, 30 Jun 2009 14:49:36 +0000

During development, a system running ATLAS Software has to be tested and validated. There are some standard tests that almost don’t need any input data, stress the system and — if they run properly — are a (very) good indicator that everything is set up correctly. I talk about the so-called JobTransforms. By combining these JobTransforms, the so-called Full Chain can be run — a convenient test. In this blog post I summarize what’s behind the Full Chain and provide a shell script to easily run it.

What is “The Full Chain”?

The so-called JobTransforms are python scripts that are used to run production tasks. They take an input file, a set of parameters and “transform” the input into one or more output files. Combined in the correct order, they build up The Full Chain. I will summarize the elements of the chain now:

Step 1) Event generation: a virtual particle gets created with a specific energy and direction.
Input: particle definition file. Output: EVGEN file.
Step 2) Simulation: the interaction between this particle and the detector is simulated.
Input: EVGEN file. Output: HITS file.
Step 3) Digitization: the ATLAS detector output is calculated.
Input: HITS file. Output: RDO file.
Step 4) Reconstruction: times and voltages are reconstructed into tracks and energy deposits.
Input: RDO file. Output: ESD file.
Step 5) Conversion: only keep the most important data from the last step.
Input: ESD file. Output: AOD file.

The following picture from here visualizes the chain and — you might be interested in that — shows where real data from the LHC will play a role in reality:

The Full Chain

How to realize “The Full Chain”

As you know, the input of step 1 is a small user-given file defining particle parameters. This file on my webserver describes a single pion with specific energy and direction. The content basically is the following:

# Single pi+ in log(E) between 200 MeV and 2 TeV
from AthenaCommon.AlgSequence import AlgSequence
topAlg = AlgSequence("TopAlg")
from ParticleGenerator.ParticleGeneratorConf import ParticleGenerator
topAlg += ParticleGenerator()
ParticleGenerator = topAlg.ParticleGenerator
# For DEBUG output from ParticleGenerator.
ParticleGenerator.OutputLevel = 2
ParticleGenerator.orders = [
  "PDGcode: constant 211",
  "e: log 200. 2000000.",
  "eta: flat -5.5 5.5",
  "phi: flat -3.14159 3.14159"
  ]
from EvgenJobOptions.SingleEvgenConfig import evgenConfig

At this point it’s not important to understand each line of this file. It works

The following shellscript downloads this file and initializes The Full Chain for exactly one event of this particle. During execution, it measures timings.

wget http://gehrcke.de/gsoc/singlepart_singlepi
mv singlepart_singlepi CSC.007410.singlepart_singlepi+_logE.py
 
echo -e "\nread CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log"
time csc_evgen_trf.py 007410 1 1 765432 CSC.007410.singlepart_singlepi+_logE.py EVGEN_007410_00001.pool.root > evgen.log
 
echo -e "\nread EVGEN file.. simulate.. create HITS file..-> simul.log"
time csc_simul_trf.py EVGEN_007410_00001.pool.root HITS_007410_00001.pool.root NONE 1 0 452368 "ATLAS-CSC-02-00-00" 0 0 "QGSP_BERT" CalHits.py > simul.log
 
echo -e "\nread HITS file.. digitize.. produce RDO file..-> digi.log"
time csc_digi_trf.py HITS_007410_00001.pool.root RDO_007410_00001.pool.root 1 0 "ATLAS-CSC-02-00-00" 740581234 29402491 'NONE' 'NONE' CalHits.py 'NONE' 'AtRndmGenSvc' 'QGSP_EMV' 'NONE' > digi.log
 
echo -e "\nread RDO file.. reconstruct.. produce ESD file..-> recoESD.log"
time csc_recoESD_trf.py RDO_007410_00001.pool.root ESD_007410_00001.pool.root 'NONE' 1 0 "ATLAS-CSC-02-00-00" 'NONE' > recoESD.log
 
echo -e "\nread ESD file.. convert.. produce AOD file..-> recoAOD.log"
time csc_recoAOD_trf.py ESD_007410_00001.pool.root AOD_007410_00001.pool.root 1 0 "ATLAS-CSC-02-00-00" 'NONE' > recoAOD.log

You can download the script here.

After setting up the runtime environment for your ATLAS Software installation, you can simply download and execute this script. It should work! I tested it with ATLAS Software 15.2.0, as you can see in this blog post: CernVM: local ATLAS Software — the clean solution

EC2: Install ATLAS Software to an EBS volume

Jan-Philip Gehrcke — Tue, 30 Jun 2009 13:58:15 +0000

In CernVM: local ATLAS Software — the clean solution I proposed to install an ATLAS Software release to an EBS volume. I did it to make it locally available in CernVM running on EC2. The approach allows to move an EBS volume from one EC2 instance to another without losing important components and functionality. Here are some hints (both, CernVM specific and general) to follow before installing the software via pacman..

Start off from a fresh CernVM — migrate to group-atlas

Start a fresh CernVM 1.2.0 instance on EC2 (I used ami-a50cebcc, aki-9b00e5f2). Bootstrap it to the following configuration (via web interface):

create the new linux user atlasuser
under Virtual Organization Configuration choose ATLAS — keep everything else as No

Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.

[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive

You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems.

With migration to group-atlas, gcc gets installed. We will need it soon!

Create and attach EBS volume, make file system

Create a new EBS volume within the same availability zone as the CernVM instance just startet up. I chose 15 GB size. This should be enough to hold one ATLAS Software release plus a bit of additional stuff.

Attach this new EBS volume as e.g. /dev/sdh to the CernVM instance.

Regarding CernVM, now e2fsprogs has to be installed to create a filesystem on the fresh block device /dev/sdh. Therefore we need the compiler.
Work as root, download&install:

[root@bla ~]# wget http://prdownloads.sourceforge.net/e2fsprogs/e2fsprogs-1.41.6.tar.gz
[...]
`e2fsprogs-1.41.6.tar.gz' saved [4422395/4422395]
[root@bla ~]# tar xzf e2fsprogs-1.41.6.tar.gz
[root@bla ~]# cd e2fsprogs-1.41.6
[root@bla e2fsprogs-1.41.6]# ./configure > cfg.log
[root@bla e2fsprogs-1.41.6]# make install > makeinst.log
make[1]: texi2dvi: Command not found
make[1]: [libext2fs.dvi] Error 127 (ignored)
/usr/bin/install: cannot stat `libext2fs.info*': No such file or directory
make[1]: [install-doc-libs] Error 1 (ignored)
gzip: /usr/share/info/libext2fs.info*: No such file or directory
make[1]: [install-doc-libs] Error 1 (ignored)

The warnings are no problem. Now mkfs.ext3 can be used:

[root@bla e2fsprogs-1.41.6]# /sbin/mkfs.ext3 /dev/sdh
mke2fs 1.41.6 (30-May-2009)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
[...]
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mount it

Now the mountpoint has to be chosen. The objective is to install everything needed by the ATLAS Software to this EBS volume so that moving around with this volume between different instances (which are running operating systems supporting ATLAS Software) becomes possible.

Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under the mountpoint allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.

I chose the mountpoint /opt/atlas-local. Now and in the future, I will always have to mount the EBS volume to this directory.

Work as root: create mountpoint, mount device and give all rights:

[root@bla opt]# mkdir /opt/atlas-local
[root@bla opt]# mount /dev/sdh /opt/atlas-local
[root@bla opt]# chmod 777 /opt/atlas-local

Now log in as atlasuser and check out the new space:

[atlasuser@bla atlas-local]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              9293008   1999624   6821316  23% /
none                    890952         0    890952   0% /dev/shm
/dev/sdh              15481840    169592  14525816   2% /opt/atlas-local

Install ATLAS Software to this EBS volume

To install a new ATLAS Software Release to a subdirectory of the mountpoint, you can use my following script which automatically checks out pacman and invokes the installing command:

PACMANDIR=/opt/atlas-local/pacman
ATLASINSTALLDIR=/opt/atlas-local/15.2.0
if [ -e ${PACMANDIR} ]; then
    echo --info: ${PACMANDIR} exists
    echo --info: cd ${PACMANDIR}/pacman-*
    cd ${PACMANDIR}/pacman-*
    echo --info: pwd: `pwd`
    echo --info: pacman setup
    source setup.sh
else
    echo --info: mkdir -p and cd to ${PACMANDIR}
    mkdir -p ${PACMANDIR}
    cd ${PACMANDIR}
    echo --info: download latest pacman
    wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
    echo --info: extract..
    tar xzf pacman-latest.tar.gz
    echo --info: cd pacman-*
    cd pacman-*
    echo --info: pwd: `pwd`
    echo --info: setup pacman..
    source setup.sh
fi
 
echo --info: mkdir -p, cd to ${ATLASINSTALLDIR}
mkdir -p ${ATLASINSTALLDIR}
cd ${ATLASINSTALLDIR}
echo --info: pwd: `pwd`
 
echo --info: start installing ATLAS:
echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. Adjust the version numbers to your needs. I use Indiana University mirror (am-IU). You perhaps want to modify this, too. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.

Furthermore, in CernVM: local ATLAS Software — the clean solution you can read about:

how to set up CMT
how to create a cmthome directory and a valid requirements file
how to set up the runtime environment for an ATLAS Software Release

Have fun and let me know what you think!

CernVM: how to set up a local ATLAS Software Release (dirty version)

Jan-Philip Gehrcke — Sun, 21 Jun 2009 00:05:05 +0000

One of the main features of CernVM is its special filesystem CVMFS with http backend (based on FUSE). Using CernVM in the standard way, the different experiment softwares work out-of-the-box and are made accessible over the web via CVMFS. Although this is a great feature, I like to set up an ATLAS Software release locally — as real offline version — to be independent of the software-providing webservers.

Currently, the access to the providing server(s) must come from within CERN network. I am working with VMs in Chicago (Teraport, Nimbus cloud), don’t have a CERN account and doubt that the performance of CVMFS would have been good enough between Geneva and Chicago for my testing purposes. Hence, I tried to set up an ATLAS Software release locally, which required some special work.

Everything is based on CernVM 1.2.0 and ATLAS Software 15.1.0. The current solution is dirty and I am still looking for the optimal way. In the end, I would like to have an easy “switch” option between

using CVMFS and being dependent of way software-providing webservers and
keeping everything local

[Update]

The introduction/motivation is still valid and the parts below are still true. But in the meanwhile I found a much cleaner solution. It allows to use local and remote ATLAS Software at the same time and does not need any “hacking”. You can read about it in this blog post: CernVM: local ATLAS Software — the clean solution

[/Update]

To make room for the heavy ATLAS Software, I at first had to extend the filesystem within CernVM’s loopback file (I worked on the Xen-based Nimbus cloud, which did not support additional partitions at that time). This procedure is described in this blog post:
Resize ext3 file system in loopback file

Btw: While acting on CernVM’s image and filesystem locally with the intention to extend everything, I added the /root/.ssh folder to make public key insertion of Nimbus and EC2 possible. The directory does not exist by default. Predrag revealed that he will add it in the next release (should be 1.3.0).

Okay, imagine you are logged in as root to a booted fresh CernVM (only with the modifications named above: bigger filesystem, /root/.ssh added). If the Xen hypervisor added a correct kernel to the system, you can start right away by migrating the system to group-atlas:

$ conary migrate group-atlas --interactive

If this throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

After migration to group-atlas, the system is ready for a pacman installation of ATLAS Software. Set up pacman:

$ wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
$ tar xzf pacman-latest.tar.gz
$ cd pacman-3.28/
$ source setup.sh

In this case, the ATLAS Software release (+KitValidation) should get installed to /opt/atlas/15.1.0:

$ mkdir /opt/atlas/15.1.0
$ cd /opt/atlas/15.1.0
$ pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-BU:15.1.0+KV

-pretend-platform SLC-4 is necessary, since we’re not on the standard ATLAS platform. Installing takes quite a while. To optimize the process, I chose Boston University mirror, which is good from the perspective of Chicago. In the end you should see

##################################################
##   AtlasProduction 15.1.0 Validation [  OK  ]
##################################################

Now the exciting part comes: setting up a working environment for the software.

Set up a cmthome directory and a requirements file:

$ mkdir /opt/atlas/15.1.0/cmthome
$ vi /opt/atlas/15.1.0/cmthome/requirements
$ cat /opt/atlas/15.1.0/cmthome/requirements
set CMTSITE STANDALONE
set SITEROOT /opt/atlas/15.1.0
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag noTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

This requirements file should be enough to use the release for local production runs (standalone and no testarea).

Now set up CMT, cd to the folder of the new requirements file and configurate CMT:

$ source /opt/atlas/15.1.0/CMT/v1r20p20081118/mgr/setup.sh
$ cd /opt/atlas/15.1.0/cmthome/
$ cmt config
sh: manpath: command not found
------------------------------------------
Configuring environment for standalone package.
CMT version v1r20p20081118.
System is Linux-i686
------------------------------------------
Creating setup scripts.
Creating cleanup scripts.

See the manpath issue? Not a real problem, but if you like to fix it for the future:

$ vi /bin/manpath
$ cat /bin/manpath
#!/bin/bash
man --path $*
$ chmod u+x /bin/manpath
$ conary update man
[...]
$ manpath
/usr/local/share/man:/usr/share/man:/usr/X11R6/man

Based on the requirements file, in the last step cmt config has created cmthome/setup.sh, which has to be sourced before each usage of the ATLAS Software to set up the environment correctly. This next step offers real issues with the platform:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
AtlasLogin: Configuration problem - CMTCONFIG (Unsupported-opt) not available for /opt/atlas/15.1.0/AtlasOffline/15.1.0
AtlasLogin: Error - Unsupported-opt installation non-existent and no fallback found
#CMT---> Warning: The tag Unsupported-opt is not used in any tag expression. Please check spelling

These warnings are not meaningless and the environment got only set up partly. The ATLAS Software release can almost not be used in this situation. The errors had to be fixed. So I started debugging.

Here is what happens basically within cmthome/setup.sh: At first CMT gets called in a special way to produce a temporary shellscript. This is executed directly after creation. At the beginning it sets a lot of environment variables (including CMTCONFIG) and then it executes several other configuration scripts. After execution, the temporary shellscript is deleted.

And here the error is:
CMT does not detect CernVM as a proper system for ATLAS Software. Hence, within this named temporary shellscript CMTCONFIG is set to Unsupported-opt or NotSupported. The following configuration scripts partly work and partly throw errors (the errors we saw above).

This is the solution:
In principle — after migration to group-atlas — CernVM should be very okay as platform. So I decided to override CMTCONFIG with the value i686-slc4-gcc34-opt in the temporary shellscript by a call to sed. This has to happen within cmthome/setup.sh, after creation and before execution. The value i686-slc4-gcc34-opt is totally correct for a 32bit CernVM after migration to group-atlas.

This is the modified cmthome/setup.sh:

# echo "Setting standalone package"
 
if test "${CMTROOT}" = ""; then
  CMTROOT=/opt/atlas/15.1.0/CMT/v1r20p20081118; export CMTROOT
fi
. ${CMTROOT}/mgr/setup.sh
 
tempfile=`${CMTROOT}/mgr/cmt -quiet build temporary_name`
if test ! $? = 0 ; then tempfile=/tmp/cmt.$$; fi
${CMTROOT}/mgr/cmt setup -sh -pack=cmt_standalone -path=/opt/atlas/15.1.0/cmthome  -no_cleanup $* >${tempfile}
echo "********************* CMTCONFIG hack start **********************"
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo ""
echo "** replacing with 'i686-slc4-gcc34-opt'"
sed -r 's/^CMTCONFIG=.+/export CMTCONFIG=i686-slc4-gcc34-opt/g' -i ${tempfile}
echo ""
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo "********** CMTCONFIG hack end: executing ${tempfile} **********"
. ${tempfile}
/bin/rm -f ${tempfile}

Now there were no more errors while setting up the environment:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
********************* CMTCONFIG hack start **********************
** appearances of CMTCONFIG in /tmp/fileMWrr85:
CMTCONFIG=NotSupported; export CMTCONFIG
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
CMTCONFIG="NotSupported"; export CMTCONFIG
 
** replacing with 'i686-slc4-gcc34-opt'
 
** appearances of CMTCONFIG in /tmp/fileMWrr85:
export CMTCONFIG=i686-slc4-gcc34-opt
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
export CMTCONFIG=i686-slc4-gcc34-opt
********** CMTCONFIG hack end: executing /tmp/fileMWrr85 **********

With this environment I was able to run the full chain of JobTransforms.

The solution is not clean. For instance, the output of cmt show path is not complete. So, I personally don’t like it very much. For the future I plan to find a solution that combines CernVM’s configUpdate to ATLAS’ config (which belongs to CernVM’s bootstrap process) and a local pacman installation. This should be much more clean.

Problems at Boston University’s ATLAS Software mirror

Jan-Philip Gehrcke — Fri, 19 Jun 2009 12:59:28 +0000

While installing an ATLAS Software release from Boston University’s mirror, I discovered a broken archive file. It was the fault of a bad network card. The mirror had to be rebuilt from scratch.

I started a pacman installation of a local ATLAS Software release, using Boston University’s mirror, because I needed the files in Chicago:

$ pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-BU:15.1.0+KV

I got this error, over and over again:

Error executing [tar -z -t -f /opt/atlas/15.1.0/ROOT_5_22_00a_i686_slc4_gcc34_opt.tar.gz] returns status code [
[...]
sw/lcg/app/releases/ROOT/5.22.00a/slc4_ia32_gcc34/root/lib/libPostscript.so
tar: Skipping to next header
gzip: stdin: invalid compressed data--crc error
gzip: stdin: invalid compressed data--length error
tar: Child returned status 1
tar: Error exit delayed from previous errors].

Using the mirror of Indiana University (am-IU), everything worked. After opening this savannah bug ticket, it turned out that a bad network card caused corrupted data arriving at Boston University. This explains why I alway got the same corrupted file when receiving data from Boston University. They took am-BU offline and rebuilt it.

Update: am-BU is up again!

CernVM on Nimbus: kernel problems

Jan-Philip Gehrcke — Sun, 14 Jun 2009 16:45:43 +0000

In the past days I tried to set up CernVM on a Nimbus cloud to get an ATLAS Software Release (local version) running. On this way some problems came up. One of them could be solved by instructing the Xen hypervisor to choose the proper Linux kernel.

The software components of CernVM can be managed with rPath’s package manager conary. The standard CernVM image is very slim. Important packages are missing, even a compiler. To transform this slim system into a platform that is able to run e.g. ATLAS Software (or any other LHC experiment software) correctly, one needs to “migrate” the system to the special pre-defined group, providing the required packages. In case of ATLAS Software one should migrate the standard CernVM to “group-atlas”. Among others then e.g. GCC, compat-libgfortran and libstdc++ get installed into the system.

The corresponding command is (must be executed as root):

$ conary migrate group-atlas --interactive

But it resulted in

Troves being installed appear to conflict:
glibc:lib -> /conary.rpath.com@rpl:devel//1/2.3.6-8.9-1[~!bootstrap,~glibc.tls,~nptl,~!xen is: x86_64]->/conary.rpath.com@rpl:devel//1/2.3.6-8.4-1[~!bootstrap,~glibc.tls,~nptl is: x86_64]
glibc:runtime -> /conary.rpath.com@rpl:devel//1/2.3.6-8.9-1[~!bootstrap,~glibc.tls,~nptl,~!xen is: x86_64]->/conary.rpath.com@rpl:devel//1/2.3.6-8.4-1[~!bootstrap,~glibc.tls,~nptl is: x86_64]

This caused confusion, but after discussing the issue some days in this savannah support ticket, the solution was found (thanks to Predrag Buncic, Tim Freeman and Artem Harutyunyan):

On Xen based systems the kernel is given to the hypervisor separately from the VM image. Nimbus deploys VMs with its own kernel (like EC2 does, too).

This is what the fresh CernVM 1.2.0 tells about itself (deployed on Nimbus Cloud)

$ uname -a
Linux tp-x001.ci.uchicago.edu 2.6.18-xen #2 SMP Wed Apr 16 12:47:36 CDT 2008 x86_64 x86_64 x86_64 GNU/Linux

Conary seems to get confused by this kernel (missing kernel modules), so that one has to extract the original kernel of CernVM and make it available to the Xen hypervisor. In contrast to EC2, where you can choose from a big pool of kernels and even submit your own kernels, Nimbus does not offer this option by default. Last year Artem Harutyunyan had the same problem. Tim Freeman, Nimbus developer, added the CernVM kernel to the cloud and made changes to the Nimbus client so the user could specify the kernel with deployment command. This change was not included into a public release of Nimbus’ cloud client.

I used this unique client to deploy a new VM with the named kernel (“alien5″), which is the kernel extracted from CernVM 1.01:

$ ./bin/cloud-client.sh --conf /path/to/cloud.properties --run --name cernvm_120x86_dotSSHadded.gz --hours 5 --kernel alien5

It started up without problems and showed this system information:

$ uname -a
Linux tp-x002.ci.uchicago.edu 2.6.21.7-5.smp.pae.gcc4.1.x86.i686.xen.domU #1 SMP Mon Oct 6 16:34:24 EDT 2008 i686 athlon i386 GNU/Linux

Now the migration ran with success:

$ conary migrate group-atlas --interactive
Resolving dependencies...The following updates will be performed:
Job 1 of 9:
    Install info-vcsa(:user)=1-1-0.1
Job 2 of 9:
    Install cernvm-plugin-releasemgr(:python :runtime)=20090223-3-1
[...]
Migrate erases all troves not referenced in the groups specified.
continue with migrate? [y/N] y
Applying update job 1 of 9:
    Install info-vcsa(:user)=1-1-0.1
Applying update job 2 of 9:
    Install cernvm-plugin-releasemgr(:python :runtime)=20090223-3-1
    Install compat-db4(:lib)=4.2-3-1
    Install compat-libgfortran(:lib)=4.1.2-1-1
    Install compat-libstdc++-slc3(:lib)=3.2.3-2-1
[...]
Applying update job 6 of 9:
    Install gcc(:devel :devellib :lib :runtime)=3.4.4-9.4-1
[...]
    Install xterm(:lib :runtime)=202-5.3-1
Applying update job 9 of 9:
[...]

After this I could start setting up a local ATLAS Software Release using pacman. There the next problem occured: Problems at Boston University’s ATLAS Software mirror

Resize ext3 file system in loopback file

Jan-Philip Gehrcke — Thu, 04 Jun 2009 17:10:18 +0000

CernVM for Xen comes as loopback file, containing an Ext3 file system of about 9 GB size, whereas about 8.5 GB are free. Using this free space I tried to set up an Offline ATLAS Software Release (15.1.0). But the filesystem ran full and pacman aborted the setup. The goal is to deploy Virtual Machines of this image within the Nimbus Cloud, which currently does not support additional partitions.

So I had to increase the size of the image / loopback file and to extend the filesystem afterwards. Therefore I basically used dd and resize2fs.

The following describes the way I did it in more detail:

This is the original image / loopback file with a size about 9 GB:

-rw-r--r--  1 root root  9822281728 May 27 14:35 cernvm-1.2.0-x86-root.ext3

At first I wanted to increase the filesize without touching the existing data. dd’s append mode is great for this, but therefore I had to compile a new version of dd because mine didn’t support append yet (I worked within a VM of Scientific Linux 4.7)

Coreutils deliver dd:

$ tar xzf coreutils-7.4.tar.gz
$ cd coreutils-7.4
$ ./configure --prefix=/opt
$ make install

Okay, now write 10000 MB of zeros to the end of the file. It is important to use oflag=append and conv=notrunc, to make sure that the file is appended properly (read this bugreport to learn more about append/notrunc):

$ /opt/bin/dd if=/dev/zero of=cernvm-1.2.0-x86-root.ext3 bs=1M count=10000 oflag=append conv=notrunc
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 65.1349 s, 161 MB/s

The file size increased by approximately 10 GB:

-rw-r--r--  1 root root 20308041728 Jun  3 13:19 cernvm-1.2.0-x86-root.ext3

Rename it to know what happened later on:

$ mv cernvm-1.2.0-x86-root.ext3 cernvm120_plus10GB_dd_appended.ext3

Mount it to analyze the filesystem: 6 % are used.

$ mount -o loop /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3 /mnt/cernVM
$ df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/hda1     ext3     4538124   4059432    248164  95% /
none         tmpfs      517248         0    517248   0% /dev/shm
/dev/hdc1     ext3    51605908  40866420   8118060  84% /mnt/big_filesystem
/mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3
              ext3     9441336    530724   8431012   6% /mnt/cernVM

Unmount it:

$ umount /mnt/cernVM/

With resize2fs it is now very easy to fit the filesystem within the loopback file to the actual size of the file. But therefore an up-to-date version is needed. e2fsprogs deliver resize2fs

$ tar xzf e2fsprogs-1.41.6.tar.gz
$ cd e2fsprogs-1.41.6
$ ./configure --prefix=/opt
$ make install

Now just apply resize2fs to the unmounted loopback file and it will resize the filesystem automatically (as you can see, it wants a filesystem check before):

$ /opt/sbin/resize2fs /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3
resize2fs 1.41.6 (30-May-2009)
Please run 'e2fsck -f /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3' first.
 
$ /opt/sbin/e2fsck -f /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3
e2fsck 1.41.6 (30-May-2009)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
root: 19050/1200576 files (0.1% non-contiguous), 170365/2398018 blocks
 
$ /opt/sbin/resize2fs /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3
resize2fs 1.41.6 (30-May-2009)
Resizing the filesystem on /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3 to 4958018 (4k) blocks.
The filesystem on /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3 is now 4958018 blocks long.

Great, it really worked. Now the filesystem is about 18 GB big and only 3 % are used:

$ mount -o loop /mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3 /mnt/cernVM
$ df -T
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/hda1     ext3     4538124   4059432    248164  95% /
none         tmpfs      517248         0    517248   0% /dev/shm
/dev/hdc1     ext3    51605908  40866420   8118060  84% /mnt/big_filesystem
/mnt/big_filesystem/cernvm120_plus10GB_dd_appended.ext3
              ext3    19522468    530724  18000148   3% /mnt/cernVM

Now the big ATLAS Software Release can be installed. Btw: Why CernVM? This is nicely described in this blogpost.