Acknowledgement

• Kate Keahey (Argnonne National Laboratory/The Globus Alliance/Nimbus team).
  She offered me a fabulous mentorship. I assess this at its true worth by looking at what I’ve learned throughout the summer just because of these great conversations . Thank you for all your support and dedication and for pushing me and the project forwards!

• Tim Freeman and David LaBissoniere (Argnonne National Laboratory/The Globus Alliance/Nimbus team).
  Wonderful, extensive and patient extra-premium-support-with-special-treatment regarding Nimbus/Workspace in the MUD. Thank you so much! You smoothed my technical way through the project.

• Ian Ward (creator of urwid, a console user interface library for Python).
  He supported me so great while I was implementing the user interface for Clobi’s Resource Manager using urwid‘s new SelectEventLoop technology. Talking to him saved so much valuable time. Thank you!

• Mitchell Garnaat (the creator of boto, a Python interface to Amazon Web Services) and the boto users mailinglist.
  Great and essential support since almost one year now. Thank you very much for answering many questions!

• Predrag Buncic (CernVM) and Artem Harutyunyan (ALICE@LHC).
  Thank you for your support regarding CernVM on Nimbus!

• Jakub Moscicki, Ulrik Egede, Johannes Elmsheuser and the rest of the Ganga crew.
  Thanks a lot for very important, effective and efficient support concerning the system behind Clobi and Clobi’s Ganga backend. You are great!

• Stefan Kluth and Stefan Stonjek (ATLAS@LHC, Max-Planck-Institut für Physik, Munich)
  Thank you for many helpful discussions, support and beautiful times in Munich!

• Borja Sotomayor (University of Chicago, Globus Alliance).
  He made a great job as GSoC mentoring organization administrator. Thank you for this and for introducing the MUD .

• Paul D Marshall.
  Thank you for discussions regarding dynamical deployment of computing resources.

• Xiaoming Gao.
  Thank you for the exciting cooperation regarding Virtual Block Stores for Nimbus.

• Alex Martelli.
  He answered an important Python question that most people could not answer. Smoothed my way to implement inter thread communication in Clobi’s Resource Manager.

Clobi @ Google Code

In this blog post I introduced Clobi, the result of this Google Summer of Code project. Today, I created a new project page for Clobi on Google Code: http://code.google.com/p/clobi/

As a first action, I pushed my local mercurial code repository into the online repository. You can browse the code here and you can look through the development history (the commits I’ve made) here.

Then I prepared a test release of all Clobi components. You can get it in the download section.

That’s all about Clobi for the next time. From tomorrow on, I will go on with my master thesis project in Physics (about Magnetic Particle Imaging). Next week, the ICMRM conference in Montanta starts and I’ve to make a poster for my contribution (abstract here).

I will go on with Clobi, when there is more free time .

Really? What happens, when the writing thread pushes a hugh amout of data into the pipe — with high frequency? I wanted to test this.

Analysis

From the writing thread, I pushed a string with fixed length L (and terminated by a newline) through the pipe; N times — one after the other, within a loop. In the other thread, I os.read() from the pipe, based on select.select() events and checked the length of each line returned by one os.read(). To evaluate the data, I needed a basic histogram. How to realize fast counting in Python? Here is a neat way to count events:

import collections
histogram = collections.defaultdict(int)
for event in events:
    histogram[event] +=1

If you access a non-existing key in a defaultdict(callable), it is initialized automatically by calling callable() without arguments. For the histogram this can be used by creating a defaultdict of type int. Accessing histogram[non-existing-key] first initializes the value with int(), which gives zero. So histogram[event] +=1 is secure and just counts your events without a need to pre-define them This is the way I used it:

lines = os.read(pipe_read,9999999).splitlines()
for line in lines:
    histogram[len(line)] += 1

I don’t need a big visualization. A sorted listing in histogram style is enough for me:

for key in sorted(histogram, key=histogram.get, reverse=1):
    print '%7d %s' % (histogram[key], key)

This sorts the dictionary by its values in reversed order (small first) and prints the key/value pairs with a bit of formatting.

The optimal result would be only one datapoint in the histogram: N times L. This is what I got for short strings. But with increasing L, I got a probability of about 10^-5 for chopped lines. For L = 150, I e.g. got ~100000 times 150 chars, and 1-3 times 1-149 chars.

This means that in some cases strings returned by os.read() looked like "thisisaline\nthisisaline\nthisis": The last few characters of the last line are chopped and will be returned with the next os.read(): “aline\nthisisaline\nthisisaline\n” — and so on. The good thing was, that the sum of all received characters always matched the number of chars written to the pipe (as expected, but I felt good by measuring it).

Result and solution

In extreme situations (long strings, high writing frequency), relying on the “one os.write(string), one os.read() returns entire string” principle is a bad idea, because in seldom cases lines will not be read entirely and, hence, get returned chopped. The solution is to accomplish some post-processing to reassamble chopped lines:

In line 2, I keep the trailing newline characters in the splitted parts. In lines 4-6, I check if the last part has a trailing newline. If not, then this is a chopped line! It is deleted from the list of lines and prepended to the next string returned by os.read() (see line 1). Before putting resulting lines into the histogram, trailing newlines are .rstrip()ped. Can you imagine what the histogram says now?

E.g. "6000000 150", which means 6000000 times received a string of length 150. That’s the N times L datapoint I wanted!

Conclusion

It was shown that one should not rely on simple os.write() and os.read() calls on an os.pipe() to accomplish a communication channel between two threads. A “communication protocol” has to be applied. Then, with a small post-processing and reassembling piece of code, the communication between the two treads is reliable and foreseeable: every trailing-newline-labeled-line written to the pipe by Thread 1 is read and evaluated entirely by Thread 2. Great .

• to use the local ATLAS Software without “hacking” anything
• to use local software and software provided by CVMFS at the same time.

The introduction and motivation from CernVM: how to set up a local ATLAS Software Release is still valid. The rest can be considered as deprecated.

The objective is to install ATLAS Software 15.2.0 to the local filesystem of CernVM 1.2.0 using pacman. Therefore we need some room in the filesystem (around 10 GB). Regarding this, the blog post Resize ext3 file system in loopback file could be interesting for you. This step is necessary on e.g. Nimbus. If you work on Amazon’s EC2, you perhaps would like to embed an EBS volume to install the software there. Useful hints for this are given in this blog post: EC2: Install ATLAS Software to an EBS volume

Prepare installation — start off from a fresh CernVM

Assume a fresh (maybe with a bigger file system than by default) CernVM 1.2.0 just booted up. Bootstrap it to the following configuration (via web interface):

• create the new linux user atlasuser
• under Virtual Organization Configuration choose ATLAS — keep everything else as No

Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.

[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive

You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

Now switch to root via sudo su. Create the directory /opt/atlas-local, which will be the place to put all the local ATLAS related software like CMT and pacman and the ATLAS Software itself. When using an EBS volume, /opt/atlas-local would be a convenient mount point. Give all rights to this directory.

[root@bla opt]# mkdir /opt/atlas-local
[root@bla opt]# chmod 777 /opt/atlas-local

Now we can already start installing an ATLAS Software release using pacman. I do this stuff so often, so I’ve created a little script for this. It checks if pacman’s already there. If not, it downloads & extracts it. pacman then is set up and the ATLAS Software installing command gets invoked. To use this script, create a file /opt/atlas-local/install_ATLAS-15-2-0.sh and fill it up with this content:

PACMANDIR=/opt/atlas-local/pacman
ATLASINSTALLDIR=/opt/atlas-local/15.2.0
if [ -e ${PACMANDIR} ]; then
    echo --info: ${PACMANDIR} exists
    echo --info: cd ${PACMANDIR}/pacman-*
    cd ${PACMANDIR}/pacman-*
    echo --info: pwd: `pwd`
    echo --info: pacman setup
    source setup.sh
else
    echo --info: mkdir -p and cd to ${PACMANDIR}
    mkdir -p ${PACMANDIR}
    cd ${PACMANDIR}
    echo --info: download latest pacman
    wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
    echo --info: extract..
    tar xzf pacman-latest.tar.gz
    echo --info: cd pacman-*
    cd pacman-*
    echo --info: pwd: `pwd`
    echo --info: setup pacman..
    source setup.sh
fi
 
echo --info: mkdir -p, cd to ${ATLASINSTALLDIR}
mkdir -p ${ATLASINSTALLDIR}
cd ${ATLASINSTALLDIR}
echo --info: pwd: `pwd`
 
echo --info: start installing ATLAS:
echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. I use Indiana University mirror (am-IU). You perhaps want to modify this. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.

If you don’t like using this script at all, then just grab out the important commands. But please think twice about the path where to install ATLAS Software. Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under /opt/atlas-local allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.

Start ATLAS Software installation using pacman

Be sure to work as atlasuser. It’s just more clean to install the software not as root. Then, source the /opt/atlas-local/install_ATLAS-15-2-0.sh script from above.

[atlasuser@bla atlas-local]$ source install_atlas_15-2-0.sh
--info: mkdir -p and cd to /opt/atlas-local/pacman
--info: download latest pacman
--07:58:25--  http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
[...]
pacman-latest.tar.gz' saved [856237/856237]
--info: extract..
--info: cd pacman-3.28 pacman-latest.tar.gz
--info: pwd: /opt/atlas-local/pacman/pacman-3.28
--info: setup pacman..
--info: mkdir -p, cd to /opt/atlas-local/15.2.0
--info: pwd: /opt/atlas-local/15.2.0
--info: start installing ATLAS:
--info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

This will take a while (something around an hour for installing from am-IU to EC2 in US). Inbetween, you can log in with a second shell and proceed with the next important steps.

Make useful CVMFS content local

While working on the dirty solution, some problems came up. In particular, I had to hack around the CMTCONFIG problem. The CernVM collaborators have spent considerable amount of time to sort out these problems. So let’s profit from their work. I figured out that “taking over” their version of the Configuration Management Tool CMT and their cmthome directory is enough to get everything working properly.

Copy CernVM’s CMT from CVMFS to the local filesystem

Work as root. At first, copy the whole CMT directory from CVMFS (which has /opt/atlas/ as mountpoint) to /opt/atlas-local:

[root@bla ~]# cp -R /opt/atlas/software/sw/CMT /opt/atlas-local

Now CMT’s path changed, which means that it has to be re-installed. By doint this, /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh gets refreshed. This is important to set up CMT’s environment later on.

[root@bla atlas-local]# cd /opt/atlas-local/CMT/v1r20p20090520/mgr/
[root@bla mgr]# ./INSTALL
============================================
       CMT installation terminated.
           --------------------
 cmt.exe is available on this site for:
    Darwin-PowerMacintosh
    Darwin-i386
    Linux-i686
    Linux-x86_64
    VisualC
============================================

CMT has successfully found its new location. Btw, let’s look at the currently existing directory structure of /opt/atlas-local:

[root@bla ~]# cd /opt/atlas-local
[root@bla atlas-local]# ls
15.2.0  CMT  install_atlas_15-2-0.sh  pacman

It should look like that

Create cmthome and modify the requirements file

The essential part of this step is to grab&modify CernVM’s requirements file for ATLAS Software releases. Work as atlasuser and copy the cmthome directory to your local ATLAS Software:

[atlasuser@bla atlas-local]$ cp -R /opt/atlas/software/15.2.0/15.2.0/cmthome /opt/atlas-local/15.2.0/

Modify the requirements file within that new cmthome directory so that SITEROOT points to the new local ATLAS Software:

[atlasuser@bla atlas-local]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ ls
Makefile  cleanup.csh  cleanup.sh  requirements  setup.csh  setup.sh
[atlasuser@bla cmthome]$ vi requirements
[... modify SITEROOT path ...]
[atlasuser@bla cmthome]$ cat requirements
#---------------------------------------------------------------------
set CMTSITE STANDALONE
set SITEROOT /opt/atlas-local/15.2.0
macro ATLAS_TEST_AREA ${HOME}/testarea
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag projectArea
macro SITE_PROJECT_AREA ${SITEROOT}
macro EXTERNAL_PROJECT_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc4-gcc34-opt

In the next step, this new requirements file will be parsed by CMT to create a setup script which will be necessary to set up the correct runtime environment for ATLAS Software.

Set up CMT environment, create the ATLAS Software setup script

Work as atlasuser. To set up the CMT environment, use the /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh, which was created while installing CMT.

[atlasuser@bla ~]$ source /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh

Now, cmt and cmt.exe should be in the path. You can check it by typing cm *TAB*.

Switch to the cmthome directory and use cmt config to create the setup script for the ATLAS Software runtime environment:

[atlasuser@bla ~]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ cmt config
sh: manpath: command not found
------------------------------------------
Configuring environment for standalone package.
CMT version v1r20p20090520.
System is Linux-i686
------------------------------------------
Creating setup scripts.
Creating cleanup scripts.

Interim report: installation is done!

Now we’re done. This new ATLAS Software installation is prepared for usage. Each of the previous steps is unique for this installation and does not have to be repeated. To use this ATLAS Software, only /opt/atlas-local/15.2.0/cmthome/setup.sh has to be invoked.

In case that /opt/atlas-local was the mountpoint to an EBS volume, this volume now contains a fully working ATLAS Software release. It may get attached to any VM running an operating system that supports ATLAS Software (Like CernVM + group-atlas, Scientific Linux 3/4, …) to start production runs with this specific installation.

Test the new local ATLAS Software: Run the Full Chain

To test & validate the new ATLAS Software installation, let’s run the Full Chain. An easy explanation of what this is and how to realize it with a shell script is given in this blog post: ATLAS Software: How to run The Full Chain

Create working directory, setup runtime environment

Work as atlasuser. Create a working directory. Many files will be produced while running the full chain.

[atlasuser@bla ~]$ mkdir atlasrun_15-2-0___1/
[atlasuser@bla ~]$ cd atlasrun_15-2-0___1

When the pacman installation from above has finished, you can set up the runtime environment for the new ATLAS Software installation.

This step always has to be executed before using the software after a new log in.

[atlasuser@bla atlasrun_15-2-0___1]$ source /opt/atlas-local/15.2.0/cmthome/setup.sh -tag=15.2.0
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
AtlasLogin: WARNING - test directory [/home/atlasuser/testarea/15.2.0] doesn't exist - the runtime environment won't reflect it

Don’t forget the -tag parameter!
The manpath problem isn’t an acutal problem. In this blog post you can read how to make manpath available. But it’s not needed. The warning about the test directory is correct, but for our full chain test we don’t need the testarea. If you need it for your production run, then create the stated directory.

Run the full chain!

I prepared a little script to run it. It’s explained in ATLAS Software: How to run The Full Chain

Download the script an run it:

[atlasuser@bla atlasrun_15-2-0___1]$ wget gehrcke.de/gsoc/athena_whole_chain_1event.sh
[...]
`athena_whole_chain_1event.sh' saved [1229/1229]
[atlasuser@bla atlasrun_15-2-0___1]$ source athena_whole_chain_1event.sh
[...]
`singlepart_singlepi' saved [961/961]
 
read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
 
real    0m42.774s
user    0m29.074s
sys     0m1.292s
 
read EVGEN file.. simulate.. create HITS file..-> simul.log
 
real    9m30.879s
user    9m2.227s
sys     0m4.936s
 
read HITS file.. digitize.. produce RDO file..-> digi.log
 
real    2m35.875s
user    2m12.147s
sys     0m5.116s
 
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
 
real    3m54.783s
user    3m17.422s
sys     0m6.427s
 
read ESD file.. convert.. produce AOD file..-> recoAOD.log
 
real    1m44.373s
user    1m31.754s
sys     0m3.687s

Success!

The full chain performed successfully with the local ATLAS Software installation. At the same time, we could have used the “remote ATLAS Software” via CVMFS. I verified this by

• logging in with an additional shell to the same machine
• creating another working directory
• setting up the runtime environment for the remote ATLAS Software 15.2.0 by sourcing /opt/atlas/software/setupScripts/setupAtlasProduction_15.2.0.sh
• running my full chain script

It resulted in

read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
 
real    6m13.287s
user    0m31.648s
sys     0m1.441s
 
read EVGEN file.. simulate.. create HITS file..-> simul.log
 
real    25m53.893s
user    9m2.742s
sys     0m5.593s
 
read HITS file.. digitize.. produce RDO file..-> digi.log
 
real    15m2.761s
user    2m12.836s
sys     0m6.455s
 
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
 
real    22m21.185s
user    3m19.384s
sys     0m6.891s
 
read ESD file.. convert.. produce AOD file..-> recoAOD.log
 
real    3m10.793s
user    1m33.584s
sys     0m4.454s

Pay attention to the long real timings. The reason is clear: The files had to be transferred via HTTP from a proxy server providing the content of CVMFS.

What is “The Full Chain”?

The so-called JobTransforms are python scripts that are used to run production tasks. They take an input file, a set of parameters and “transform” the input into one or more output files. Combined in the correct order, they build up The Full Chain. I will summarize the elements of the chain now:

• Step 1) Event generation: a virtual particle gets created with a specific energy and direction.
  Input: particle definition file. Output: EVGEN file.
• Step 2) Simulation: the interaction between this particle and the detector is simulated.
  Input: EVGEN file. Output: HITS file.
• Step 3) Digitization: the ATLAS detector output is calculated.
  Input: HITS file. Output: RDO file.
• Step 4) Reconstruction: times and voltages are reconstructed into tracks and energy deposits.
  Input: RDO file. Output: ESD file.
• Step 5) Conversion: only keep the most important data from the last step.
  Input: ESD file. Output: AOD file.

The following picture from here visualizes the chain and — you might be interested in that — shows where real data from the LHC will play a role in reality:

The Full Chain

How to realize “The Full Chain”

As you know, the input of step 1 is a small user-given file defining particle parameters. This file on my webserver describes a single pion with specific energy and direction. The content basically is the following:

# Single pi+ in log(E) between 200 MeV and 2 TeV
from AthenaCommon.AlgSequence import AlgSequence
topAlg = AlgSequence("TopAlg")
from ParticleGenerator.ParticleGeneratorConf import ParticleGenerator
topAlg += ParticleGenerator()
ParticleGenerator = topAlg.ParticleGenerator
# For DEBUG output from ParticleGenerator.
ParticleGenerator.OutputLevel = 2
ParticleGenerator.orders = [
  "PDGcode: constant 211",
  "e: log 200. 2000000.",
  "eta: flat -5.5 5.5",
  "phi: flat -3.14159 3.14159"
  ]
from EvgenJobOptions.SingleEvgenConfig import evgenConfig

At this point it’s not important to understand each line of this file. It works

The following shellscript downloads this file and initializes The Full Chain for exactly one event of this particle. During execution, it measures timings.

wget http://gehrcke.de/gsoc/singlepart_singlepi
mv singlepart_singlepi CSC.007410.singlepart_singlepi+_logE.py
 
echo -e "\nread CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log"
time csc_evgen_trf.py 007410 1 1 765432 CSC.007410.singlepart_singlepi+_logE.py EVGEN_007410_00001.pool.root > evgen.log
 
echo -e "\nread EVGEN file.. simulate.. create HITS file..-> simul.log"
time csc_simul_trf.py EVGEN_007410_00001.pool.root HITS_007410_00001.pool.root NONE 1 0 452368 "ATLAS-CSC-02-00-00" 0 0 "QGSP_BERT" CalHits.py > simul.log
 
echo -e "\nread HITS file.. digitize.. produce RDO file..-> digi.log"
time csc_digi_trf.py HITS_007410_00001.pool.root RDO_007410_00001.pool.root 1 0 "ATLAS-CSC-02-00-00" 740581234 29402491 'NONE' 'NONE' CalHits.py 'NONE' 'AtRndmGenSvc' 'QGSP_EMV' 'NONE' > digi.log
 
echo -e "\nread RDO file.. reconstruct.. produce ESD file..-> recoESD.log"
time csc_recoESD_trf.py RDO_007410_00001.pool.root ESD_007410_00001.pool.root 'NONE' 1 0 "ATLAS-CSC-02-00-00" 'NONE' > recoESD.log
 
echo -e "\nread ESD file.. convert.. produce AOD file..-> recoAOD.log"
time csc_recoAOD_trf.py ESD_007410_00001.pool.root AOD_007410_00001.pool.root 1 0 "ATLAS-CSC-02-00-00" 'NONE' > recoAOD.log

You can download the script here.

After setting up the runtime environment for your ATLAS Software installation, you can simply download and execute this script. It should work! I tested it with ATLAS Software 15.2.0, as you can see in this blog post: CernVM: local ATLAS Software — the clean solution

Start off from a fresh CernVM — migrate to group-atlas

Start a fresh CernVM 1.2.0 instance on EC2 (I used ami-a50cebcc, aki-9b00e5f2). Bootstrap it to the following configuration (via web interface):

• create the new linux user atlasuser
• under Virtual Organization Configuration choose ATLAS — keep everything else as No

Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.

[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive

You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems.

With migration to group-atlas, gcc gets installed. We will need it soon!

Create and attach EBS volume, make file system

Create a new EBS volume within the same availability zone as the CernVM instance just startet up. I chose 15 GB size. This should be enough to hold one ATLAS Software release plus a bit of additional stuff.

Attach this new EBS volume as e.g. /dev/sdh to the CernVM instance.

Regarding CernVM, now e2fsprogs has to be installed to create a filesystem on the fresh block device /dev/sdh. Therefore we need the compiler.
Work as root, download&install:

[root@bla ~]# wget http://prdownloads.sourceforge.net/e2fsprogs/e2fsprogs-1.41.6.tar.gz
[...]
`e2fsprogs-1.41.6.tar.gz' saved [4422395/4422395]
[root@bla ~]# tar xzf e2fsprogs-1.41.6.tar.gz
[root@bla ~]# cd e2fsprogs-1.41.6
[root@bla e2fsprogs-1.41.6]# ./configure > cfg.log
[root@bla e2fsprogs-1.41.6]# make install > makeinst.log
make[1]: texi2dvi: Command not found
make[1]: [libext2fs.dvi] Error 127 (ignored)
/usr/bin/install: cannot stat `libext2fs.info*': No such file or directory
make[1]: [install-doc-libs] Error 1 (ignored)
gzip: /usr/share/info/libext2fs.info*: No such file or directory
make[1]: [install-doc-libs] Error 1 (ignored)

The warnings are no problem. Now mkfs.ext3 can be used:

[root@bla e2fsprogs-1.41.6]# /sbin/mkfs.ext3 /dev/sdh
mke2fs 1.41.6 (30-May-2009)
/dev/sdh is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
[...]
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

Mount it

Now the mountpoint has to be chosen. The objective is to install everything needed by the ATLAS Software to this EBS volume so that moving around with this volume between different instances (which are running operating systems supporting ATLAS Software) becomes possible.

Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under the mountpoint allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.

I chose the mountpoint /opt/atlas-local. Now and in the future, I will always have to mount the EBS volume to this directory.

Work as root: create mountpoint, mount device and give all rights:

[root@bla opt]# mkdir /opt/atlas-local
[root@bla opt]# mount /dev/sdh /opt/atlas-local
[root@bla opt]# chmod 777 /opt/atlas-local

Now log in as atlasuser and check out the new space:

[atlasuser@bla atlas-local]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1              9293008   1999624   6821316  23% /
none                    890952         0    890952   0% /dev/shm
/dev/sdh              15481840    169592  14525816   2% /opt/atlas-local

Install ATLAS Software to this EBS volume

To install a new ATLAS Software Release to a subdirectory of the mountpoint, you can use my following script which automatically checks out pacman and invokes the installing command:

PACMANDIR=/opt/atlas-local/pacman
ATLASINSTALLDIR=/opt/atlas-local/15.2.0
if [ -e ${PACMANDIR} ]; then
    echo --info: ${PACMANDIR} exists
    echo --info: cd ${PACMANDIR}/pacman-*
    cd ${PACMANDIR}/pacman-*
    echo --info: pwd: `pwd`
    echo --info: pacman setup
    source setup.sh
else
    echo --info: mkdir -p and cd to ${PACMANDIR}
    mkdir -p ${PACMANDIR}
    cd ${PACMANDIR}
    echo --info: download latest pacman
    wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
    echo --info: extract..
    tar xzf pacman-latest.tar.gz
    echo --info: cd pacman-*
    cd pacman-*
    echo --info: pwd: `pwd`
    echo --info: setup pacman..
    source setup.sh
fi
 
echo --info: mkdir -p, cd to ${ATLASINSTALLDIR}
mkdir -p ${ATLASINSTALLDIR}
cd ${ATLASINSTALLDIR}
echo --info: pwd: `pwd`
 
echo --info: start installing ATLAS:
echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. Adjust the version numbers to your needs. I use Indiana University mirror (am-IU). You perhaps want to modify this, too. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.

Furthermore, in CernVM: local ATLAS Software — the clean solution you can read about:

• how to set up CMT
• how to create a cmthome directory and a valid requirements file
• how to set up the runtime environment for an ATLAS Software Release

Have fun and let me know what you think!

You perhaps know Amazon Web Services (AWS). If not: hurry up: it’s Infrastructure-as-a-Service-cloud-computing at its finest . AWS consists of real web services: they all are controlled over HTTP with different “abstraction levels” — the different Application Programming Interfaces: Some services can be controlled via the REST API, some via the Query API, some via the SOAP API and mostly a mixture is allowed. Even the term REST-Query API appears. The following table gives an overview of the services and the corresponding available APIs, as stated by AWS (taken out of the documentations):

The red fields indicate that I think there something is wrong. Read on to learn why.

At first, I was a bit confused about the different APIs. In particular, it was not clear to me why we have to distinguish REST and Query API. In both cases the requests to AWS are based on standard HTTP protocol requests like GET and POST. REST and Query API do not use a protocol on top of HTTP, in contrast to the SOAP API: it relies on XML documents transported via HTTP.

Additionally, it was not clear to me why we have to have at least three different APIs. Is that necessary? What are pros and cons?

If you look around for client libraries providing the AWS API for famous programming languages like Python, Java, Ruby and so on, you mostly find libraries implementing the REST and Query API. SOAP API — which is available for the majority of the services — is implemented quite seldom. This confused me.

Although the cooperative guys in the boto mainlinglist already helped me on this topic, I looked for a book to read about this in more detail. In Programming Amazon Web Services by James Murty, I found a very good summary about the topic and — with that — answers to my questions. I will now quote big parts, because I think that it could be very useful to some people out there:

Interfaces: REST and Query Versus SOAP

AWS infrastructure services are made available through three separate APIs: REST, Query, and SOAP. In this book we will focus only on the REST and Query APIs and will not demonstrate how to use the SOAP APIs. We have a number of reasons for doing this, reasons which will become clearer after a brief explanation of the differences between the interfaces.

REST interfaces

The REST interfaces offered by AWS use only the standard components of HTTP request messages to represent the API action that is being performed. These components include:

• HTTP method: describes the action the request will perform
• Universal Resource Identifier (URI): path and query elements that indicate the resource on which the action will be performed
• Request Headers: pieces of metadata that provide more information about the request itself or the requester
• Request Body: the data on which the service will perform an action

Web services that use these components to describe operations are often termed RESTful services, a categorization for services that use the HTTP protocol as it was originally intended.

Query interfaces

The Query interfaces offered by AWS also use the standard components of the HTTP protocol to represent API actions; however these interfaces use them in a different way. Query requests rely on parameters, simple name and value pairs, to express both the action the service will perform and the data the action will be performed on. When you are using a Query interface, the HTTP envelope serves merely as a way of delivering these parameters to the service.

To perform an operation with a Query interface, you can express the parameters in the URI of a GET request, or in the body of a POST request. The method component of the HTTP request merely indicates where in the message the parameters are expressed, while the URI may or may not indicate a resource to act upon.

These characteristics mean that the Query interfaces cannot be considered properly RESTful because they do not use the HTTP message components to fully describe API operations. Instead, the Query interfaces can be considered REST-like, because although they do things differently, they still only use standard HTTP message components to perform operations.

SOAP interfaces

The SOAP interfaces offered by AWS use XML documents to express the action that will be performed and the data that will be acted upon. These SOAP XML documents are constructed as another layer on top of the underlying HTTP request, such that all the information about the operation is moved out of the HTTP message and encapsulated in the SOAP message instead.

[...]

The approach used in the SOAP interfaces are very different from those used by the REST and Query interfaces. Operations expressed in SOAP messages are completely divorced from the underlying HTTP message used to transmit the request, and the HTTP message components, such as method and URI, reveal nothing about the operation being performed.

The main reason we eschew the SOAP interface in this book is because we believe that SOAP interfaces in general add unnecessary complexity and overhead, effectively spoiling the simplicity and transparency that can make web services such powerful and flexible tools. [...] We are not alone in feeling this way. According to Amazon staff members, a vast majority of developers use the REST-based APIs to interact with AWS.

Thank you, James Murty for providing a deeper understanding. With this knowledge, I looked through the developer guides of the different services to check if everything just learned really fits to the actual API realizations and their terms (given by AWS). I found some inconsistencies:

Amazon’s DevPay service: two names for the same thing

As you can read here, this service is managed via SOAP API or “REST-Query” API. “REST-Query API” characterizes exactly the same method of invoking requests as the “Query API” anywhere else within AWS, as stated in the DevPay documentation: “REST-Query requests are simple HTTPS requests, using the GET or POST method with query parameters in the URL”.

An example REST-Query API request for DevPay:

https://ls.amazonaws.com/?Action=ActivateHostedProduct​&ActivationKey=XX​&ProductToken=XX&AWSAccessKeyId=XX&Version=XX&Timestamp=XX​&Signature=XYZ

For comparison, an example Query API request for EC2:

https://ec2.amazonaws.com/?Action=DescribeImages&ImageId.1=XX&Version=XX&Expires=XX&Signature=XX&SignatureVersion=XX&SignatureMethod=XX&AWSAccessKeyId=XX

Hence, this method should simply get called “Query API”, too.

Amazon’s Mechanical Turk service: it’s not RESTful

This service can be controlled via SOAP or REST API — says Amazon. But when we look at the description of this “REST API”, it’s doubtable that “REST API” is the right term here: “REST requests are simple HTTP requests, using either the GET method with parameters in the URL, or the POST method with parameters in the POST body”

Let’s look at an example request of this Mechanical Turk “REST API”:

http://mechanicalturk.amazonaws.com/?Service=AWSMechanicalTurkRequester&AWSAccessKeyId=XX&Version=XX&Operation=XX&Signature=XX&Timestamp=XX&ResponseGroup.0=XX&ResponseGroup.1=XX

There is almost no difference to the EC2 Query API request example above: It’s just a GET request and a bunch of parameters. From my perspective, the URI does not represent the resource on which the action will be performed, although ?Service=AWSMechanicalTurkRequester possibly looks a bit like that. But all the parameters afterwards argue against calling this API RESTful. Hence, this API should have been called Query API, too.

Resulting, for DevPay and Mechanical Turk the corrections in the table from above should look like this:

What do you think? I did not read Fielding’s thesis, so maybe I’ve still too few knowledge here.

Currently, the access to the providing server(s) must come from within CERN network. I am working with VMs in Chicago (Teraport, Nimbus cloud), don’t have a CERN account and doubt that the performance of CVMFS would have been good enough between Geneva and Chicago for my testing purposes. Hence, I tried to set up an ATLAS Software release locally, which required some special work.

Everything is based on CernVM 1.2.0 and ATLAS Software 15.1.0. The current solution is dirty and I am still looking for the optimal way. In the end, I would like to have an easy “switch” option between

• using CVMFS and being dependent of way software-providing webservers and
• keeping everything local

[Update]

The introduction/motivation is still valid and the parts below are still true. But in the meanwhile I found a much cleaner solution. It allows to use local and remote ATLAS Software at the same time and does not need any “hacking”. You can read about it in this blog post: CernVM: local ATLAS Software — the clean solution

[/Update]

To make room for the heavy ATLAS Software, I at first had to extend the filesystem within CernVM’s loopback file (I worked on the Xen-based Nimbus cloud, which did not support additional partitions at that time). This procedure is described in this blog post:
Resize ext3 file system in loopback file

Btw: While acting on CernVM’s image and filesystem locally with the intention to extend everything, I added the /root/.ssh folder to make public key insertion of Nimbus and EC2 possible. The directory does not exist by default. Predrag revealed that he will add it in the next release (should be 1.3.0).

Okay, imagine you are logged in as root to a booted fresh CernVM (only with the modifications named above: bigger filesystem, /root/.ssh added). If the Xen hypervisor added a correct kernel to the system, you can start right away by migrating the system to group-atlas:

$ conary migrate group-atlas --interactive

If this throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

After migration to group-atlas, the system is ready for a pacman installation of ATLAS Software. Set up pacman:

$ wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
$ tar xzf pacman-latest.tar.gz
$ cd pacman-3.28/
$ source setup.sh

In this case, the ATLAS Software release (+KitValidation) should get installed to /opt/atlas/15.1.0:

$ mkdir /opt/atlas/15.1.0
$ cd /opt/atlas/15.1.0
$ pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-BU:15.1.0+KV

-pretend-platform SLC-4 is necessary, since we’re not on the standard ATLAS platform. Installing takes quite a while. To optimize the process, I chose Boston University mirror, which is good from the perspective of Chicago. In the end you should see

##################################################
##   AtlasProduction 15.1.0 Validation [  OK  ]
##################################################

Now the exciting part comes: setting up a working environment for the software.

Set up a cmthome directory and a requirements file:

$ mkdir /opt/atlas/15.1.0/cmthome
$ vi /opt/atlas/15.1.0/cmthome/requirements
$ cat /opt/atlas/15.1.0/cmthome/requirements
set CMTSITE STANDALONE
set SITEROOT /opt/atlas/15.1.0
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag noTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

This requirements file should be enough to use the release for local production runs (standalone and no testarea).

Now set up CMT, cd to the folder of the new requirements file and configurate CMT:

$ source /opt/atlas/15.1.0/CMT/v1r20p20081118/mgr/setup.sh
$ cd /opt/atlas/15.1.0/cmthome/
$ cmt config
sh: manpath: command not found
------------------------------------------
Configuring environment for standalone package.
CMT version v1r20p20081118.
System is Linux-i686
------------------------------------------
Creating setup scripts.
Creating cleanup scripts.

See the manpath issue? Not a real problem, but if you like to fix it for the future:

$ vi /bin/manpath
$ cat /bin/manpath
#!/bin/bash
man --path $*
$ chmod u+x /bin/manpath
$ conary update man
[...]
$ manpath
/usr/local/share/man:/usr/share/man:/usr/X11R6/man

Based on the requirements file, in the last step cmt config has created cmthome/setup.sh, which has to be sourced before each usage of the ATLAS Software to set up the environment correctly. This next step offers real issues with the platform:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
AtlasLogin: Configuration problem - CMTCONFIG (Unsupported-opt) not available for /opt/atlas/15.1.0/AtlasOffline/15.1.0
AtlasLogin: Error - Unsupported-opt installation non-existent and no fallback found
#CMT---> Warning: The tag Unsupported-opt is not used in any tag expression. Please check spelling

These warnings are not meaningless and the environment got only set up partly. The ATLAS Software release can almost not be used in this situation. The errors had to be fixed. So I started debugging.

Here is what happens basically within cmthome/setup.sh: At first CMT gets called in a special way to produce a temporary shellscript. This is executed directly after creation. At the beginning it sets a lot of environment variables (including CMTCONFIG) and then it executes several other configuration scripts. After execution, the temporary shellscript is deleted.

And here the error is:
CMT does not detect CernVM as a proper system for ATLAS Software. Hence, within this named temporary shellscript CMTCONFIG is set to Unsupported-opt or NotSupported. The following configuration scripts partly work and partly throw errors (the errors we saw above).

This is the solution:
In principle — after migration to group-atlas — CernVM should be very okay as platform. So I decided to override CMTCONFIG with the value i686-slc4-gcc34-opt in the temporary shellscript by a call to sed. This has to happen within cmthome/setup.sh, after creation and before execution. The value i686-slc4-gcc34-opt is totally correct for a 32bit CernVM after migration to group-atlas.

This is the modified cmthome/setup.sh:

# echo "Setting standalone package"
 
if test "${CMTROOT}" = ""; then
  CMTROOT=/opt/atlas/15.1.0/CMT/v1r20p20081118; export CMTROOT
fi
. ${CMTROOT}/mgr/setup.sh
 
tempfile=`${CMTROOT}/mgr/cmt -quiet build temporary_name`
if test ! $? = 0 ; then tempfile=/tmp/cmt.$$; fi
${CMTROOT}/mgr/cmt setup -sh -pack=cmt_standalone -path=/opt/atlas/15.1.0/cmthome  -no_cleanup $* >${tempfile}
echo "********************* CMTCONFIG hack start **********************"
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo ""
echo "** replacing with 'i686-slc4-gcc34-opt'"
sed -r 's/^CMTCONFIG=.+/export CMTCONFIG=i686-slc4-gcc34-opt/g' -i ${tempfile}
echo ""
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo "********** CMTCONFIG hack end: executing ${tempfile} **********"
. ${tempfile}
/bin/rm -f ${tempfile}

Now there were no more errors while setting up the environment:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
********************* CMTCONFIG hack start **********************
** appearances of CMTCONFIG in /tmp/fileMWrr85:
CMTCONFIG=NotSupported; export CMTCONFIG
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
CMTCONFIG="NotSupported"; export CMTCONFIG
 
** replacing with 'i686-slc4-gcc34-opt'
 
** appearances of CMTCONFIG in /tmp/fileMWrr85:
export CMTCONFIG=i686-slc4-gcc34-opt
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
export CMTCONFIG=i686-slc4-gcc34-opt
********** CMTCONFIG hack end: executing /tmp/fileMWrr85 **********

With this environment I was able to run the full chain of JobTransforms.

The solution is not clean. For instance, the output of cmt show path is not complete. So, I personally don’t like it very much. For the future I plan to find a solution that combines CernVM’s configUpdate to ATLAS’ config (which belongs to CernVM’s bootstrap process) and a local pacman installation. This should be much more clean.