CernVM: local ATLAS Software — the clean solution

In my blog post CernVM: how to set up a local ATLAS Software Release, I presented a brutal approach how to override CVMFS (CernVM‘s filesystem with HTTP backend) to install a local ATLAS Software release. Now I worked out a very clean and smooth solution. This approach allows:

  • to use the local ATLAS Software without “hacking” anything
  • to use local software and software provided by CVMFS at the same time.

The introduction and motivation from CernVM: how to set up a local ATLAS Software Release is still valid. The rest can be considered as deprecated.

The objective is to install ATLAS Software 15.2.0 to the local filesystem of CernVM 1.2.0 using pacman. Therefore we need some room in the filesystem (around 10 GB). Regarding this, the blog post Resize ext3 file system in loopback file could be interesting for you. This step is necessary on e.g. Nimbus. If you work on Amazon’s EC2, you perhaps would like to embed an EBS volume to install the software there. Useful hints for this are given in this blog post: EC2: Install ATLAS Software to an EBS volume

Prepare installation — start off from a fresh CernVM

Assume a fresh (maybe with a bigger file system than by default) CernVM 1.2.0 just booted up. Bootstrap it to the following configuration (via web interface):

  • create the new linux user atlasuser
  • under Virtual Organization Configuration choose ATLAS — keep everything else as No

Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.

[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive

You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

Now switch to root via sudo su. Create the directory /opt/atlas-local, which will be the place to put all the local ATLAS related software like CMT and pacman and the ATLAS Software itself. When using an EBS volume, /opt/atlas-local would be a convenient mount point. Give all rights to this directory.

[root@bla opt]# mkdir /opt/atlas-local
[root@bla opt]# chmod 777 /opt/atlas-local

Now we can already start installing an ATLAS Software release using pacman. I do this stuff so often, so I’ve created a little script for this. It checks if pacman’s already there. If not, it downloads & extracts it. pacman then is set up and the ATLAS Software installing command gets invoked. To use this script, create a file /opt/atlas-local/ and fill it up with this content:

if [ -e ${PACMANDIR} ]; then
    echo --info: ${PACMANDIR} exists
    echo --info: cd ${PACMANDIR}/pacman-*
    cd ${PACMANDIR}/pacman-*
    echo --info: pwd: `pwd`
    echo --info: pacman setup
    echo --info: mkdir -p and cd to ${PACMANDIR}
    mkdir -p ${PACMANDIR}
    cd ${PACMANDIR}
    echo --info: download latest pacman
    echo --info: extract..
    tar xzf pacman-latest.tar.gz
    echo --info: cd pacman-*
    cd pacman-*
    echo --info: pwd: `pwd`
    echo --info: setup pacman..
echo --info: mkdir -p, cd to ${ATLASINSTALLDIR}
echo --info: pwd: `pwd`
echo --info: start installing ATLAS:
echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. I use Indiana University mirror (am-IU). You perhaps want to modify this. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.

If you don’t like using this script at all, then just grab out the important commands. But please think twice about the path where to install ATLAS Software. Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under /opt/atlas-local allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.

Start ATLAS Software installation using pacman

Be sure to work as atlasuser. It’s just more clean to install the software not as root. Then, source the /opt/atlas-local/ script from above.

[atlasuser@bla atlas-local]$ source
--info: mkdir -p and cd to /opt/atlas-local/pacman
--info: download latest pacman
pacman-latest.tar.gz' saved [856237/856237]
--info: extract..
--info: cd pacman-3.28 pacman-latest.tar.gz
--info: pwd: /opt/atlas-local/pacman/pacman-3.28
--info: setup pacman..
--info: mkdir -p, cd to /opt/atlas-local/15.2.0
--info: pwd: /opt/atlas-local/15.2.0
--info: start installing ATLAS:
--info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0

This will take a while (something around an hour for installing from am-IU to EC2 in US). Inbetween, you can log in with a second shell and proceed with the next important steps.

Make useful CVMFS content local

While working on the dirty solution, some problems came up. In particular, I had to hack around the CMTCONFIG problem. The CernVM collaborators have spent considerable amount of time to sort out these problems. So let’s profit from their work. I figured out that “taking over” their version of the Configuration Management Tool CMT and their cmthome directory is enough to get everything working properly.

Copy CernVM’s CMT from CVMFS to the local filesystem

Work as root. At first, copy the whole CMT directory from CVMFS (which has /opt/atlas/ as mountpoint) to /opt/atlas-local:

[root@bla ~]# cp -R /opt/atlas/software/sw/CMT /opt/atlas-local

Now CMT’s path changed, which means that it has to be re-installed. By doint this, /opt/atlas-local/CMT/v1r20p20090520/mgr/ gets refreshed. This is important to set up CMT’s environment later on.

[root@bla atlas-local]# cd /opt/atlas-local/CMT/v1r20p20090520/mgr/
[root@bla mgr]# ./INSTALL
       CMT installation terminated.
 cmt.exe is available on this site for:

CMT has successfully found its new location. Btw, let’s look at the currently existing directory structure of /opt/atlas-local:

[root@bla ~]# cd /opt/atlas-local
[root@bla atlas-local]# ls
15.2.0  CMT  pacman

It should look like that :-)

Create cmthome and modify the requirements file

The essential part of this step is to grab&modify CernVM’s requirements file for ATLAS Software releases. Work as atlasuser and copy the cmthome directory to your local ATLAS Software:

[atlasuser@bla atlas-local]$ cp -R /opt/atlas/software/15.2.0/15.2.0/cmthome /opt/atlas-local/15.2.0/

Modify the requirements file within that new cmthome directory so that SITEROOT points to the new local ATLAS Software:

[atlasuser@bla atlas-local]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ ls
Makefile  cleanup.csh  requirements  setup.csh
[atlasuser@bla cmthome]$ vi requirements
[... modify SITEROOT path ...]
[atlasuser@bla cmthome]$ cat requirements
set SITEROOT /opt/atlas-local/15.2.0
macro ATLAS_TEST_AREA ${HOME}/testarea
apply_tag projectArea
apply_tag opt
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc4-gcc34-opt

In the next step, this new requirements file will be parsed by CMT to create a setup script which will be necessary to set up the correct runtime environment for ATLAS Software.

Set up CMT environment, create the ATLAS Software setup script

Work as atlasuser. To set up the CMT environment, use the /opt/atlas-local/CMT/v1r20p20090520/mgr/, which was created while installing CMT.

[atlasuser@bla ~]$ source /opt/atlas-local/CMT/v1r20p20090520/mgr/

Now, cmt and cmt.exe should be in the path. You can check it by typing cm *TAB*.

Switch to the cmthome directory and use cmt config to create the setup script for the ATLAS Software runtime environment:

[atlasuser@bla ~]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ cmt config
sh: manpath: command not found
Configuring environment for standalone package.
CMT version v1r20p20090520.
System is Linux-i686
Creating setup scripts.
Creating cleanup scripts.

Interim report: installation is done!

Now we’re done. This new ATLAS Software installation is prepared for usage. Each of the previous steps is unique for this installation and does not have to be repeated. To use this ATLAS Software, only /opt/atlas-local/15.2.0/cmthome/ has to be invoked.

In case that /opt/atlas-local was the mountpoint to an EBS volume, this volume now contains a fully working ATLAS Software release. It may get attached to any VM running an operating system that supports ATLAS Software (Like CernVM + group-atlas, Scientific Linux 3/4, …) to start production runs with this specific installation.

Test the new local ATLAS Software: Run the Full Chain

To test & validate the new ATLAS Software installation, let’s run the Full Chain. An easy explanation of what this is and how to realize it with a shell script is given in this blog post: ATLAS Software: How to run The Full Chain

Create working directory, setup runtime environment

Work as atlasuser. Create a working directory. Many files will be produced while running the full chain.

[atlasuser@bla ~]$ mkdir atlasrun_15-2-0___1/
[atlasuser@bla ~]$ cd atlasrun_15-2-0___1

When the pacman installation from above has finished, you can set up the runtime environment for the new ATLAS Software installation.

This step always has to be executed before using the software after a new log in.

[atlasuser@bla atlasrun_15-2-0___1]$ source /opt/atlas-local/15.2.0/cmthome/ -tag=15.2.0
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
sh: manpath: command not found
AtlasLogin: WARNING - test directory [/home/atlasuser/testarea/15.2.0] doesn't exist - the runtime environment won't reflect it

Don’t forget the -tag parameter!
The manpath problem isn’t an acutal problem. In this blog post you can read how to make manpath available. But it’s not needed. The warning about the test directory is correct, but for our full chain test we don’t need the testarea. If you need it for your production run, then create the stated directory.

Run the full chain!

I prepared a little script to run it. It’s explained in ATLAS Software: How to run The Full Chain

Download the script an run it:

[atlasuser@bla atlasrun_15-2-0___1]$ wget
`' saved [1229/1229]
[atlasuser@bla atlasrun_15-2-0___1]$ source
`singlepart_singlepi' saved [961/961]
read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
real    0m42.774s
user    0m29.074s
sys     0m1.292s
read EVGEN file.. simulate.. create HITS file..-> simul.log
real    9m30.879s
user    9m2.227s
sys     0m4.936s
read HITS file.. digitize.. produce RDO file..-> digi.log
real    2m35.875s
user    2m12.147s
sys     0m5.116s
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
real    3m54.783s
user    3m17.422s
sys     0m6.427s
read ESD file.. convert.. produce AOD file..-> recoAOD.log
real    1m44.373s
user    1m31.754s
sys     0m3.687s


The full chain performed successfully with the local ATLAS Software installation. At the same time, we could have used the “remote ATLAS Software” via CVMFS. I verified this by

  • logging in with an additional shell to the same machine
  • creating another working directory
  • setting up the runtime environment for the remote ATLAS Software 15.2.0 by sourcing /opt/atlas/software/setupScripts/
  • running my full chain script

It resulted in

read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log
real    6m13.287s
user    0m31.648s
sys     0m1.441s
read EVGEN file.. simulate.. create HITS file..-> simul.log
real    25m53.893s
user    9m2.742s
sys     0m5.593s
read HITS file.. digitize.. produce RDO file..-> digi.log
real    15m2.761s
user    2m12.836s
sys     0m6.455s
read RDO file.. reconstruct.. produce ESD file..-> recoESD.log
real    22m21.185s
user    3m19.384s
sys     0m6.891s
read ESD file.. convert.. produce AOD file..-> recoAOD.log
real    3m10.793s
user    1m33.584s
sys     0m4.454s

Pay attention to the long real timings. The reason is clear: The files had to be transferred via HTTP from a proxy server providing the content of CVMFS.

One Pingback/Trackback