In my blog post CernVM: how to set up a local ATLAS Software Release, I presented a brutal approach how to override CVMFS (CernVM‘s filesystem with HTTP backend) to install a local ATLAS Software release. Now I worked out a very clean and smooth solution. This approach allows:
- to use the local ATLAS Software without “hacking” anything
- to use local software and software provided by CVMFS at the same time.
The introduction and motivation from CernVM: how to set up a local ATLAS Software Release is still valid. The rest can be considered as deprecated.
The objective is to install ATLAS Software 15.2.0 to the local filesystem of CernVM 1.2.0 using pacman. Therefore we need some room in the filesystem (around 10 GB). Regarding this, the blog post Resize ext3 file system in loopback file could be interesting for you. This step is necessary on e.g. Nimbus. If you work on Amazon’s EC2, you perhaps would like to embed an EBS volume to install the software there. Useful hints for this are given in this blog post: EC2: Install ATLAS Software to an EBS volume
Prepare installation — start off from a fresh CernVM
Assume a fresh (maybe with a bigger file system than by default) CernVM 1.2.0 just booted up. Bootstrap it to the following configuration (via web interface):
- create the new linux user
atlasuser - under Virtual Organization Configuration choose ATLAS — keep everything else as No
Log in via ssh, using the new atlasuser account. At first, the necessary system components have to be installed using Conary.
[atlasuser@bla ~]$ sudo conary migrate group-atlas --interactive
You will need the “admin password” you set at bootstrap. If this migration step throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems
Now switch to root via sudo su. Create the directory /opt/atlas-local, which will be the place to put all the local ATLAS related software like CMT and pacman and the ATLAS Software itself. When using an EBS volume, /opt/atlas-local would be a convenient mount point. Give all rights to this directory.
[root@bla opt]# mkdir /opt/atlas-local [root@bla opt]# chmod 777 /opt/atlas-local
Now we can already start installing an ATLAS Software release using pacman. I do this stuff so often, so I’ve created a little script for this. It checks if pacman’s already there. If not, it downloads & extracts it. pacman then is set up and the ATLAS Software installing command gets invoked. To use this script, create a file /opt/atlas-local/install_ATLAS-15-2-0.sh and fill it up with this content:
PACMANDIR=/opt/atlas-local/pacman ATLASINSTALLDIR=/opt/atlas-local/15.2.0 if [ -e ${PACMANDIR} ]; then echo --info: ${PACMANDIR} exists echo --info: cd ${PACMANDIR}/pacman-* cd ${PACMANDIR}/pacman-* echo --info: pwd: `pwd` echo --info: pacman setup source setup.sh else echo --info: mkdir -p and cd to ${PACMANDIR} mkdir -p ${PACMANDIR} cd ${PACMANDIR} echo --info: download latest pacman wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz echo --info: extract.. tar xzf pacman-latest.tar.gz echo --info: cd pacman-* cd pacman-* echo --info: pwd: `pwd` echo --info: setup pacman.. source setup.sh fi echo --info: mkdir -p, cd to ${ATLASINSTALLDIR} mkdir -p ${ATLASINSTALLDIR} cd ${ATLASINSTALLDIR} echo --info: pwd: `pwd` echo --info: start installing ATLAS: echo --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0 pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
With this script, the ATLAS Software 15.2.0 will get installed to /opt/atlas-local/15.2.0 while pacman resides at /opt/atlas-local/pacman. I use Indiana University mirror (am-IU). You perhaps want to modify this. Additionally, one could use :15.2.0+KV to automatically perform the Kit Validation after installation. This is a good idea for a new platform. But for CernVM 1.2.0 + group-atlas this KV always succeeds.
If you don’t like using this script at all, then just grab out the important commands. But please think twice about the path where to install ATLAS Software. Keep in mind that an ATLAS Software — once installed to a specific path — should always work under this path. By moving it around, you will get problems, even with new set up scripts. Furthermore, keeping everything necessary under /opt/atlas-local allows to e.g. move an EBS volume from one EC2 instance to another without losing important components and functionality.
Start ATLAS Software installation using pacman
Be sure to work as atlasuser. It’s just more clean to install the software not as root. Then, source the /opt/atlas-local/install_ATLAS-15-2-0.sh script from above.
[atlasuser@bla atlas-local]$ source install_atlas_15-2-0.sh --info: mkdir -p and cd to /opt/atlas-local/pacman --info: download latest pacman --07:58:25-- http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz [...] pacman-latest.tar.gz' saved [856237/856237] --info: extract.. --info: cd pacman-3.28 pacman-latest.tar.gz --info: pwd: /opt/atlas-local/pacman/pacman-3.28 --info: setup pacman.. --info: mkdir -p, cd to /opt/atlas-local/15.2.0 --info: pwd: /opt/atlas-local/15.2.0 --info: start installing ATLAS: --info: invoke: pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-IU:15.2.0
This will take a while (something around an hour for installing from am-IU to EC2 in US). Inbetween, you can log in with a second shell and proceed with the next important steps.
Make useful CVMFS content local
While working on the dirty solution, some problems came up. In particular, I had to hack around the CMTCONFIG problem. The CernVM collaborators have spent considerable amount of time to sort out these problems. So let’s profit from their work. I figured out that “taking over” their version of the Configuration Management Tool CMT and their cmthome directory is enough to get everything working properly.
Copy CernVM’s CMT from CVMFS to the local filesystem
Work as root. At first, copy the whole CMT directory from CVMFS (which has /opt/atlas/ as mountpoint) to /opt/atlas-local:
[root@bla ~]# cp -R /opt/atlas/software/sw/CMT /opt/atlas-local
Now CMT’s path changed, which means that it has to be re-installed. By doint this, /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh gets refreshed. This is important to set up CMT’s environment later on.
[root@bla atlas-local]# cd /opt/atlas-local/CMT/v1r20p20090520/mgr/
[root@bla mgr]# ./INSTALL
============================================
CMT installation terminated.
--------------------
cmt.exe is available on this site for:
Darwin-PowerMacintosh
Darwin-i386
Linux-i686
Linux-x86_64
VisualC
============================================CMT has successfully found its new location. Btw, let’s look at the currently existing directory structure of /opt/atlas-local:
[root@bla ~]# cd /opt/atlas-local [root@bla atlas-local]# ls 15.2.0 CMT install_atlas_15-2-0.sh pacman
It should look like that :-)
Create cmthome and modify the requirements file
The essential part of this step is to grab&modify CernVM’s requirements file for ATLAS Software releases. Work as atlasuser and copy the cmthome directory to your local ATLAS Software:
[atlasuser@bla atlas-local]$ cp -R /opt/atlas/software/15.2.0/15.2.0/cmthome /opt/atlas-local/15.2.0/
Modify the requirements file within that new cmthome directory so that SITEROOT points to the new local ATLAS Software:
[atlasuser@bla atlas-local]$ cd /opt/atlas-local/15.2.0/cmthome
[atlasuser@bla cmthome]$ ls
Makefile cleanup.csh cleanup.sh requirements setup.csh setup.sh
[atlasuser@bla cmthome]$ vi requirements
[... modify SITEROOT path ...]
[atlasuser@bla cmthome]$ cat requirements
#---------------------------------------------------------------------
set CMTSITE STANDALONE
set SITEROOT /opt/atlas-local/15.2.0
macro ATLAS_TEST_AREA ${HOME}/testarea
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag projectArea
macro SITE_PROJECT_AREA ${SITEROOT}
macro EXTERNAL_PROJECT_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag simpleTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)
set CMTCONFIG i686-slc4-gcc34-optIn the next step, this new requirements file will be parsed by CMT to create a setup script which will be necessary to set up the correct runtime environment for ATLAS Software.
Set up CMT environment, create the ATLAS Software setup script
Work as atlasuser. To set up the CMT environment, use the /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh, which was created while installing CMT.
[atlasuser@bla ~]$ source /opt/atlas-local/CMT/v1r20p20090520/mgr/setup.sh
Now, cmt and cmt.exe should be in the path. You can check it by typing cm *TAB*.
Switch to the cmthome directory and use cmt config to create the setup script for the ATLAS Software runtime environment:
[atlasuser@bla ~]$ cd /opt/atlas-local/15.2.0/cmthome [atlasuser@bla cmthome]$ cmt config sh: manpath: command not found ------------------------------------------ Configuring environment for standalone package. CMT version v1r20p20090520. System is Linux-i686 ------------------------------------------ Creating setup scripts. Creating cleanup scripts.
Interim report: installation is done!
Now we’re done. This new ATLAS Software installation is prepared for usage. Each of the previous steps is unique for this installation and does not have to be repeated. To use this ATLAS Software, only /opt/atlas-local/15.2.0/cmthome/setup.sh has to be invoked.
In case that /opt/atlas-local was the mountpoint to an EBS volume, this volume now contains a fully working ATLAS Software release. It may get attached to any VM running an operating system that supports ATLAS Software (Like CernVM + group-atlas, Scientific Linux 3/4, …) to start production runs with this specific installation.
Test the new local ATLAS Software: Run the Full Chain
To test & validate the new ATLAS Software installation, let’s run the Full Chain. An easy explanation of what this is and how to realize it with a shell script is given in this blog post: ATLAS Software: How to run The Full Chain
Create working directory, setup runtime environment
Work as atlasuser. Create a working directory. Many files will be produced while running the full chain.
[atlasuser@bla ~]$ mkdir atlasrun_15-2-0___1/ [atlasuser@bla ~]$ cd atlasrun_15-2-0___1
When the pacman installation from above has finished, you can set up the runtime environment for the new ATLAS Software installation.
This step always has to be executed before using the software after a new log in.
[atlasuser@bla atlasrun_15-2-0___1]$ source /opt/atlas-local/15.2.0/cmthome/setup.sh -tag=15.2.0 sh: manpath: command not found sh: manpath: command not found sh: manpath: command not found sh: manpath: command not found AtlasLogin: WARNING - test directory [/home/atlasuser/testarea/15.2.0] doesn't exist - the runtime environment won't reflect it
Don’t forget the -tag parameter!
The manpath problem isn’t an acutal problem. In this blog post you can read how to make manpath available. But it’s not needed. The warning about the test directory is correct, but for our full chain test we don’t need the testarea. If you need it for your production run, then create the stated directory.
Run the full chain!
I prepared a little script to run it. It’s explained in ATLAS Software: How to run The Full Chain
Download the script an run it:
[atlasuser@bla atlasrun_15-2-0___1]$ wget gehrcke.de/gsoc/athena_whole_chain_1event.sh [...] `athena_whole_chain_1event.sh' saved [1229/1229] [atlasuser@bla atlasrun_15-2-0___1]$ source athena_whole_chain_1event.sh [...] `singlepart_singlepi' saved [961/961] read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log real 0m42.774s user 0m29.074s sys 0m1.292s read EVGEN file.. simulate.. create HITS file..-> simul.log real 9m30.879s user 9m2.227s sys 0m4.936s read HITS file.. digitize.. produce RDO file..-> digi.log real 2m35.875s user 2m12.147s sys 0m5.116s read RDO file.. reconstruct.. produce ESD file..-> recoESD.log real 3m54.783s user 3m17.422s sys 0m6.427s read ESD file.. convert.. produce AOD file..-> recoAOD.log real 1m44.373s user 1m31.754s sys 0m3.687s
Success!
The full chain performed successfully with the local ATLAS Software installation. At the same time, we could have used the “remote ATLAS Software” via CVMFS. I verified this by
- logging in with an additional shell to the same machine
- creating another working directory
- setting up the runtime environment for the remote ATLAS Software 15.2.0 by sourcing
/opt/atlas/software/setupScripts/setupAtlasProduction_15.2.0.sh - running my full chain script
It resulted in
read CSC file, generate 1 single pion.. create EVGEN file..-> evgen.log real 6m13.287s user 0m31.648s sys 0m1.441s read EVGEN file.. simulate.. create HITS file..-> simul.log real 25m53.893s user 9m2.742s sys 0m5.593s read HITS file.. digitize.. produce RDO file..-> digi.log real 15m2.761s user 2m12.836s sys 0m6.455s read RDO file.. reconstruct.. produce ESD file..-> recoESD.log real 22m21.185s user 3m19.384s sys 0m6.891s read ESD file.. convert.. produce AOD file..-> recoAOD.log real 3m10.793s user 1m33.584s sys 0m4.454s
Pay attention to the long real timings. The reason is clear: The files had to be transferred via HTTP from a proxy server providing the content of CVMFS.
Leave a Reply