One of the main features of CernVM is its special filesystem CVMFS with http backend (based on FUSE). Using CernVM in the standard way, the different experiment softwares work out-of-the-box and are made accessible over the web via CVMFS. Although this is a great feature, I like to set up an ATLAS Software release locally — as real offline version — to be independent of the software-providing webservers.

Currently, the access to the providing server(s) must come from within CERN network. I am working with VMs in Chicago (Teraport, Nimbus cloud), don’t have a CERN account and doubt that the performance of CVMFS would have been good enough between Geneva and Chicago for my testing purposes. Hence, I tried to set up an ATLAS Software release locally, which required some special work.

Everything is based on CernVM 1.2.0 and ATLAS Software 15.1.0. The current solution is dirty and I am still looking for the optimal way. In the end, I would like to have an easy “switch” option between

• using CVMFS and being dependent of way software-providing webservers and
• keeping everything local

[Update]

The introduction/motivation is still valid and the parts below are still true. But in the meanwhile I found a much cleaner solution. It allows to use local and remote ATLAS Software at the same time and does not need any “hacking”. You can read about it in this blog post: CernVM: local ATLAS Software — the clean solution

[/Update]

To make room for the heavy ATLAS Software, I at first had to extend the filesystem within CernVM’s loopback file (I worked on the Xen-based Nimbus cloud, which did not support additional partitions at that time). This procedure is described in this blog post:
Resize ext3 file system in loopback file

Btw: While acting on CernVM’s image and filesystem locally with the intention to extend everything, I added the /root/.ssh folder to make public key insertion of Nimbus and EC2 possible. The directory does not exist by default. Predrag revealed that he will add it in the next release (should be 1.3.0).

Okay, imagine you are logged in as root to a booted fresh CernVM (only with the modifications named above: bigger filesystem, /root/.ssh added). If the Xen hypervisor added a correct kernel to the system, you can start right away by migrating the system to group-atlas:

$ conary migrate group-atlas --interactive

If this throws errors for you, consider this blog post: CernVM on Nimbus: kernel problems

After migration to group-atlas, the system is ready for a pacman installation of ATLAS Software. Set up pacman:

$ wget http://atlas.bu.edu/~youssef/pacman/sample_cache/tarballs/pacman-latest.tar.gz
$ tar xzf pacman-latest.tar.gz
$ cd pacman-3.28/
$ source setup.sh

In this case, the ATLAS Software release (+KitValidation) should get installed to /opt/atlas/15.1.0:

$ mkdir /opt/atlas/15.1.0
$ cd /opt/atlas/15.1.0
$ pacman -pretend-platform SLC-4 -allow trust-all-caches -get am-BU:15.1.0+KV

-pretend-platform SLC-4 is necessary, since we’re not on the standard ATLAS platform. Installing takes quite a while. To optimize the process, I chose Boston University mirror, which is good from the perspective of Chicago. In the end you should see

##################################################
##   AtlasProduction 15.1.0 Validation [  OK  ]
##################################################

Now the exciting part comes: setting up a working environment for the software.

Set up a cmthome directory and a requirements file:

$ mkdir /opt/atlas/15.1.0/cmthome
$ vi /opt/atlas/15.1.0/cmthome/requirements
$ cat /opt/atlas/15.1.0/cmthome/requirements
set CMTSITE STANDALONE
set SITEROOT /opt/atlas/15.1.0
macro ATLAS_DIST_AREA ${SITEROOT}
apply_tag opt
apply_tag setup
apply_tag noTest
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA)

This requirements file should be enough to use the release for local production runs (standalone and no testarea).

Now set up CMT, cd to the folder of the new requirements file and configurate CMT:

$ source /opt/atlas/15.1.0/CMT/v1r20p20081118/mgr/setup.sh
$ cd /opt/atlas/15.1.0/cmthome/
$ cmt config
sh: manpath: command not found
------------------------------------------
Configuring environment for standalone package.
CMT version v1r20p20081118.
System is Linux-i686
------------------------------------------
Creating setup scripts.
Creating cleanup scripts.

See the manpath issue? Not a real problem, but if you like to fix it for the future:

$ vi /bin/manpath
$ cat /bin/manpath
#!/bin/bash
man --path $*
$ chmod u+x /bin/manpath
$ conary update man
[...]
$ manpath
/usr/local/share/man:/usr/share/man:/usr/X11R6/man

Based on the requirements file, in the last step cmt config has created cmthome/setup.sh, which has to be sourced before each usage of the ATLAS Software to set up the environment correctly. This next step offers real issues with the platform:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
AtlasLogin: Configuration problem - CMTCONFIG (Unsupported-opt) not available for /opt/atlas/15.1.0/AtlasOffline/15.1.0
AtlasLogin: Error - Unsupported-opt installation non-existent and no fallback found
#CMT---> Warning: The tag Unsupported-opt is not used in any tag expression. Please check spelling

These warnings are not meaningless and the environment got only set up partly. The ATLAS Software release can almost not be used in this situation. The errors had to be fixed. So I started debugging.

Here is what happens basically within cmthome/setup.sh: At first CMT gets called in a special way to produce a temporary shellscript. This is executed directly after creation. At the beginning it sets a lot of environment variables (including CMTCONFIG) and then it executes several other configuration scripts. After execution, the temporary shellscript is deleted.

And here the error is:
CMT does not detect CernVM as a proper system for ATLAS Software. Hence, within this named temporary shellscript CMTCONFIG is set to Unsupported-opt or NotSupported. The following configuration scripts partly work and partly throw errors (the errors we saw above).

This is the solution:
In principle — after migration to group-atlas — CernVM should be very okay as platform. So I decided to override CMTCONFIG with the value i686-slc4-gcc34-opt in the temporary shellscript by a call to sed. This has to happen within cmthome/setup.sh, after creation and before execution. The value i686-slc4-gcc34-opt is totally correct for a 32bit CernVM after migration to group-atlas.

This is the modified cmthome/setup.sh:

# echo "Setting standalone package"
 
if test "${CMTROOT}" = ""; then
  CMTROOT=/opt/atlas/15.1.0/CMT/v1r20p20081118; export CMTROOT
fi
. ${CMTROOT}/mgr/setup.sh
 
tempfile=`${CMTROOT}/mgr/cmt -quiet build temporary_name`
if test ! $? = 0 ; then tempfile=/tmp/cmt.$$; fi
${CMTROOT}/mgr/cmt setup -sh -pack=cmt_standalone -path=/opt/atlas/15.1.0/cmthome  -no_cleanup $* >${tempfile}
echo "********************* CMTCONFIG hack start **********************"
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo ""
echo "** replacing with 'i686-slc4-gcc34-opt'"
sed -r 's/^CMTCONFIG=.+/export CMTCONFIG=i686-slc4-gcc34-opt/g' -i ${tempfile}
echo ""
echo "** appearances of CMTCONFIG in ${tempfile}:"
cat ${tempfile} | grep CMTCONFIG
echo "********** CMTCONFIG hack end: executing ${tempfile} **********"
. ${tempfile}
/bin/rm -f ${tempfile}

Now there were no more errors while setting up the environment:

$ source /opt/atlas/15.1.0/cmthome/setup.sh -tag=15.1.0
********************* CMTCONFIG hack start **********************
** appearances of CMTCONFIG in /tmp/fileMWrr85:
CMTCONFIG=NotSupported; export CMTCONFIG
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
CMTCONFIG="NotSupported"; export CMTCONFIG
 
** replacing with 'i686-slc4-gcc34-opt'
 
** appearances of CMTCONFIG in /tmp/fileMWrr85:
export CMTCONFIG=i686-slc4-gcc34-opt
NEWCMTCONFIG="i686-unknown00-gcc344"; export NEWCMTCONFIG
export CMTCONFIG=i686-slc4-gcc34-opt
********** CMTCONFIG hack end: executing /tmp/fileMWrr85 **********

With this environment I was able to run the full chain of JobTransforms.

The solution is not clean. For instance, the output of cmt show path is not complete. So, I personally don’t like it very much. For the future I plan to find a solution that combines CernVM’s configUpdate to ATLAS’ config (which belongs to CernVM’s bootstrap process) and a local pacman installation. This should be much more clean.