gehrcke.de » Python

Python WebSocket server powered by gevent and ws4py

Jan-Philip Gehrcke — Fri, 07 Oct 2011 15:56:21 +0000

I am following the emerging WebSocket standard with a lot of interest. Today, I would like to update my recommendation of tools presented in the article “The best and simplest tools to create a basic WebSocket application with Flash fallback and Python on the server side”. ws4py (WebSocket for Python) by Sylvain Hellegouarch is worth spreading the word.

ws4py is still under development and may underlie considerable structural changes in the future. But what we can see so far is a cleanly written generic pure-Python WebSocket protocol implementation that can be used for servers as well as for clients. I would like to point out that — among others — a gevent-driven server is already included. I have not seen or done any benchmarks about it, but theoretically this server should be very well performing. Overall, ws4py looks very promising.

In my playground setups, I’ve replaced gevent-websocket by ws4py. This way, I successfully ran some WebSocket communication tests between the clients implemented in Firefox 7, Chrome 14, Chrome 15 and the ws4py/gevent-based server — making use of the HyBI WebSocket protocol version 10. Together with a recent version of the perfect web-socket-js, the communication also works with older browsers like Firefox 3.

Standard conformance and performance of ws4py is tested via the Autobahn WebSocket test suite. To have a testsuite like that is extremely important in order to push the development of high-quality WebSocket libraries. You should definitely look at it if you are a WebSocket client/server developer yourself.

Another thing I would like to show to you is the table at http://en.wikipedia.org/wiki/User:Oberstet/Comparison_of_WebSocket_implementations. If maintained properly, this will always be a very good reference to get an overview about WebSocket implementations on different platforms.

The best and simplest tools to create a basic WebSocket application with Flash fallback and Python on the server side

Jan-Philip Gehrcke — Sun, 26 Jun 2011 15:56:46 +0000

Currently, I am playing around with WebSockets. This is due to an application idea I have in my mind which requires a bidirectional connection between browser and server with low latency. The communication will happen in a stream-like fashion at low bandwidth. Real network sockets using TCP/UDP are often the desired optimum for things like that, but within a browser they can only be provided by Java or Flash plugins. The future belongs to WebSockets. Implemented directly in the browser they are providing a much lower level network connection between the browser and the server than HTTP — without any plugin. WebSockets still work on a layer above TCP, but low latency and efficient data transport in both directions is warranted in a TCP-like fashion. Therefore, real-time application developers are very keen to use WebSockets. As this still is a young development, there is only few browser support and a lot of non-mature client/server libraries. Most importantly, there is a huge lack of documentation how to use these.

In this blog post, I present a very simple “echo application” where the user sends a message from his browser to the server — which in turn sends this message back to the client. Simple so far, but the main focus while realizing this is on selecting the right tools for the task, as there is already a lot of stuff out there — some very good, undocumented and hidden things, some totally overloaded things, and some bad things. I tried to fulfill the following conditions:

Make use of a “Flashbridge” to realize a fallback when WebSockets are not available in a browser (which is true for Firefox at the moment)
use Python on the server side
use the best / most solid tools available
at the same time, use the simplest tools available that do not bring along loads of stuff that you do not need for simple applications or if you want to desing your own communication protocol anyway.

Tools for the client side

First thing, the client running in the browser has to be implemented in JavaScript, of course. I do not like JavaScript, but this is the language of the browsers (is there a chance to revolutionize this to Python syntax?). So, when you search the web you will at first find Socket.IO and jWebSocket. Socket.IO is consisting of the client “Socket.IO” and the server “Socket.IO-node” — designed to be used together. Yes, it is great — it seems mature, it already provides a lot of different protocols on top of WebSockets and it provides loads of fallback mechanisms if there are no WebSockets available on the client side. There even is a Python implementation of the server side. But my feeling was: Nope. I have no experience so far with WebSockets, so I would like to start implementing a basic core. I don’t want to use any communication protocol so far. Just sending messages as designed by the specification of WebSockets. Furthermore, I only want Flash fallback as an equivalently fast replacement, no slow backup techniques like xhr-multipart, xhr-polling, and jsonp-polling. Socket.IO overwhelms with a lot of functionality and few of documentation — making it difficult for newcomers to grasp the essentials.

A lot of above also applies for jWebSocket, so they have many things in common why I did not want to use them. And as it turned out while browsing through their sources, they have one very special thing in common:

web-socket-js

Socket.IO as well as jWebSocket both base their core WebSocket-Flash-switch on a very neat and small JavaScript library called web-socket-js written by Hiroshi Ichikawa. You can check this yourself by looking at Socket.IO’s repository and jWebSocket’s repository. If required, web-socket-js replaces the standard JavaScript new WebSocket constructor by its own implementation. This does the trick:

(function() {
 
  if (window.WebSocket) return;
 [...]
}

If WebSockets are supported by the browser natively, then they are used natively and web-socket-js does not do anything more than return;. If the browser lacks support, a Flash plugin Hiroshi wrote is loaded, providing the same API as the original JavaScript WebSocket. This API is implemented in the [...] of the code snippet above.

This is the way I want to start learning WebSockets: using the original API and with a lot of browser support. Decision made: on the client side, I use web-socket-js.

Tools for the server side

Answer upfront this time: gevent in combination with gevent-websocket. Gevent is a nicely performing networking library for Python, based on the brilliant idea of greenlets. It allows you to write extremely simple server code. Gevent-websocket is just a very small piece of code on top of this, basically saving you from implementing the WebSocket handshake yourself.

The code

Let’s start with websocket_echoserver.py:

from geventwebsocket.handler import WebSocketHandler
from gevent import pywsgi
import gevent
 
 
def handle_echo_client(ws):
    while True:
        msg = ws.wait()
        if msg == "quit":
            ws.close_connection()
            break
        ws.send(msg)
 
 
def app(environ, start_response):
    if environ['PATH_INFO'] == "/echo":
        handle_echo_client(environ['wsgi.websocket'])
    else:
        print "404, PATH_INFO: %s" %  environ["PATH_INFO"]
        start_response("404 Not Found", [])
        return []
 
 
server = pywsgi.WSGIServer(('', 8087), app, handler_class=WebSocketHandler)
server.serve_forever()

It listens on localhost:8087 for incoming connections. WebSocket connections are accepted and handled. If the requested resource is /echo, the greenlet handle_echo_client() deals with the client. I think it is obvious what this function does. To reach this server, the client has to connect to ws://yourhost:8087/echo.

Now the client side. Call this HTML code in your browser (make sure to set the paths to the JavaScript sources and SWF file correctly and keep everything under the same domain):


"http://www.w3.org/TR/html4/transitional.dtd">
<html>
<head>
<title>websocket client: send/receive test</title>
<script type="text/javascript" src=".........../web-socket-js/swfobject.js"></script>
<script type="text/javascript" src=".........../web-socket-js/web_socket.js"></script>
<script type="text/javascript">
WEB_SOCKET_SWF_LOCATION = "............/web-socket-js/WebSocketMain.swf";
function log(msg) {
    var d = new Date();
    var hour = d.getHours();
    var min = d.getMinutes();
    var sec = d.getSeconds();
    var msec = d.getMilliseconds();
    var time = hour + ":" + min + ":" + sec + ":" + msec
    document.getElementById('foo').innerHTML += time + ": " + msg + "<br />";
    }
 
function ws_init(url) {
    log("connecting to " + url + "...")
    ws = new WebSocket(url);
    ws.onopen = function() {
        log("connection established.");
        };
    ws.onmessage = function(e) {
        log("message received: '" + e.data + "'");
        };
    ws.onclose = function() {
        log("connection closed.");
        };
    }
 
function ws_send(msg) {
    log("send: " + msg);
    ws.send(msg);
    }
 
function ws_close() {
    log("closing connection..");
    ws.close();
    }
</script>
</head>
<body>
<form name="input_form">
<table>
<tr>
    <td>host/resource:</td>
    <td><input type="text" name="host" size="50" value="ws://host:port/resource"></td>
    <td><input type="button" value="connect" onclick="ws_init(document.input_form.host.value)"></td>
    <td><input type="button" value="disconnect" onclick="ws_close()"></td>
</tr>
<tr>
    <td>msg:</td>
    <td><input type="text" name="msg" size="50"></td>
    <td><input type="button" value="send" onclick="ws_send(document.input_form.msg.value)"></td>
</tr>
</table>
</form>
<div id="foo"> </div>
</body>
</html>

What is used here is the standard JavaScript WebSocket API. For me, this HTML renders as:

Now… maybe you opened this in Chrome. Then, WebSocket support is natively included and activated. In Firefox, it is disabled. There the Flashbridge would come into play. But Flash does not just send and receive data to and from anywhere. It at first asks the host who delivered the SWF file with whom it is allowed to exchange data: it expects to get a so-called socket policy file from the server, after asking him for this on port 843. Yeah, just accept this — it is not a bad idea that the internet communication of Flash plugins is restricted. Anyway, you have to set up a server listening on port 843 to provide this policy. I have made a blog post for setting up a Flash socket policy server in Python based on gevent. I just assume that you have set up such a server. Then, our HTML code from above works (at least for me) under Firefox 4+, Chrome 6+ and Internet Explorer 9. Enter your echo server’s details (ws://yourhost:8087/echo) and play around with it. For me, it looks like that:

Isn’t that great? It works. The latency times between send and receive are pretty low (at least for such small words to send): they are comparable with the ping latencies I measured against the same server. The Flashbridge transparantly does its job in Firefox and Internet Explorer and introduces around 5 ms more latency.

Conclusion

The tools used for this test (gevent, gevent-websocket and web-socket-js) are solid, perform very good and are very simple to use. The code executed in the browser uses the standard WebSocket API. But, due to web-socket-js and its nice Flashbridge, it works in all browsers supporting Flash. gevent-websocket and web-socket-js do not provide more functionality than simple message passing over WebSockets, which was desired.

Flash socket policy server in Python based on gevent

Jan-Philip Gehrcke — Sat, 18 Jun 2011 16:25:30 +0000

Currently, I am looking into gevent — a nicely performing networking library for Python, based on the brilliant idea of greenlets. I try to use this for WebSocket communication with browsers. As of today, WebSockets are still not available or disabled in some browsers. That’s why there are Javascript implementations like web-socket-js providing a transparent “Flashbridge” as a fallback for the WebSocket communication. But the Flash plugin in the client’s browser does not simply communicate with any server on any port in the world. As a first step, it always tries to receive a so-called socket policy file from the same server the SWF file was received from; on port 843. This policy file tells the client from whom he is allowed to receive data. Today, I present a simple server based on gevent providing this policy file for connecting Flash clients.

By using gevent, you automatically get one of the highest performing Python servers you can get for this purpose. Due to the greenlet concept, it is able to handle a huge amount of concurrent connections without any threading overhead. Due to benchmarks like this, the server should be faster than comparable Python solutions (based on e.g. Twisted or Eventlet) while almost reaching the level of optimized C/C++ solutions. Of course, all this performance is not really necessary for a Flash policy server, but it is good to know what we are dealing with.

Now, the code. It is already “bloated” with comments and logging functionality. Due to the use of gevent, the server code itself is very simple:

#!/usr/bin/env python
 
# Flash access policy server based on gevent
# Jan-Philip Gehrcke, June 2011
 
# Listen on port 843; send acess policy to client; disconnect.
 
 
from gevent.server import StreamServer
import datetime
import socket
 
 
# Should we log something? Where to?
LOG = 1
LOGFILE = "flash_access_policy_server.log"
 
# The policy that is sent to the clients.
POLICY = """\0"""
 
# The string the client has to send in order to receive the policy.
POLICYREQUEST = ""
 
 
# This function is called for each incoming connection
# (in a non-blocking fashion in a greenlet)
def client_handle(sock, address):
    log("%s:%s: Connection accepted." % address)
    # send and read functions should not wait longer than three seconds
    sock.settimeout(3)
    try:
        # try to receive at most 128 bytes (`POLICYREQUEST` is shorter)
        input = sock.recv(128)
        if input.startswith(POLICYREQUEST):
            sock.sendall(POLICY)
            log("%s:%s: Policy sent. Closing connection." % address)
        else:
            log("%s:%s: Crap received. Closing connection." % address)
    except socket.timeout:
        log("%s:%s: Timed out. Closing." % address)
    sock.close()
 
 
# Write `msg` to file and stdout, prepended by a date/time string
def log(msg):
    if LOG:
        l = "%s: %s" % (datetime.datetime.now().isoformat(), msg)
        lf.write("%s\n" % l)
        print l
 
 
if __name__ == '__main__':
    if LOG:
        lf = open(LOGFILE, "a")
    server = StreamServer(('0.0.0.0', 843), client_handle)
    log('Starting server...')
    server.serve_forever()

Note that the policy in this case is not very sophisticated and basically allows everything;-)

Average date strings using Python

Jan-Philip Gehrcke — Sun, 06 Feb 2011 19:31:23 +0000

Assume, you’ve measured values over time and now you want to average your data. This means you have to a) average your measured values — which is a trivial task — and b) average your points in time. Here, I present a solution how to average arbitrary date strings in Python.

Let’s talk about a specific example. This is the content of dates.dat, our input:

20110130_195243
20110130_200003
20110130_200803
20110130_200909
20110130_201003
20110130_202004
20110130_203003
20110130_204003
20110130_205003
20110130_210003
20110130_211003
20110130_212004
20110130_213003
20110130_214003
20110130_215003
20110130_220003
20110130_221003

Each of the 17 lines contains a string representing a point in time using some distinct format.

Now, let’s say that the goal is to build the mean of every 3 points in time. An output date string representing a mean time should have the same format as the date strings in dates.dat. Hence, the outputfile dates_meanof3values.dat should look like this:

20110130_200016
20110130_201305
20110130_204003
20110130_211003
20110130_214003

These 5 date strings represent the average points in time of the first 5*3 date strings in our input.

The following Python code accomplishes this:

```
import time
```
```
 
```

meanof = 3 # number of dates taken into account for averaging

```
inputfile = open('dates.dat')
```

outputfile = open("dates_meanof%svalues.dat" % meanof,'w')

```
 
```
```
def datestring_to_timestamp(str):
```
```
    """
```

    Assume `str` representing a time in local time and convert it to a timestamp

    (time as a floating point number expressed in seconds since the epoch, in

```
    UTC) using the format given below.
```
```
    """
```

    return time.mktime(time.strptime(str, "%Y%m%d_%H%M%S"))

```
 
```
```
def timestamp_to_datestring(timestamp):
```
```
    """
```

    Inverse of the function `datestring_to_timestamp`

```
    """
```

    return time.strftime("%Y%m%d_%H%M%S", time.localtime(timestamp))

```
 
```
```
def chunks(list, n, strict=False):
```
```
    """
```

    Split `list` in sub-lists: yield successive `n`-sized chunks from `list`.

    If `strict` is True, the last chunk is only yielded if its length is `n`.

```
    """
```
```
    for i in xrange(0, len(list), n):
```

        if not strict or len(list[i:i+n]) == n:

```
            yield list[i:i+n]
```
```
 
```

# read lines from file and remove trailing spaces; don't consider empty lines

cleanlines = [line.strip() for line in inputfile.readlines() if line.strip()]

# devide data into chunks and interate over them

for chunk in chunks(cleanlines, meanof, True):

    # build timestamp of each datestring in current chunk and build the median

    mean_timestamp = sum(map(datestring_to_timestamp, chunk)) / meanof

    # convert mean timestamp back to datestring and write this to file

    outputfile.write("%s\n" % timestamp_to_datestring(mean_timestamp))

Timestamps are linearly related in the decimal system, so that time can be easily averaged by summation and division of timestamps. The functions datestring_to_timestamp() and timestamp_to_datestring() perform the conversion of date strings from/to timestamps, using a user-given date string format (you can edit lines 13 and 19 corresponding to these format specifiers).

The function chunks(), which makes use of Python generators, devides a given list into sub-lists (“chunks”). This is very useful at this point — the mean date string of a chunk then is calculated as simple as

timestamp_to_datestring(sum(map(datestring_to_timestamp, chunk)) / meanof)

MySQL backup script with email support and lzma compression (for cron)

Jan-Philip Gehrcke — Tue, 16 Nov 2010 23:23:45 +0000

Let me share a shell script to perform a backup of a MySQL database. It is intended to run regularly (e.g. as cronjob). I created it carefully. Finally, it may distinguish itself from similar scripts by the following points:

It stores the backup file locally. Additionally, it sends an email with the backup file via any SMTP server you like (localhost:25 by default). STARTTLS/SSL is supported. Therefore, I use mailsend.py (an SMTP-based email sender I’ve written in Python). If mysqldump does not exit with return code 0, a special email is sent. mysqldump’s stderr output is always fetched and sent via mail.

It does not use your database password directly in the mysqldump command (in which case other users would be able to see the your password in the process list while the command is running). This mistake seems to be done often.

It uses lzma compression, which produces higher compression ratios than gzip/bz2 (at the cost of computing power).

It automatically deletes local backup files older than a user-given number of days.

The script:

```
#!/bin/sh
```
```
 
```

# database backup settings. edit these accordingly.

```
EMAIL="user@host.com"
```
```
BACKUP_NAME="FORUM_DB_BACKUP"
```
```
BACKUP_DIR="/path/to/backup/dir"
```
```
DB_NAME="dbname"
```
```
DB_HOST="127.0.0.1"
```
```
DB_USER="username"
```

DB_PASSWORDFILE="/path_to/.db_password_file"

```
DEL_OLDER_THAN_X_DAYS="30"
```
```
 
```
```
if [[ ! -d ${BACKUP_DIR} ]]; then
```
```
    mkdir ${BACKUP_DIR}
```
```
fi
```

BACKUP_PATH=${BACKUP_DIR}/$(date +%Y%m%d-%H%M%S)_${BACKUP_NAME}.sql.lzma

```
 
```

# dump database content to stdout. read db password from file (never put it

# on the commandline directly, as other users may read it in the process

# list). forward both stdout and stderr to lzma, which is more efficient

# than gzip/bz2. store the data in a file with a meaningful name.

```
mysqldump \
```
```
    --user=${DB_USER} \
```

    --password=$(cat ${DB_PASSWORDFILE}) \

```
    --host=${DB_HOST} \
```
```
    --opt --quote-names \
```
```
    ${DB_NAME} \
```
```
    |& lzma -c > \
```
```
    ${BACKUP_PATH}
```
```
 
```

# send the backup file to your email address using the SMTP server listening

# on localhost:25. if you wish to use another SMTP server (localhost:465 or

# smtp.gmail.com:587 or whatever), see mailsend.py --help.

# note: PIPESTATUS may not work within other shells than bash.

```
if [[ ${PIPESTATUS[0]} -eq 0 ]]; then
```

    python mailsend.py -s ${BACKUP_NAME} -f ${BACKUP_PATH} ${EMAIL}

```
else
```

    python mailsend.py -s ${BACKUP_NAME}_ERROR -f ${BACKUP_PATH} \

    -m "mysqldump threw an error. It's stderr is within the attached archive." \

```
    ${EMAIL}
```
```
fi
```
```
 
```

# delete backup files older than X days (as set above)

```
find \
```
```
    ${BACKUP_DIR} \
```
```
    -name \*.sql.lzma \
```
```
    -mtime +${DEL_OLDER_THAN_X_DAYS} \
```
```
    -exec rm -f '{}' \;
```

How to use it:

Put the lines above in a file on your server and make it executable.
Create the password file containing your database password in clear text (apply chmod 600 to the file!).
Adjust the settings within the header of the script.
If you like the email feature, store mailsend.py (download link below) in the same directory as the backup script (or adjust the paths). If you don’t like the email feature, outcomment the corresponding lines 35 to 41.
If your server does not know the lzma command, replace it with gzip (line 28).
Configure your new backup script as cronjob.

Mailsend.py:

I will publish a blog post introducing mailsend.py soon. This is what I can provide until then:
View the help: here
View the code: here

Download it:
mailsend_2010_11_16.zip
mailsend_2010_11_16.tar.gz

Python backup/snapshot script with compression (BZ2, 7zip)

Jan-Philip Gehrcke — Sat, 13 Feb 2010 18:18:49 +0000

While writing my master thesis with LaTeX, I often change source code files, graphics etc. It is important for me to keep track of the major changes as well as I have to prevent losing the whole work due to some data loss issue. To solve these two problems, I wrote a convenient Python backup script.

On invocation, it creates a snapshot of my thesis directorie’s content. It writes everything to another directory, with a timestamp in its name. With these snapshots, I can simply look into older versions of my work later on, just to be prepared if worst comes to worst.

But… what if the hard drive with the snapshots on it crashes? Therefore, I added the option to simply copy the whole directory tree to an additional destination as well. If this is another physical storage, it’s pretty unlikely to lose both snapshot copies at the same time. Feels good.

Over the time, I created more and more files within my project. For convenience, I added an option to compress the directorie’s snapshot into an archive. At this point, the user can choose between Python’s built-in .tar.bz2 compression method or let the script use 7zip, which provides ultra strong compression and is very fast.

After only a few seconds of configuration (directory definition, backup method selection), you only have to doubleclick the script file everytime you want to feel safe

For Windows users:
0) Download and install Python 2.6.x: http://python.org/download/
1) Copy the script (below), edit “SETTINGS” section of the script, save it within a .py file
2) Doubleclick as often as you want.

```
#!/usr/bin/env python
```

#********************************* README **************************************

```
#
```

# SNAPSHOT / BACKUP script for Windows/Linux

# script by Jan-Philip Gehrcke -- jgehrcke@gmail.com -- http://gehrcke.de

```
#
```
```
# 0) DESCRIPTION:
```
```
# ===============
```

# On invocation, the script creates a snapshot of ORIG_DIR's contents and writes

# it to BACKUP_DIR into 1) a new subdirectory or 2) a .tar.bz2 archive or 3) a

# 7zip archive (choose it!). The time of snapshot creation is written into the

# subdirectorie's name / archive file name. An optional second location can

# be defined to which the snapshot will be written additionally.

```
#
```

# This script is useful to manually and quickly create snapshots of a multi-file

# project you're working on, enabling _rollbacks_ to an older version of your

# project's files. Furthermore, using the additional backup location on another

# physical storage, the script prevents _data loss_.

```
#
```

# Primarily, this script is written for Windows users: simple double-click .py

# file invocation is considered. Should work on Linux systems, too, but the

# "press any key to continue" dialogue is quite un-unixoid.

```
#
```
```
# 1) USAGE:
```
```
# =========
```
```
#
```

# Download and install Python 2.6.x: http://python.org/download/

# For 7zip method, download http://www.7-zip.org/download.html

```
#
```

# Put the script file into the directory containing the directory you want to

# back up, adjust settings (below) and then run the script (doubleclick on Win).

```
#
```
```
# The snapshot/backup of
```
```
#  ./ORIG_DIR/*
```
```
# will go to
```

#  ./BACKUP_DIR/BACKUP_PREFIX_timestring/*       (SIMPLE method, built-in)

```
# OR to the archive
```

#  ./BACKUP_DIR/BACKUP_PREFIX_timestring.tar.bz2 (BZ2 method, built-in)

```
# OR to the archive
```

#  ./BACKUP_DIR/BACKUP_PREFIX_timestring.7z      (7zip method; ultra strong

#                                                 compression; faster than BZ2;

#                                                 requires 7zip to be available)

```
#
```

# Of course, ORIG_DIR and BACKUP_DIR can be absolute paths, too. Then, the

# location of this script does not matter.

```
#
```
```
# 2) SETTINGS:
```
```
# ============
```

# always use SLASHES ("/") in paths, even on Windows -> don't use "\"

ORIG_DIR = "C:/project"          # e.g. "." or "C:/project"

BACKUP_DIR = "C:/backups"        # e.g. "C:/backups"

BACKUP_PREFIX = "thesis_bckp"      # e.g. "thesis_bckp"

```
 
```

# choose backup method: 'SIMPLE' OR 'BZ2' OR '7zip':

METHOD = '7zip'     # quick and really strong compression, 7zip.exe required

#METHOD = 'SIMPLE'  # copy directory tree; e.g. if you don't have many files..

#METHOD = 'BZ2'     # builtin method; if you like compression, but no 7zip.

```
 
```

# in case of 7zip, specify 7z executable path:

SEVENZIPPATH = "c:/Programs/7-Zip/7z.exe" # e.g. "c:/Programs/7-Zip/7z.exe"

```
 
```

# set ADDITIONAL_BACKUP_DIR to double-save backup (e.g. on another hard disk)

# (outcomment the line if this is undesired behavior)

ADDITIONAL_BACKUP_DIR = "F:/backups"   # e.g. "F:/backups"

#*******************************************************************************

```
 
```

import os, time, shutil, sys, tarfile, subprocess, traceback

```
 
```

def backup_directory_simple(srcdir,dstdir):

```
    if os.path.exists(dstdir):
```

        exit_stop("backup path %s already exists!" % dstdir)

```
    try:
```
```
        shutil.copytree(srcdir,dstdir)
```
```
    except:
```

        print "Error while copying tree in %s to %s" % (srcdir,dstdir)

        print "Traceback:\n%s"%traceback.format_exc()

```
        return False
```
```
    return dstdir
```
```
 
```

def backup_directory_bz2(srcdir,tarpath):

```
    if os.path.exists(tarpath):
```

        exit_stop("backup path %s already exists!" % tarpath)

```
    try:
```

        tar = tarfile.open(tarpath, "w:bz2")

        for filedir in os.listdir(srcdir):

           tar.add(os.path.join(srcdir,filedir),arcname=filedir)

```
        tar.close()
```
```
    except:
```

        print "Error while creating tar archive: %s" % tarpath

        print "Traceback:\n%s"%traceback.format_exc()

```
        return False
```
```
    return tarpath
```
```
 
```

def backup_directory_7zip(srcdir,arcpath):

```
    if os.path.exists(arcpath):
```

        exit_stop("backup path %s already exists!" % arcpath)

```
    try:
```

        # -mx9 means maximum compression

        arglist = [SEVENZIPPATH,"a",arcpath,"*","-r","-mx9"]

        print ("try running cmd:\n %s\nin directory\n %s" %

```
            (' '.join(arglist),srcdir))
```

        # run 7zip (in the directory to be backupped!)

```
        sp = subprocess.Popen(
```
```
            args=arglist,
```
```
            stdout=subprocess.PIPE,
```
```
            stderr=subprocess.PIPE,
```
```
            cwd=srcdir)
```
```
    except:
```

        print "Error while running 7zip subprocess. Traceback:"

        print "Traceback:\n%s"%traceback.format_exc()

```
        return False
```

    # wait for process to terminate, get stdout and stderr

```
    stdout, stderr = sp.communicate()
```
```
    if stdout:
```

        print ("\n>>> 7zip subprocess STDOUT START:\n%s"

                ">>> 7zip subprocess STDOUT END\n" % stdout)

```
    if stderr:
```

        print "7zip STDERR:\n%s" % stderr

```
        return False
```
```
    return arcpath
```
```
 
```
```
def any_key():
```
```
    print "Press any key to continue."
```
```
    getch()
```
```
 
```
```
def exit_stop(exitstring):
```
```
    print exitstring
```
```
    any_key()
```
```
    sys.exit(exitstring)
```
```
 
```
```
def so_flushwr(string):
```
```
    sys.stdout.write(string)
```
```
    sys.stdout.flush()
```
```
 
```
```
# provide getch() method
```

# (http://stackoverflow.com/questions/1394956/how-to-do-hit-any-key-in-python

```
try:
```
```
    # Win32
```
```
    from msvcrt import getch
```
```
except ImportError:
```
```
    # UNIX
```
```
    def getch():
```
```
        import sys, tty, termios
```
```
        fd = sys.stdin.fileno()
```
```
        old = termios.tcgetattr(fd)
```
```
        try:
```
```
            tty.setraw(fd)
```
```
            return sys.stdin.read(1)
```
```
        finally:
```

            termios.tcsetattr(fd, termios.TCSADRAIN, old)

```
 
```

# build timestring, check settings and invoke corresponding backup function

print "*********************************************************************"

print "* snapshot backup script by Jan-Philip Gehrcke -- http://gehrcke.de *"

print "*********************************************************************\n"

```
 
```

timestr = time.strftime("_%y%m%d_%H%M%S",time.localtime())

if METHOD not in ["SIMPLE", "BZ2", "7zip"]:

    exit_stop("METHOD not 'SIMPLE' OR 'BZ2' OR '7zip'")

```
if not os.path.exists(ORIG_DIR):
```

    exit_stop("ORIG_DIR does not exist: %s" % os.path.abspath(ORIG_DIR))

```
if not os.path.exists(BACKUP_DIR):
```

    exit_stop("BACKUP_DIR does not exist: %s" % os.path.abspath(BACKUP_DIR))

```
else:
```

    print ("write snapshot of\n  %s\nto\n  %s\nusing the %s method...\n" %

            (os.path.abspath(ORIG_DIR),os.path.abspath(BACKUP_DIR),METHOD))

```
    if METHOD == "SIMPLE":
```

        rv = backup_directory_simple(srcdir=ORIG_DIR,

            dstdir=os.path.join(BACKUP_DIR, BACKUP_PREFIX + timestr))

```
    elif METHOD == "BZ2":
```

        rv = backup_directory_bz2(srcdir=ORIG_DIR,

            tarpath=os.path.join(BACKUP_DIR,

                BACKUP_PREFIX + timestr + ".tar.bz2"))

```
    else:
```
```
        try:
```

            if not os.path.exists(SEVENZIPPATH):

                exit_stop("7zip executable not found: %s" % SEVENZIPPATH)

```
        except NameError:
```

            exit_stop("variable SEVENZIPPATH not defined")

        rv = backup_directory_7zip(srcdir=os.path.abspath(ORIG_DIR),

            arcpath=os.path.abspath(os.path.join(BACKUP_DIR,

                BACKUP_PREFIX + timestr + ".7z")))

```
 
```
```
if rv:
```

    print "Snapshot successfully written to\n  %s" % os.path.abspath(rv)

```
else:
```
```
    print "Failure during backup :-("
```
```
 
```

if 'ADDITIONAL_BACKUP_DIR' in globals() and rv:

    if not os.path.exists(ADDITIONAL_BACKUP_DIR):

        exit_stop(("ADDITIONAL_BACKUP_DIR does not exist: %s"

            % os.path.abspath(ADDITIONAL_BACKUP_DIR)))

    so_flushwr("\nwrite additional backup to %s.." % ADDITIONAL_BACKUP_DIR)

```
    try:
```

        dst = os.path.join(ADDITIONAL_BACKUP_DIR,os.path.basename(rv))

```
        if os.path.isdir(rv):
```

            shutil.copytree(rv,dst) # simple method, copy directory tree

```
        else:
```

            shutil.copy(rv,dst) # copy 7zip or bz2 archive

```
        so_flushwr("success\n")
```
```
    except:
```

        print "Traceback:\n%s"%traceback.format_exc()

        print "Additional backup not written. For diagnosis look above."

```
 
```
```
any_key()
```

Just for fun: binary to ascii in Python

Jan-Philip Gehrcke — Tue, 26 Jan 2010 17:08:57 +0000

0110100001110100011101000111000000111010001011110010111101111000011010110110001101100100001011100110001101101111011011010010111100110101001100010011100000101111

Seriously, you really want to know what’s behind this!

BuHa.info’s New Year’s Challenge gave me the feeling that I quickly have to write a Python script, decoding ones and zeros to text, using ASCII code

```
import sys
```
```
 
```
```
# define file pointers and bytelength
```
```
file_in = open("bin2txt_in.txt")
```
```
file_out = open("bin2txt_out.txt","wb")
```
```
bl = 8
```
```
 
```
```
# read input file; remove whitespaces
```
```
# make list of bits (ints) from input
```

bitlist = map(int,''.join(file_in.read().split()))

```
if len(bitlist)%bl != 0:
```

    sys.exit("length of bitlist not integer multiple of %s" % bl)

```
 
```
```
# assemble resultstring:
```
```
# - make list of bytes from `bitlist`
```

# - evaluate each byte -> int -> ascii char

rs = ''.join([chr(sum(bit<<abs(idx-bl)-1 for idx,bit in enumerate(y)))

               for y in zip(*[bitlist[x::bl] for x in range(bl)])

```
              ])
```
```
 
```
```
# write output
```
```
file_out.write(rs)
```
```
file_out.close()
```

Put the data above in bin2txt_in.txt, run the script(*) in the same directory and read bin2txt_out.txt’s content. You won’t regret.

(*)For Windows users not knowing how to run the script:
- Download Python & install it.
- Copy the script above into a simple text file with the extension .py and doubleclick to run.

And, yeah, you’re right, this is senseless (except for the programming exercise). And.. true, you can decode this using online converters like this one here.

Convert all PDF files in a directory to PNG images

Jan-Philip Gehrcke — Thu, 29 Oct 2009 17:40:45 +0000

I just needed to convert several PDF graphics to PNG raster graphics. The cleanest way to do this is via Ghostscript (example: gs -sDEVICE=png16m -sOutputFile=tiger.png tiger.pdf). For convenience I made a Python script that converts all PDF files in the working directory to PNG files via Ghostscript. I use it under Windows, but this works for “all” operating systems. Let me share it with you.

Python script:

```
import subprocess
```
```
import os
```
```
import traceback
```
```
 
```

GHOSTSCRIPTPATH = "C:/Program Files (x86)/gs/gs8.70/bin/gswin32c.exe"

```
 
```
```
def main():
```
```
    cwd = os.getcwd()
```
```
    for filename in os.listdir(cwd):
```

        (name, ext) = os.path.splitext(filename)

```
        if ext.lower().endswith("pdf"):
```

            print "*** FOUND %s" % filename

            convert_pdf_file_to_png(GHOSTSCRIPTPATH,

                os.path.join(cwd, filename))

```
 
```

def convert_pdf_file_to_png(ghostscriptpath, pdffilepath):

```
    if not os.path.isfile(pdffilepath):
```

        print "%s is not a file" % pdffilepath

```
        return False
```

    if not os.path.isfile(ghostscriptpath):

        print "%s is not a file" % ghostscriptpath

```
        return False
```
```
 
```

    pdffiledir = os.path.dirname(pdffilepath)

    pdffilename = os.path.basename(pdffilepath)

    pdfname, ext = os.path.splitext(pdffilepath)

```
 
```
```
    try:    
```

        # change the "-rXXX" option to set the PNG's resolution.

        # -> http://ghostscript.com/doc/current/Devices.htm#File_formats

        # for other commandline options see

        # http://ghostscript.com/doc/current/Use.htm#Options

```
        arglist = [ghostscriptpath,
```
```
                  "-dBATCH",
```
```
                  "-dNOPAUSE",
```

                  "-sOutputFile=%s.png" % pdfname,

```
                  "-sDEVICE=png16m",
```
```
                  "-r800",
```
```
                  pdffilepath]
```
```
 
```

        print "try running cmd:\n%s" % arglist

```
 
```
```
        sp = subprocess.Popen(
```
```
            args=arglist,
```
```
            stdout=subprocess.PIPE,
```
```
            stderr=subprocess.PIPE)
```
```
    except:
```

        print "Error while running Ghostscript subprocess. Traceback:"

        print "Traceback:\n%s"%traceback.format_exc()

```
        return False
```

    # wait for process to terminate, get stdout and stderr

```
    stdout, stderr = sp.communicate()
```

    print "Ghostscript subprocess STDOUT:\n%s" % stdout

```
    if stderr:
```

        print "Ghostscript STDERR:\n%s" % stderr

```
        return False
```
```
    return True
```
```
 
```
```
 
```
```
if __name__ == "__main__":
```
```
    main()
```

Usage:

Download & Install Python 2.6.x: http://python.org/download/ (should work with Python 2.4 and above, but not with Python 3)
Download & Install Ghostscript: http://mirror.cs.wisc.edu/pub/mirrors/ghost/GPL/current/
Copy the script source from above, save it as e.g. make_all_pdfs_to_png.pyw
adjust the variable GHOSTSCRIPTPATH in line 5 of the script to the location of your Ghostscript executable. Even for Windows, it’s okay to use normal slashes in the path (or escaped backslashes: \\).
If needed, change the resolution option “-rXXX” in line 38 of the script. This determines the resolution of the PNGs (read in the Ghostscript documentation about the exact meaning of the options — links to the docs are given within the source above).
Now the Python script is ready to use. Copy it to the directory containing the PDF files you want to convert and start it (under Windows: simply double click the script file). The corresponding PNG files should be created now.

Google Summer of Code end: code upload and acknowledgement

Jan-Philip Gehrcke — Mon, 24 Aug 2009 19:17:01 +0000

The Google Summer of Code 2009 final evaluation deadline is today; 19 UTC. I don’t have time to summarize my summer here now, but there are two things I want to say to the world. First, I want to thank many people for enriching my summer. Second, I would like to announce the Clobi project on Google Code.

Acknowledgement

Kate Keahey (Argnonne National Laboratory/The Globus Alliance/Nimbus team).
She offered me a fabulous mentorship. I assess this at its true worth by looking at what I’ve learned throughout the summer just because of these great conversations . Thank you for all your support and dedication and for pushing me and the project forwards!

Tim Freeman and David LaBissoniere (Argnonne National Laboratory/The Globus Alliance/Nimbus team).
Wonderful, extensive and patient extra-premium-support-with-special-treatment regarding Nimbus/Workspace in the MUD. Thank you so much! You smoothed my technical way through the project.

Ian Ward (creator of urwid, a console user interface library for Python).
He supported me so great while I was implementing the user interface for Clobi’s Resource Manager using urwid‘s new SelectEventLoop technology. Talking to him saved so much valuable time. Thank you!

Mitchell Garnaat (the creator of boto, a Python interface to Amazon Web Services) and the boto users mailinglist.
Great and essential support since almost one year now. Thank you very much for answering many questions!

Predrag Buncic (CernVM) and Artem Harutyunyan (ALICE@LHC).
Thank you for your support regarding CernVM on Nimbus!

Jakub Moscicki, Ulrik Egede, Johannes Elmsheuser and the rest of the Ganga crew.
Thanks a lot for very important, effective and efficient support concerning the system behind Clobi and Clobi’s Ganga backend. You are great!

Stefan Kluth and Stefan Stonjek (ATLAS@LHC, Max-Planck-Institut für Physik, Munich)
Thank you for many helpful discussions, support and beautiful times in Munich!

Borja Sotomayor (University of Chicago, Globus Alliance).
He made a great job as GSoC mentoring organization administrator. Thank you for this and for introducing the MUD .

Paul D Marshall.
Thank you for discussions regarding dynamical deployment of computing resources.

Xiaoming Gao.
Thank you for the exciting cooperation regarding Virtual Block Stores for Nimbus.

Alex Martelli.
He answered an important Python question that most people could not answer. Smoothed my way to implement inter thread communication in Clobi’s Resource Manager.

Clobi @ Google Code

In this blog post I introduced Clobi, the result of this Google Summer of Code project. Today, I created a new project page for Clobi on Google Code: http://code.google.com/p/clobi/

As a first action, I pushed my local mercurial code repository into the online repository. You can browse the code here and you can look through the development history (the commits I’ve made) here.

Then I prepared a test release of all Clobi components. You can get it in the download section.

That’s all about Clobi for the next time. From tomorrow on, I will go on with my master thesis project in Physics (about Magnetic Particle Imaging). Next week, the ICMRM conference in Montanta starts and I’ve to make a poster for my contribution (abstract here).

I will go on with Clobi, when there is more free time .

new system successfully tested: “Distribution of High Performance Computing Jobs among Multiple Computing Clouds”

Jan-Philip Gehrcke — Tue, 18 Aug 2009 12:14:26 +0000

Hello you out there!

I just started running the first serious test of the system I’ve developed during this year’s Google Summer of Code. If I wanted to put it in sensational words, the test could be called “Distribution of Particle Physics High Performance Computing Jobs among Multiple Computing Clouds”; just to get some readers . During the test, there will be some time I just sit around and watch my monitor, so I decided to share my experience about the new system with you and keep record of the test progress within this blog post.

Content (don’t worry, the sections are short):

0 Introduction
1 Preparation
2 VM startup
3 Session monitoring (number of Job Agents)
4 Job submission
5 Session monitoring (number of jobs)
6 Job monitoring
7 Job completion, output receipt
8 Examine output
9 VM shutdown
10 Appendix

0 Introduction

First of all, I’ve to introduce the system. This is the longest section.

It’s a job scheduling system supporting Virtual Machines (VMs) in multiple Infrastructure-as-a-Service computing clouds. Because of this, I found the name “Clobi”, which somehow comes from “cloud” and “combination”. If you know a better name, then please let me know . Currently, the system is using and supporting Nimbus (and it’s prepared for Cumulus, Nimbus’ storage service) and Amazon Web Services (more precisely EC2, SQS, SimpleDB, S3).

It’s an “elastic” and “scalable” job system that can set up a huge computing resource pool almost instantly. VMs are added to or removed from the pool dynamically; based on need and demand. Jobs are submitted and processed using an asynchronous queueing system. An arbitrary number of clients is allowed to submit jobs to an existing resource pool. The basical realiability is inherited from the reliability of the core messaging components Amazon SQS (for queueing) and Amazon SimpleDB (for bookkeeping): to prevent data from being lost or becoming unavailable, it is stored redundantly and geographically dispersed across multiple datacenters.

Furthermore, the job system’s components are highly decoupled, which allows single components to fail or to get re-initialized without affecting the others.

The motivating application for this system is ATLAS Computing (for the ATLAS experiment at LHC, CERN (Geneva)): a common ATLAS Computing application (the so-called “full chain”) will be run during this test.

But: the basic system is totally generic and can be used in any case whenever it’s convenient to distribute jobs among different clouds. This is always the case when one tries to satisfy the basic computing power needs for a low price by e.g. operating an own Nimbus cloud, but wants to able to instantly balance out peaks of desired computing power by simply adding Amazon’s EC2 to the resource pool for a certain amout of time. By using Clobi, combining different clouds to one big resource pool becomes very easy.

These are the main components used during this test:

a special ATLAS Software Virtual Machine image based on CernVM. I placed it on Nimbus Teraport Cloud and (as Amazon Machine Image) on S3 for EC2

the Clobi Resource Manager (observing job queues, starting/killing VMs, …)

the Clobi Job Agent (running on VMs, polling & running jobs, bookkeeping, …)

the Clobi Job Management Interface (providing methods to submit / remove / kill / monitor / … jobs)

a Ganga Clobi Backend (integrates Clobi into Ganga, which is “an easy-to-use frontend for job definition and management”)

The meaning of these components will become clearer in the following parts.

Let me show step by step — but only very roughly — how I use Clobi within the first serious test. Many details are left out, but you will get it in principle. After reading this blog post, you’ve an overview about what the system does and what I’ve actually done during the summer.

1 Preparation

I’ve prepared session/cloud configuration files. Using them, I started a new Clobi session (a resource pool) with the Clobi Resource Manager (It’s a Python application and I use it locally; here on my desktop machine). At first, it does much configuration and initialization stuff, including setting up SQS queues and SimpleDB domains. Interaction with Amazon Web Services is done via the boto module for Python. After initialization, the Resource Manager offers an interactive mode (it’s a multi-threaded console application with user interface, built using the urwid module for Python).

2 VM startup

Using the Resource Manager, I started one VM on EC2 and one on Nimbus Teraport Cloud, both based on “the special ATLAS Software VM”, containing ATLAS Software 15.2.0 and the Clobi Job Agent. Starting VMs manually is done with a very simple command. The driving forces in the background are boto in case of EC2 and the Nimbus cloud reference client, which I’ve wrapped and controlled via Python’s subprocess module. The main loop thread of the Resource Manager periodically polls the states of just started EC2 instances and the states of Nimbus client subprocesses to figure out if the instructed actions result in success or not.

The following screenshot shows the Resource Manager in action (you basically see a terminal window). Follow a bit of the log. As you can see, it’s very easy to run VMs and — after a certain amout of time — the Resource Manager detects that both VMs have successfully started booting:

Clobi Resource Manager showing two started Virtual Machines

The EC2 VM needed around 10 minutes to start up, while the Nimbus VM needed 20 minutes. Reason: the AMI is ~10 GB big; the image on Nimbus ~20 GB (wasted space & time, but it’s just a test..).

You might say: “Only two VMs? Boring..”. I say: I could have taken several hundred. The point is: it would not make any difference, except in cost and in the amount of space used for log files. Clobi uses technology / is designed to always work reliably; even in different orders of magnitude. This is often called “scalable” or “elastic”. Basically, this positive characteristic is inherited from Amazon’s SQS, SDB and S3, which are used by Clobi to do management and control of the system.

3 Session monitoring (number of Job Agents)

I conceal almost all the details of how the components exchange information. But I’ve to tell the following to you, to not completely confuse you:

the Resource Manager gave some bootstrap information to the VMs.

the Job Agent is automatically invoked on VM operating system startup.

Using the bootstrap information, each VM’s Job Agent “registers with SimpleDB”. The Resource Manager has a monitoring functionality to check SimpleDB for running Job Agents:

Clobi Resource Manager showing two started Clobi Job Agents

Voilà, now it’s definitely known that both VMs successfully started their Job Agent’s. These start in a “watching/lurking” state, periodically polling SQS for jobs.

4 Job submission

The SimpleDB / SQS / S3 data structure together with Clobi’s Job Management Interface allows to submit jobs with different priorities, to remove jobs, to kill running jobs and to monitor jobs. Furthermore, transmission and receipt of an input/output sandbox archive is possible. This is needed to deliver executables and small input data and to receive small output data as well as stdout/err and other logs.

I’ve downloaded Ganga and installed it to my local machine. Then, I’ve started developing a new “Clobi backend” to integrate Clobi’s Job Management Interface into Ganga. Using this new backend, it’s possible to submit/kill/monitor/.. jobs right away from the Ganga interface, using Ganga’s common job description and management commands.

To test the system, I’ve prepared some shellscripts that invoke running “The Full Chain” on the worker nodes. This is a very good test to validate the whole system: it needs some very small input files, only works if the ATLAS Software was set up properly (uoooh.. not trivial!), stresses the VM (the simulation step consumes much CPU power) and leaves some small output files for the output sandbox.

At Ganga startup, I provided a configuration file containing few but essential information about the Clobi session that I’ve set up before via Resource Manager. From this configuration file Ganga’s Clobi backend e.g. knows which SimpleDB domain to query and to which SQS queues the job messages must be submitted. Using this bootstrap information, an arbitrary number of Gangas could be used from anywhere to submit and manage jobs within this special Clobi session.

I will now submit the same job (the “full chain” thing) three times: the Nimbus VM has two virtual cores and its Job Agent will try to receive and run two jobs at the same time. The EC2 VM (m1.small) only has one virtual core. Hence, three jobs are needed to use the VMs to full capacity . This is a screenshot from the Ganga terminal session where I submitted the jobs:

Ganga: job submission via Clobi backend

The Clobi backend successfully did its job: it created three Clobi job IDs, submitted three SQS messages and uploaded three input sandbox archives.

5 Session monitoring (number of jobs)

The Resource Manager is able to observe the queues and to determine the number of jobs submitted to them. It recognizes two jobs in the queue for priority 2:

Clobi Resource Manager detected two jobs in the queues

Only two? Maybe one Job Agent polled a job right away after submission, or maybe the SQS measurement was not exact (this is possible, too). Anyway, few time later there is only one job left in the queues and then they are empty. This means that the Job Agent on the Nimbus VM successfully grabbed two jobs and the EC2 VM grabbed one:

Clobi Resource Manager detects zero jobs in the queues

While me and Ganga are waiting for the jobs to finish (this takes some time and Ganga periodically polls the state of the jobs via Clobi backend / Clobi Job Management Interface), I use the time to advise you of an important fact: it’s the objective to automate the observe-queues-and-start/kill-VMs-as-required process in the future. The current Resource Manager is very prepared for this. Let me show the monitoring loop to you:

Clobi Resource Manager showing its monitoring loop

It observes the number of jobs in the queues and the number of running Job Agents periodically. Based on this information the Resource Manager easily could start / kill virtual machines (I’ve already demonstrated how easy starting is; killing is described later). I did not implement this algorithm until now, because a) I had no time and b) I could have done it quick and dirty, but I really did not need this feature to develop and test the rest of the system. But this feature will come, because if it’s implemented properly with intelligent policies, it’s just great.

6 Job monitoring

As I’ve already mentioned, Ganga periodically checks the jobs’ states. Therefore, the Ganga Clobi backend provides a special method that Ganga calls from time to time from one of its monitoring threads. Normally this happens quietly, but I’ve put some debug output into this method. Let’s check it out:

Ganga receives monitoring information via the Clobi backend

7 Job completion, output receipt

After some more time, Ganga discovered that one of the three jobs finished successfully. This means that the Job Agent detected a returncode of 0 of the job shellscript and could successfully store the output sandbox archive to S3. At this point, the Clobi backend triggers to download and extract the output sandbox archive. This looks like:

Clobi                              : INFO     status for job-090818044445-3585-3088: completed_success
Ganga.GPIDev.Lib.Job               : INFO     job 60 status changed to "completed"
Clobi                              : INFO     download atlassessions/0907210728-testsess-0c7e/jobs/out_sndbx_job-090818044445-3585-3088.tar.bz2 from S3
Clobi                              : INFO     store key 0907210728-testsess-0c7e/jobs/out_sndbx_job-090818044445-3585-3088.tar.bz2 as file /home/gurke/gangadir/workspace/gurke/LocalAMGA/60/output/out_sndbx_job-090818044445-3585-3088.tar.bz2 from bucket atlassessions
Clobi                              : INFO     Download of output sandbox archive successfull.

Did you have doubts that this is my first serious test and everything worked until now? Some parts of the system are already tested very much, of course. But the Clobi backend for Ganga made the transition from vitally-important-features-missing to just-scratch-along-usability only a few hours ago. I’m really very happy that everything worked until now, but the output sandbox archive extraction could be improved:

Clobi                              : CRITICAL Error while extracting output sandbox
Clobi                              : CRITICAL Traceback:
Traceback (most recent call last):
  File "/mnt/hgfs/E/gsoc_code_repo/ganga_clobi_backend/Clobi/Clobi.py", line 243, in clobi_dl_extrct_outsandbox_arc
    sp = subprocess.Popen(
NameError: global name 'subprocess' is not defined

Yoooah, I (want to) use Python’s subprocess module to extract the tar.bz2 archive with system’s tar, but I forgot to import subprocess . This is forgotten and fixed easily . Btw: of course I got three downloaded output archives and three extraction errors .

8 Examine output

The output sandbox archive files were stored on my local machine by Ganga’s Clobi backend. I take a look into one of it by extracting it manually:

$ ls
out_sndbx_job-090818044451-3585-01c2.tar.bz2
$ tar xjf out_sndbx_job-090818044451-3585-01c2.tar.bz2
$ ls
AOD_007410_00001.pool.root  evgen.log  joblog_job-090818044451-3585-01c2  recoAOD.log
EVGEN_007410_00001.pool.root  jobagent_log  out_sndbx_job-090818044451-3585-01c2.tar.bz2

Great! The AOD file is there. This means that 1) Clobi did perfect work to control and manage the job and 2) the ATLAS Computing part (“The Full Chain”) worked perfectly:

the interaction between a certain particle (which was defined within the input sandbox) and the ATLAS detector was successfully simulated.

the ATLAS detector output (basically times and voltages) was calculated successfully.

particle tracks and energy deposits were successfully reconstructed from times and voltages.

an event summary was successfully built from tracks and energy deposits.

“The summary” is saved within the AOD file, which successfully returned to my local machine. Cool. Every single part of the system worked as it should (psss, don’t think of the extraction…).

9 VM shutdown

You perhaps asked yourself how to dynamically kill VMs. Besides the “hard kill” (invoking Nimbus/EC2 API calls to shut down a VM), I’ve implemented a mechanism that I call “soft kill”: the Resource Manager sets the “soft kill flag” for a specific VM (in SimpleDB) and the corresponding Job Agent checks it from time to time. When it is set, it waits until all currently running jobs are done and then the Job Agent shuts down the VM. Let’s watch it in action (I had to look up the command of my own application, too few sleep recently!):

Clobi Resource Manager setting up the softkill flag for both VMs

After waiting some time we see the number of running Job Agents decrease to zero..

Clobi Resource Manager detected that both Job Agents / VMs have shut down

10 Appendix

If you are very interested, you can find some additional material:

My earlier work on this topic (from last year), “Amazon Web Services for ATLAS Computing” (AWSAC) can be found here: http://gehrcke.de/awsac.

The aboriginal GSoC project description can be found here.

I’ve already written some blog posts about this project during Google Summer of Code.

The last visualizations of the system (from the planning period) are these two schemes: one, two.

A detailled, up-to-date and exact description of the system (“Clobi“) is planned for the future.

I will need the last days of GSoC to implement some missing and important features, to search and fix bugs and to clean everything up to make it presentable. I will definitely work on this project after GSoC (as the time allows it, of course). Currently I think about pushing the project to either bitbucket or Google code. Both support mercurial repositories and this is what I used for my code until now (locally).

If you like this, spread it! Every question and/or comment is much appreciated!

Thanks for listening