API reference

Spawning child processes

gipc.start_process(target, args=(), kwargs={}, daemon=None, name=None)

Start child process and execute function target(*args, **kwargs). Any existing instance of gipc._GIPCHandle or gipc._GIPCDuplexHandle can be passed to the child process via args and/or kwargs. If any such instance is passed to the child, gipc automatically closes the corresponding file descriptor(s) in the parent.

Note

Compared to the canonical multiprocessing.Process() constructor, this function

  • returns a gipc._GProcess instance which is compatible with the multiprocessing.Process API.
  • just as well takes the target, arg=(), and kwargs={} arguments.
  • introduces the daemon=None argument.
  • does not accept the group argument (being an artifact from multiprocessing’s compatibility with threading).
  • starts the process, i.e. a subsequent call to the start() method of the returned object is not required.
Parameters:
  • target – Function to be called in the child process. Signature: target(*args, **kwargs).
  • args – Tuple defining the positional arguments provided to target.
  • kwargs – Dictionary defining the keyword arguments provided to target.
  • name – Forwarded to multiprocessing.Process.name.
  • daemon – Forwarded to multiprocessing.Process.daemon.
Returns:

gipc._GProcess instance (inherits from multiprocessing.Process and re-implements some of its methods in a gevent-cooperative fashion).

start_process() triggers most of the magic in gipc. Process creation is based on multiprocessing.Process(), i.e. fork() on POSIX-compliant systems and CreateProcess() on Windows.

Warning

Please note that in order to provide reliable signal handling in the context of libev, the default disposition (action) is restored for all signals in the child before executing the user-given target function. You can (re)install any signal handler within target. The notable exception is the SIGPIPE signal, whose handler is not reset to its default handler in child processes created by gipc. That is, the SIGPIPE action in children is inherited from the parent. In CPython, the default action for SIGPIPE is SIG_IGN, i.e. the signal is ignored.

Creating a pipe and its handle pair

gipc.pipe(duplex=False, encoder='default', decoder='default')

Create a pipe-based message transport channel and return two corresponding handles for reading and writing data.

Allows for gevent-cooperative transmission of data between greenlets within one process or across processes (created via start_process()). The default behavior allows for transmission of any picklable Python object.

The transport layer is based on os.pipe() (i.e. CreatePipe() on Windows and pipe() on POSIX-compliant systems).

Parameters:
  • duplex –
    • If False (default), create a unidirectional pipe-based message transport channel and return the corresponding handle pair, a 2-tuple with the first element of type _GIPCReader and the second element of type _GIPCWriter.
    • If True, create a bidirectional message transport channel (using two pipes internally) and return the corresponding 2-tuple with both elements being of type _GIPCDuplexHandle.
  • encoder – Defines the entity used for object serialization before writing object o to the pipe via put(o). Must be either a callable returning a byte string, None, or 'default'. 'default' translates to pickle.dumps (in this mode, any pickleable Python object can be provided to put() and transmitted through the pipe). When setting this to None, no automatic object serialization is performed. In that case only byte strings are allowed to be provided to put(), and a TypeError is thrown otherwise. A TypeError will also be thrown if the encoder callable does not return a byte string.
  • decoder – Defines the entity used for data deserialization after reading raw binary data from the pipe. Must be a callable retrieving a byte string as first and only argument, None or 'default'. 'default' translates to pickle.loads. When setting this to None, no data decoding is performed, and a raw byte string is returned.
Returns:

gipc._GIPCHandle and gipc._GIPCDuplexHandle instances are recommended to be used with a context manager as indicated in the following examples:

with pipe() as (r, w):
    do_something(r, w)

reader, writer = pipe()
with reader:
    do_something(reader)
    with writer as w:
        do_something(w)

with pipe(duplex=True) as (h1, h2):
    h1.put(1)
    assert h2.get() == 1
    h2.put(2)
    assert h1.get() == 2

An example for using the encoder/decoder arguments for implementing JSON (de)serialization:

import json
enc = lambda o: json.dumps(o).encode("ascii")
dec = lambda b: json.loads(b.decode("ascii"))
with pipe(encoder=enc, decoder=dec) as (r, w):
    ...

Note that JSON representation is text whereas the encoder/decoder callables must return/accept byte strings, as ensured here by ASCII en/decoding. Also note that in practice JSON serializaton has normally no advantage over pickling, so this is just an educational example.

Handling handles

class gipc.gipc._GIPCHandle

The _GIPCHandle class implements common features of read and write handles. _GIPCHandle instances are created via pipe().

close()

Close underlying file descriptor and de-register handle from further usage. Is called on context exit.

Raises:
  • GIPCError
  • GIPCClosed
  • GIPCLocked
class gipc.gipc._GIPCWriter

Bases: gipc.gipc._GIPCHandle

A _GIPCWriter instance manages the write end of a pipe. It is created via pipe().

put(o)

Encode object o and write it to the pipe. Block gevent-cooperatively until all data is written. The default encoder is pickle.dumps.

Parameters:o – a Python object that is encodable with the encoder of choice.
Raises:
  • GIPCError
  • GIPCClosed
  • pickle.PicklingError
class gipc.gipc._GIPCReader

Bases: gipc.gipc._GIPCHandle

A _GIPCReader instance manages the read end of a pipe. It is created via pipe().

get(timeout=None)

Receive, decode and return data from the pipe. Block gevent-cooperatively until data is available or timeout expires. The default decoder is pickle.loads.

Parameters:timeout – None (default) or a gevent.Timeout instance. The timeout must be started to take effect and is canceled when the first byte of a new message arrives (i.e. providing a timeout does not guarantee that the method completes within the timeout interval).
Returns:a Python object.
Raises:
  • gevent.Timeout (if provided)
  • GIPCError
  • GIPCClosed
  • pickle.UnpicklingError

Recommended usage for silent timeout control:

with gevent.Timeout(TIME_SECONDS, False) as t:
    reader.get(timeout=t)

Warning

The timeout control is currently not available on Windows, because Windows can’t apply select() to pipe handles. An OSError is expected to be raised in case you set a timeout.

class gipc.gipc._GIPCDuplexHandle

A _GIPCDuplexHandle instance manages one end of a bidirectional pipe-based message transport created via pipe() with duplex=True. It provides put(), get(), and close() methods which are forwarded to the corresponding methods of gipc._GIPCWriter and gipc._GIPCReader.

Controlling child processes

class gipc.gipc._GProcess

Bases: multiprocessing.context.Process

Compatible with the multiprocessing.Process API.

For cooperativeness with gevent and compatibility with libev, it currently re-implements start(), is_alive(), close(), exitcode on Unix and join() on Windows as well as on Unix.

Note

On Unix, child monitoring is implemented via libev child watchers. To that end, libev installs its own SIGCHLD signal handler. Any call to os.waitpid() would compete with that handler, so it is not recommended to call it in the context of this module. gipc prevents multiprocessing from calling os.waitpid() by monkey-patching multiprocessing’s Popen.poll to be no-op and to always return None. Calling gipc._GProcess.join() is not required for cleaning up after zombies (libev does). It just waits for the process to terminate.

Exception types

exception gipc.GIPCError

Is raised upon general errors. All other exception types derive from this one.

exception gipc.GIPCLocked

Is raised upon attempt to close a handle which is currently locked for I/O.

exception gipc.GIPCClosed

Is raised upon operation on closed handle.