gipc 0.5.0 released

I just released gipc 0.5.0. It contains a backwards-incompatible change: the SIGPIPE signal action is not automatically reset to the default action anymore in child processes. This will hopefully satisfy developers expecting the SIGPIPE signal to be ignored, resulting in a proper Python exception when attempting to write to a pipe that has been closed on the read end (if interested, you can follow the related discussion here). Furthermore, this release improves the performance when sending large messages through gipc pipes on Linux. This release also introduces workarounds for two corner case issues present in Mac OS X and FreeBSD.

This is the full changelog:

  • Improve large message throughput on Linux (see issue #13).
  • Work around read(2) system call flaw on Mac OS X (see issue #13).
  • Work around signal.NSIG-related problem on FreeBSD (see issue #10).
  • Do not alter SIGPIPE action during child bootstrap (breaking change, see issue #12).

Thanks to Dustin Oprea, bra, John Ricklefs, Heungsub Lee, Miguel Turner, and Alex Besogonov for contributing. As usual, the release is available via PyPI (https://pypi.python.org/pypi/gipc). The documentation and further detail are available at http://gehrcke.de/gipc.

Microsoft installer: a solution to Error 1402

Windows 7 is a “stable” (in the sense of “constant”, “long-term”) system which I like to use for various desktop-based workflows. If you know how to administer it, it is a reliable platform, with many professional features. Recently, however, certain MSI-based installations started to fail on my system. I am not sure how it happened, but I obviously reached an inconsistent state. For example, I was not able to remove, repair, or install the .NET framework 4.5. Also Windows update was affected and could not install .NET updates properly anymore. There are various “FixIt” tools available from Microsoft for addressing similar issues. However, neither Windows Install Clean Up (see Wikipedia entry) nor a specialized .NET install FixIt tool helped.

The first insight I had is that MSI installations actually leave proper log files behind, containing quite some detail. In %USERPROFILE%\AppData\Local\Temp\Microsoft .NET Framework 4.5.1 Setup_20140917_191645234.html I found that

MSI (D:\4113948000d14764646c\netfx_Full_GDR_x64.msi) Installation failed.
Msi Log: Microsoft .NET Framework 4.5.1 Setup_20140917_191645234-MSI_netfx_Full_GDR_x64.msi.txt

In said ....20140917_191645234-MSI_netfx_Full_GDR_x64.msi.txt I found an issue which looked problematic:

MSI (s) (4C:80) [19:19:19:653]: Product: Microsoft .NET Framework 4.5.1 -- Error 1402.
Could not open key:
UNKNOWN\Components\D30312665D61E0E44A85EEBAFAF27C86\BE4EBED704B66673BB53C5BB3C58AD73.
System error 5.
Verify that you have sufficient access to that key, or contact your support personnel.

I ran regedit.exe with administrator privileges and found that, indeed, said key existed, whereas the UNKNOWN in above’s error message really is HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData\S-1-5-18\. Said key indeed had weird permissions set — precisely none. It had no owner and absolutely no access control list (ACL) specifications. I figured that this is improper, and assumed that other keys below Components might be affected, by whichever cause.

Likewise, I tried to recursively change (or “normalize”) the permissions of all keys in Components. So, first of all, I added ownership to the group of Administrators to Components and selected Replace owner on subcontainers and objects. This is important as a first step, because without having defined the ownership, setting ACLs is impossible. I then added “full control” permissions for the Administrators group while selecting Replace all child object permissions with inheritable permissions from this object. In the first run, this actually errored by saying that not all children could be re-configured. The second attempt succeeded. It is annoying, but the number of repetitions required to normalize the permissions of the entire tree might actually depend on the depth of the hierarchy. In this Components directory, two runs were sufficient.

Again, I have no idea how that inconsistent state was reached, but this ACL normalization actually fixed the issue.

And before someone flames at Windows for reaching this inconsistent software installation state: one has to appreciate that one is in equally deep trouble once one manually tampers with package-related files in a Linux system without making the package manager aware of the changes. Such a situation might lead to daunting debugging sessions, often followed by the insight that a complete rewind is the best solution. What I want to say is that it is likely that I myself produced that erroneous state, without realizing.

Repopulate a Minecraft world from the command line

From http://minecraft.gamepedia.com/Chunk_format

TerrainPopulated: 1 or not present (true/false) indicate whether the terrain in this chunk was populated with special things. (Ores, special blocks, trees, dungeons, flowers, waterfalls, etc.) If set to zero then Minecraft will regenerate these features in the blocks that already exist.

Sometimes people want to “repopulate” their worlds, i.e. set the TerrainPopulated property to False for all chunks of their world, in order to regenerate the special things mentioned above. MCEdit, a Minecraft world editor with a graphical user interface can do this. However, there is a more straight-forward solution to this task, especially if you are running a remote Minecraft server on a headless machine.

MCEdit is backed by pymclevel, a Python library for reading and modifying Minecraft worlds. It has been created by David Rio Vierra (kudos!) and is in development since 2010. Its documentation is rather weak, but its API is not too difficult to understand. I could rather quickly come up with a small Python application that reads a world, iterates through all chunks, resets the TerrainPopulated property for all of them, and saves the world back to disk. This is the code, which I just tested for our own world, but it should very well work for yours, too:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
 
"""
Copyright 2014 Jan-Philip Gehrcke (http://gehrcke.de)
 
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
 
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
 
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
"""
 
"""
Re-populate a minecraft world. That is, set the TerrainPopulated property
to False for all chunks in a world.
 
From http://minecraft.gamepedia.com/Chunk_format:
 
TerrainPopulated: 1 or not present (true/false) indicate whether the terrain
in this chunk was populated with special things. (Ores, special blocks, trees,
dungeons, flowers, waterfalls, etc.) If set to zero then Minecraft will
regenerate these features in the blocks that already exist.
 
Based on the notes in
https://github.com/mcedit/pymclevel/blob/master/mclevel.py
 
and on the repop method used in
https://github.com/mcedit/mcedit/blob/master/editortools/chunk.py
 
Usage:
 
repop.py path/to/world/directory
 
Currently, an Alpha level format is expected:
http://minecraft.gamepedia.com/Alpha_level_format
 
An world in Alpha level format is a single directory containing at least one
file named level.dat.
"""
 
 
import os
import sys
import time
import logging
 
logging.basicConfig(
    format='%(asctime)s,%(msecs)-6.1f - %(module)s:%(levelname)s: %(message)s',
    datefmt='%H:%M:%S')
log = logging.getLogger()
log.setLevel(logging.INFO)
 
 
install_note = """
$ virtualenv pyvenv
$ source pyvenv/bin/activate
$ pip install cython numpy pyyaml
$ pip install git+git://github.com/mcedit/pymclevel.git
"""
 
 
try:
    from pymclevel import mclevel
except ImportError:
    sys.exit("Cannot import pymclevel. Consider setting it up via %s" %
        install_note)
 
usage = "%s path/to/world/directory" % os.path.basename(sys.argv[0])
if not len(sys.argv) == 2:
    sys.exit("One argument is required. Usage: %s" % usage)
 
world_directory = sys.argv[1]
 
if not os.path.isdir(world_directory):
    sys.exit("Not a directory: %s" % world_directory)
 
log.info(("Attempting to read world. This scans the directory "
    "for chunks. Might take a while."))
world = mclevel.fromFile(world_directory)
 
log.info("Get chunk positions iterator.")
chunk_positions = world.allChunks
 
log.info("Iterate through chunks, set TerrainPopulated=0 for all of them.")
t0 = time.time()
for idx, (x, z) in enumerate(chunk_positions):
    if (idx + 1) % 1000 == 0:
        log.info("Processed %s chunks." % (idx + 1))
    # Retrieve an AnvilChunk object. This object will load and
    # decompress the chunk as needed, and remember whether it
    # needs to be saved or relighted.
    chunk = world.getChunk(x, z)
    chunk.TerrainPopulated = False
    # The above sets `chunk.dirty` which is processed during
    # `saveInPlace()` below. Leads to `saveChunk(cx, cz, data)`.
duration = time.time() - t0
chunks_per_sec = (idx + 1) / duration
log.info("Total number of modified chunks: %s." % (idx+1))
log.info("Duration: %.2f s. Chunks per second: %.2f." % (
    duration, chunks_per_sec))
 
# Save the level.dat and any chunks that have been marked for
# writing to disk. This also compresses any chunks marked for
# recompression.
log.info("Save modified world to disk (might take a moment).")
world.saveInPlace()
log.info("Exiting.")

I called it repop.py, and this is how it executes:

$ python repop.py schaumwelt
23:55:08,262.2  - materials:INFO: Loading block info from <open file 'pymclevel/minecraft.yaml', mode 'r' at 0x21546f0>
23:55:09,22.4   - materials:INFO: Loading block info from <open file 'pymclevel/classic.yaml', mode 'r' at 0x21546f0>
23:55:09,118.4  - materials:INFO: Loading block info from <open file 'pymclevel/indev.yaml', mode 'r' at 0x2154660>
23:55:09,233.3  - materials:INFO: Loading block info from <open file 'pymclevel/pocket.yaml', mode 'r' at 0x21546f0>
23:55:09,628.0  - repop:INFO: Attempting to read world. This scans the directory for chunks. Might take a while.
23:55:09,628.2  - mclevel:INFO: Identifying schaumwelt
23:55:09,628.5  - mclevel:INFO: Detected Infdev level.dat
23:55:09,728.0  - infiniteworld:INFO: Found dimension DIM1
23:55:09,728.5  - infiniteworld:INFO: Found dimension DIM-1
23:55:09,728.9  - infiniteworld:INFO: Found dimension DIM-17
23:55:09,729.2  - repop:INFO: Get chunk positions iterator.
23:55:09,729.3  - infiniteworld:INFO: Scanning for regions...
23:55:09,745.4  - regionfile:INFO: Found region file r.0.4.mca with 731/807 sectors used and 598 chunks present
[...]
23:55:10,672.4  - regionfile:INFO: Found region file r.-2.-1.mca with 1324/1418 sectors used and 1024 chunks present
23:55:10,674.2  - repop:INFO: Iterate through chunks, set TerrainPopulated=0 for all of them.
23:55:12,70.1   - regionfile:INFO: Found region file r.-2.1.mca with 2/2 sectors used and 0 chunks present
[...]
23:55:16,51.1   - regionfile:INFO: Found region file r.-2.3.mca with 2/2 sectors used and 0 chunks present
23:55:16,559.9  - regionfile:INFO: Found region file r.3.3.mca with 2/2 sectors used and 0 chunks present
23:55:18,69.6   - repop:INFO: Processed 1000 chunks.
23:55:18,958.6  - regionfile:INFO: Found region file r.-4.2.mca with 2/2 sectors used and 0 chunks present
23:55:21,334.1  - regionfile:INFO: Found region file r.2.-3.mca with 2/2 sectors used and 0 chunks present
23:55:26,524.2  - repop:INFO: Processed 2000 chunks.
23:55:35,573.8  - repop:INFO: Processed 3000 chunks.
23:55:44,324.8  - repop:INFO: Processed 4000 chunks.
[...]
00:00:18,820.3  - repop:INFO: Processed 35000 chunks.
00:00:28,940.9  - repop:INFO: Processed 36000 chunks.
00:00:38,30.5   - repop:INFO: Processed 37000 chunks.
00:00:41,787.5  - repop:INFO: Total number of modified chunks: 37426.
00:00:41,788.0  - repop:INFO: Duration: 331.11 s. Chunks per second: 113.03.
00:00:41,788.2  - repop:INFO: Save modified world to disk (might take a moment).
00:00:41,815.3  - infiniteworld:INFO: Saved 0 chunks (dim 1)
00:00:41,842.6  - infiniteworld:INFO: Saved 0 chunks (dim -1)
00:00:41,870.5  - infiniteworld:INFO: Saved 0 chunks (dim -17)
00:00:43,857.3  - regionfile:INFO: Found region file r.0.4.mca with 720/720 sectors used and 589 chunks present
[...]00:00:44,723.8  - regionfile:INFO: Found region file r.-2.-1.mca with 1310/1310 sectors used and 1014 chunks present
00:01:13,807.5  - infiniteworld:INFO: Saved 37426 chunks (dim 0)
00:01:13,808.0  - repop:INFO: Exiting.

Hope that helps.

Leitung ahoi

Ich trinke seit Jahren hauptsächlich Wasser aus der Leitung. Im Büro und zu Hause. Du nicht? Zwei kleine Zitate von Martin Wagner (Toxikologe, Uni Frankfurt):

Das Leitungswasser, das wir untersucht haben, war nicht mit Umwelthormonen belastet. Warum nicht einfach das am strengsten kontrollierte Wasser in Deutschland trinken, nämlich das, was aus dem Hahn kommt? Das ist 1000- bis 5000-mal günstiger, muss nicht verpackt, mit hohem Energieaufwand abgefüllt und transportiert werden und verursacht keinen Plastikmüll. Für mich ist die Wahl da offensichtlich.

und

Grundsätzlich gelten für Leitungswasser strengere Richtlinien. Was viele nicht wissen: Mineralwasser darf nicht aufbereitet werden, Leitungswasser hingegen wird aufwendig kontrolliert und gereinigt. Dadurch ist die Belastung mit Pestiziden in der Regel niedriger und seltener. Auch eine hohe Keimbelastung, wie in einem der Wasser im Test, ist beim Leitungswasser deshalb äußerst unwahrscheinlich.

Quelle: Spiegel Online.

“Aber, aber!” mag manch einer sagen, “Ich brauche doch mein Mondwasser!”. Auch das kannst Du Dir selbst herstellen… :-)

Um Mondwasser herzustellen, muss einfach normales Wasser in eine Karaffe gefüllt werden und diese am Fenster aufgestellt werden. Besser ist es noch, wenn die Karaffe im Freien aufgestellt wird, damit sie auch direkt vom Mond angestrahlt werden kann. Stellt man die Karaffe in einer Nacht mit Vollmond auf, dann bekommt dieses Wasser sehr viel Energie vom Mond geliefert.

Ob das vielleicht die Bachblüten-Ilse geschrieben hat? Es gibt offenbar auch Mondkäse, Mondholz und Vollmondsalami zu kaufen. Die Quelle zu diesem Unsinn verrate ich jedenfalls nicht, das Internet sollte davor bewahrt werden.

Hello SPF record.

Introduction

Say Google’s or Yahoo’s mail transfer agent (MTA) retrieves an email sent from an MTA running on the host with the (fictional) IP address 1.1.1.1. Say the sending MTA identifies itself as sending on behalf of important.com (within the EHLO/HELO handshake, or with a From address with the domain important.com). Then the receiving MTA (Google or Yahoo, or something else) usually queries the important.com domain via DNS for the so-called SPF (Sender Policy Framework) record. Using this SPF record the retrieving MTA can verify whether the owner of the domain important.com intended to send mail from 1.1.1.1 or not.

The SPF record might say that important.com does not send mails. Then the receiving MTA knows for sure that the incoming mail is invalid/fake/spam. Or this record says that important.com actually sends mail, but only from the (fictional) IP address 2.2.2.2. Then the retrieving MTA also knows for sure that something is wrong with the mail. Another possibility is that the record states that important.com actually sends mail from 1.1.1.1, in which case the incoming mail likely is intended by the owner of the domain important.com. This does not mean that this mail is not spam, but at least one can be quite sure that nobody crafted that mail on behalf of important.com.

That was a simplified description for why it is important to have a proper SPF record set for a certain domain in case one intends to send mail from this domain. In the next paragraphs I explain how I have set the SPF record for gehrcke.de and why.

SPF record for gehrcke.de

Domainfactory is the tech-c for gehrcke.de. However, the DNS entries for gehrcke.de are managed by Cloudflare (free, and quite comfortable). Today, I added an SPF record for gehrcke.de via Cloudflare’s web interface. The first important thing to take note of is that the SPF record actually is a TXT record. There once was a distinct SPF record proposed, but in RFC 7208 (which specifies SPF) we clearly read:

The SPF record is expressed as a single string of text found in the RDATA of a single DNS TXT resource record

and

SPF records MUST be published as a DNS TXT (type 16) Resource Record (RR) [RFC1035] only. Use of alternative DNS RR types was supported in SPF’s experimental phase but has been discontinued.

By now it should be clear that I had to add a new TXT record for gehrcke.de, with a special syntax, the SPF syntax. This syntax is specified here. What I need to express in the record is that mail sent on behalf of gehrcke.de is only sent by the machine with the IPv4 address 5.45.109.1 / or from the IPv6 subnet fe80::5054:8dff:fe9b:358d/64. This is the machine that I know should send mail. No other people / machines should send mail on behalf of my domain. The corresponding record is:

v=spf1 ip4:5.45.109.1 ip6:fe80::5054:8dff:fe9b:358d/64 -all

The prefix v=spf1 declares the SPF version, the ip4 and ip6 markers declare that mail from a certain address or subnet are valid. -all expresses that mail from all other hosts is disallowed. The fact that I do not “allow” others to send mail on behalf of gehrcke.de is just my declaration of intent. A retrieving MTA is not required to respect my SPF record. However, larger mail providers should respect it and at least classify a mail that does not pass the SPF test as spam, if not reject it right away.

Whenever I switch machines or IP addresses, I need to remember to update this record. This is a minor disadvantage. To circumvent this, in many cases a short

v=spf1 a -all

would be enough. It expresses that mail sent from the IP address corresponding to the DNS A record of the domain is allowed. In my case, however, this does not work since the A record points to Cloudflare’s cache rather than directly to my machine.

Testing the record

There are many web-based tools for checking whether a certain domain has an SPF record set and whether its syntax is valid. Particularly useful is, for instance, mxtoolbox.com/SuperTool.aspx. However, a real full-system validation obviously requires you to send a mail to an external service that resembles a serious mail provider and debugs the communication for you, meaning that it informs you about all externally visible details of your mail setup, including the SPF record. A great verification tool that does exactly this is provided by Port25. On the command line I used my local MTA (Exim4) to send mail to their service, instructing them to return the corresponding report to may Googlemail address:

$ echo "This is a test." | mail -s Testing check-auth-jgehrcke=googlemail.com@verifier.port25.com

The following is an excerpt of the test result, indicating that the actual mail sender matches the data provided in the SPF record:

HELO hostname:  gurke2.gehrcke.de
Source IP:      5.45.109.1
mail-from:      user@gehrcke.de
 
----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass 
ID(s) verified: smtp.mailfrom=user@gehrcke.de
DNS record(s):
    gehrcke.de. SPF (no records)
    gehrcke.de. 300 IN TXT "v=spf1 ip4:5.45.109.1 ip6:fe80::5054:8dff:fe9b:358d/64 -all"

Great.