Hello SPF record.

Introduction

Say Google’s or Yahoo’s mail transfer agent (MTA) retrieves an email sent from an MTA running on the host with the (fictional) IP address 1.1.1.1. Say the sending MTA identifies itself as sending on behalf of important.com (within the EHLO/HELO handshake, or with a From address with the domain important.com). Then the receiving MTA (Google or Yahoo, or something else) usually queries the important.com domain via DNS for the so-called SPF (Sender Policy Framework) record. Using this SPF record the retrieving MTA can verify whether the owner of the domain important.com intended to send mail from 1.1.1.1 or not.

The SPF record might say that important.com does not send mails. Then the receiving MTA knows for sure that the incoming mail is invalid/fake/spam. Or this record says that important.com actually sends mail, but only from the (fictional) IP address 2.2.2.2. Then the retrieving MTA also knows for sure that something is wrong with the mail. Another possibility is that the record states that important.com actually sends mail from 1.1.1.1, in which case the incoming mail likely is intended by the owner of the domain important.com. This does not mean that this mail is not spam, but at least one can be quite sure that nobody crafted that mail on behalf of important.com.

That was a simplified description for why it is important to have a proper SPF record set for a certain domain in case one intends to send mail from this domain. In the next paragraphs I explain how I have set the SPF record for gehrcke.de and why.

SPF record for gehrcke.de

Domainfactory is the tech-c for gehrcke.de. However, the DNS entries for gehrcke.de are managed by Cloudflare (free, and quite comfortable). Today, I added an SPF record for gehrcke.de via Cloudflare’s web interface. The first important thing to take note of is that the SPF record actually is a TXT record. There once was a distinct SPF record proposed, but in RFC 7208 (which specifies SPF) we clearly read:

The SPF record is expressed as a single string of text found in the RDATA of a single DNS TXT resource record

and

SPF records MUST be published as a DNS TXT (type 16) Resource Record (RR) [RFC1035] only. Use of alternative DNS RR types was supported in SPF’s experimental phase but has been discontinued.

By now it should be clear that I had to add a new TXT record for gehrcke.de, with a special syntax, the SPF syntax. This syntax is specified here. What I need to express in the record is that mail sent on behalf of gehrcke.de is only sent by the machine with the IPv4 address 5.45.109.1 / or from the IPv6 subnet fe80::5054:8dff:fe9b:358d/64. This is the machine that I know should send mail. No other people / machines should send mail on behalf of my domain. The corresponding record is:

v=spf1 ip4:5.45.109.1 ip6:fe80::5054:8dff:fe9b:358d/64 -all

The prefix v=spf1 declares the SPF version, the ip4 and ip6 markers declare that mail from a certain address or subnet are valid. -all expresses that mail from all other hosts is disallowed. The fact that I do not “allow” others to send mail on behalf of gehrcke.de is just my declaration of intent. A retrieving MTA is not required to respect my SPF record. However, larger mail providers should respect it and at least classify a mail that does not pass the SPF test as spam, if not reject it right away.

Whenever I switch machines or IP addresses, I need to remember to update this record. This is a minor disadvantage. To circumvent this, in many cases a short

v=spf1 a -all

would be enough. It expresses that mail sent from the IP address corresponding to the DNS A record of the domain is allowed. In my case, however, this does not work since the A record points to Cloudflare’s cache rather than directly to my machine.

Testing the record

There are many web-based tools for checking whether a certain domain has an SPF record set and whether its syntax is valid. Particularly useful is, for instance, mxtoolbox.com/SuperTool.aspx. However, a real full-system validation obviously requires you to send a mail to an external service that resembles a serious mail provider and debugs the communication for you, meaning that it informs you about all externally visible details of your mail setup, including the SPF record. A great verification tool that does exactly this is provided by Port25. On the command line I used my local MTA (Exim4) to send mail to their service, instructing them to return the corresponding report to may Googlemail address:

$ echo "This is a test." | mail -s Testing check-auth-jgehrcke=googlemail.com@verifier.port25.com

The following is an excerpt of the test result, indicating that the actual mail sender matches the data provided in the SPF record:

HELO hostname:  gurke2.gehrcke.de
Source IP:      5.45.109.1
mail-from:      user@gehrcke.de
 
----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass 
ID(s) verified: smtp.mailfrom=user@gehrcke.de
DNS record(s):
    gehrcke.de. SPF (no records)
    gehrcke.de. 300 IN TXT "v=spf1 ip4:5.45.109.1 ip6:fe80::5054:8dff:fe9b:358d/64 -all"

Great.

Discourse on Debian Wheezy via Docker

Introduction

The makers of the StackExchange network have been working on Discourse for quite a while now. It is a modern communication platform, with the goal to be a nice mixture of classical internet forums, mailing lists, and established social media features:

Discourse is a from-scratch reboot, an attempt to reimagine what a modern, sustainable, fully open-source Internet discussion platform should be today – both from a technology standpoint and a sociology standpoint.

Technically, Discourse is based on various single components with Ruby on Rails at its core, Redis for caching, Sidekiq for micro job management, and PostgreSQL as a persistent database backend. Likewise, deploying Discourse right from the git repository can be a time-consuming task, not to mention the implications of maintaining a Discourse instance or running multiple Discourse instances on the same host. Discourse realized that the complicated deployment certainly prevents a couple of people from using it. Fortunately, the awesome “leightweight VM” container system Docker became production-ready in June 2014:

This release’s “1.0” label signifies a level of quality, feature completeness, backward compatibility and API stability to meet enterprise IT standards. In addition, to provide a full solution for using Docker in production we’re also delivering complete documentation, training programs, professional services, and enterprise support.

… and Discourse consequently decided to work on a Docker-based deployment system, which is the default by now, and even the only supported method (as you can see, Discourse evolves quite quickly). However, Docker is officially supported only on very modern Linux distributions, because Docker’s libcontainer as well as AuFS require kernel features that were only recently introduced. Consequently, Docker (and with that Discourse) is not supported on e.g. Debian 7 (Wheezy), which is the current “stable” Debian as sysadmins love it. But there is something we can do about it.

Although not officially supported or recommended, Docker and Discourse can be run on Debian 7 by backporting a more modern Linux kernel. Currently, Wheezy comes with a kernel from the 3.2 branch. The wheezy-backports (or Debian 8) kernel currently is from the 3.14 branch, which is new enough. Is using a kernel from backports safe? I can only tell that my system runs without quirks, and as you will find on the web, a couple of other people are also running successfully with a kernel from backports.

Hence, if you want to deploy Discourse on Wheezy, you can do it, and it will work perfectly fine. I will quickly go through the required steps in the next sections.

Using a kernel from backports

Follow the official instructions for using the backports repository:

# echo "deb http://ftp.de.debian.org/debian wheezy-backports main" > \
    /etc/apt/sources.list.d/wheezy-backports.list
# apt-get update

Get the newer kernel:

# apt-get -t wheezy-backports install linux-image-amd64 linux-headers-amd64

Reboot:

# reboot

Verify:

$ uname -a
Linux gurke2 3.14-0.bpo.1-amd64 #1 SMP Debian 3.14.12-1~bpo70+1 (2014-07-13) x86_64 GNU/Linux

Install Docker

Following the official installation instructions for Ubuntu (which Debian is very close to):

# apt-get install apt-transport-https
# apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
# sh -c "echo deb https://get.docker.io/ubuntu docker main \
   > /etc/apt/sources.list.d/docker.list"
# apt-get update
# apt-get install lxc-docker

Install Discourse ecosystem

Following the instructions from here and here:

# install -g docker -m 2775 -d /var/docker
# adduser discourseuser
# usermod -a -G docker discourseuser

Now, as user discourseuser:

$ git clone https://github.com/discourse/discourse_docker.git /var/docker

Configure Discourse app

As user discourseuser:

$ cd /var/docker
$ cp samples/standalone.yml containers/app.yml

Edit app.yml to your needs. In my case, I made the Discourse container’s web server and SSH server not listen on an external interface, by changing the expose section to:

expose:
  - "127.0.0.1:8200:80"
  - "127.0.0.1:2222:22"

That means that web and SSH servers of the container are reachable through localhost, but not from outside. This has important security advantages:

  • We can proxy the HTTP traffic of the Discourse web application through a web server running on the host (e.g. nginx), and encrypt the connection (the Discourse Docker container does not have TLS support built-in, duh).
  • Why would you want to expose the Discourse container via SSH to the internet, anyway? That’s a severe mistake, in my opinion, and you should disable that unless you have a very good reason not to.

I have set DISCOURSE_SMTP_ADDRESS: 172.17.42.1, as 172.17.42.1 is the IP address of the host in the private virtual Docker network in my case. On my Debian host, I am running an Exim4 MTA, which is configured with dc_local_interfaces='172.17.42.1' and dc_relay_nets='172.17.0.0/16', i.e. it listens on the local Docker network and relays mails incoming from that network. That way, the Discourse instance running within a Docker container can send mail through the Exim MTA running on the Debian host. I have described that in more detail in another blog post.

Each change of the configuration file app.yml requires a re-build via ./launcher rebuild app. A re-build implicates stopping, destructing, bootstrapping, and launching the container. What I have learned just recently: don’t worry too much about the word “destruction”. It does not implicate data loss. You can re-build your Discourse container at any time. Persistent data is stored in the PostgreSQL database and in the shared volume, both of which are not affected by a re-build.

Hope that helps!

Discourse Docker container: send mail through Exim

Introduction

The Discourse deployment was greatly simplified by introducing Docker support (as I have written about before). Discourse heavily depends on e-mail, and its ability to send mail to arbitrary recipients is essential. While the recommended way is to use an external service like Mandrill, it is also possible to use a local MTA, such as Exim. However, when you set up the vanilla Discourse Docker container, it does not contain an pre-configured MTA, which is fine, since many have a well-configured MTA running on the host already. The question is how to use that MTA for letting Discourse send mail.

Usually, MTAs on smaller machines are configured to listen on localhost only, to not be exposed to the Internet and to not be mis-used for spam. localhost on the host itself, however, is different from localhost within a Docker container. The network within the container is a virtual one, and it is cleanly separated from the host. That is, when Discourse running in a container tries to reach an SMTP server on localhost, it cannot reach an MTA listening on localhost outside of the container. There is a straight-forward solution: Docker comes along with a network bridge. In fact, it provides a private network (in the 172.17.x.x range) that connects single containers with the host. This network can be used for establishing connectivity between a network application within a Docker container and the host.

Exim’s network configuration

Likewise, I have set up Exim4 on the Debian host for relaying mails that are incoming from localhost or from the local virtual Docker network. First I looked up the IP address of the docker bridge on the host, being 172.17.42.1 in my case (got that from /sbin/ifconfig). I then instructed Exim to treat this as local interface and listen on it. Also, Exim was explicitly told to relay mail incoming from the subnet 172.17.0.0/16, otherwise it would reject incoming mails from that network. These are the relevant keys in /etc/exim4/update-exim4.conf.conf:

dc_local_interfaces='127.0.0.1;172.17.42.1'
dc_relay_nets='172.17.0.0/16'

The config update is in place after calling update-exim4.conf and restarting Exim via service exim4 restart.

Testing SMTP access from within container

I tested if Exim’s SMTP server can be reached from within the container. I used the bare-bones SMTP implementation of Python’s smtplib for that. First of all, I SSHd into the container by calling launcher ssh app. I then called python. The following Python session demonstrates how I attempted to establish an SMTP connection right to the host via its IP address in Docker’s private network:

>>> import smtplib
>>> server = smtplib.SMTP('172.17.42.1')
>>> server.set_debuglevel(1)
>>> server.sendmail("rofl@gehrcke.de", "jgehrcke@googlemail.com", "test")
send: 'ehlo [172.17.0.54]\r\n'
reply: '250-localhost Hello [172.17.0.54] [172.17.0.54]\r\n'
reply: '250-SIZE 52428800\r\n'
reply: '250-8BITMIME\r\n'
reply: '250-PIPELINING\r\n'
reply: '250 HELP\r\n'
reply: retcode (250); Msg: localhost Hello [172.17.0.54] [172.17.0.54]
SIZE 52428800
8BITMIME
PIPELINING
HELP
send: 'mail FROM:<rofl@gehrcke.de> size=4\r\n'
reply: '250 OK\r\n'
reply: retcode (250); Msg: OK
send: 'rcpt TO:<jgehrcke@googlemail.com>\r\n'
reply: '250 Accepted\r\n'
reply: retcode (250); Msg: Accepted
send: 'data\r\n'
reply: '354 Enter message, ending with "." on a line by itself\r\n'
reply: retcode (354); Msg: Enter message, ending with "." on a line by itself
data: (354, 'Enter message, ending with "." on a line by itself')
send: 'test\r\n.\r\n'
reply: '250 OK id=1X9bpF-0000st-Od\r\n'
reply: retcode (250); Msg: OK id=1X9bpF-0000st-Od
data: (250, 'OK id=1X9bpF-0000st-Od')
{}

Indeed, the mail arrived at my Google Mail account. This test shows that the Exim4 server running on the host is reachable via SMTP from within the Discourse Docker instance. Until I got the configuration right, I observed essentially two different classes of errors:

  • socket.error: [Errno 111] Connection refused in case there is no proper network routing or connectivity established.
  • smtplib.SMTPRecipientsRefused: {'jgehrcke@googlemail.com': (550, 'relay not permitted')} in case the Exim4 SMTP server is reachable, but rejecting your mail (for this to solve I had to add the dc_relay_nets='172.17.0.0/16' to the config shown above).

Obviously, in order to make Discourse use that SMTP server, it needs to be configured with DISCOURSE_SMTP_ADDRESS being set to the IP address of the host in the Docker network, i.e. 172.17.42.1 in my case.

Hope that helps!

You should let ‘SamKnows’ know.

For a couple of years now I have a network device in my place, called “Whitebox” by SamKnows. It is part of the following endeavor:

Together, the European Commission and SamKnows aim to provide Europe with reliable and accurate statistics of broadband performance across Europe.

Volunteers will receive a purpose-built broadband measurement unit which can be plugged into the existing modem/router. This is called the SamKnows Whitebox.

The network device periodically performs measurements without interfering with actual payload traffic in your network. It monitors Ethernet as well as wireless LAN. The data becomes uploaded, aggregated, and nicely visualized in a personalized dashboard.

If you are paranoid, you probably do not want to have such a device in your home and you might argue that you can measure for yourself. However, I think that this project is trust-able and the statistics obtained by SamKnows are essential for evaluating and developing broadband Internet access in Europe. Furthermore, the data obtained by my Whitebox were already multiple times useful for convincing my ISP that it is not holding up its end of the contract.

I recommend signing up: https://www.samknows.eu/sign-up/

Allen & Heath Xone:23C — hidden technical detail and quirks

The brand new Allen & Heath Xone:23C has been presented in countless preview videos and smaller reviews, all mentioning the main features of this great device. I have just obtained mine. There are some important details to know about, which are not mentioned in the documentation and in the reviews I have found so far. I want to share these non-obvious technical details with you, particularly regarding the USB sound hardware that is built into the Xone:23C.

The Xone:23C is a true 3+3 channel mixer

The Xone:23 (without the C) is a 2+2 channel mixer, with two main channels, whereas each main channel can be fed from two analogue sources that have an independent gain control. Still, Chris Brackley from DJ TechTools found that the Xone:23 has too few input options with only 2 true stereo line inputs, because the other two stereo inputs are made for low voltage (phono) input devices and cannot be switched to line level without hacking the device. The Xone:23C adds two USB audio stereo channels to the mix which can be fed via USB, rendering the Xone:23C to effectively be a 3+3 channel mixer. However, the internet resources available so far and especially the manual do not explain explicitly how to toggle between USB and line/phono sound. The most obvious observation is that there is no switch to toggle.

The circuit diagram at the end of the manual explains the behavior. I have taken a screenshot from the relevant part and labeled a few components:

xone_23c_circuit_diagram_sum_02

The USB audio is processed and converted (to an analogue signal) by the USB soundcard block shown in the diagram. This block has two stereo outputs (send 1+2 and send 3+4). I have marked these two stereo channels with blue arrows. This is where your USB audio starts its way into the mixer after being converted to an analogue signal.

The important thing now are the summing amplifiers which I have labeled with green circles (what looks like an M or W actually is a capital greek sigma, the symbol for summation). The circuit diagram tells us that each main channel mixes the stereo signals from phono, from line, and from USB, in equal parts. Phono and line have their distinct hardware gain controls on the mixer for controlling the mix ratio. Such a control is missing for the USB audio stream. But it is not needed: the volume of the USB audio can easily be controlled digitally, in the source (your computer).

One of the first things I tried when I got the mixer was to attach a line device and USB audio at the same time to the same main channel. Indeed, both audio sources are mixed into each other, and the loudness of the line signal and the USB signal can be set independently. Hence, the Xone:23C is a true 3+3 channel mixer. No need to toggle between line/phono and USB.

Keep the digital master output low enough!

Obviously, I choose to mix externally with the Xone:23C, using the ASIO drivers for transporting the audio signal from within Traktor to the USB soundcard in the mixer. For tracks that are mastered quite loudly, the default master output volume of Traktor is too high, already clamping the signal, and going into the reds on the VU-meter on the mixer. Add some EQ effects or some HPF/LPF with resonance, and your signal becomes horribly distorted. I found that with a Traktor master output volume set to somewhere between -5 dB and -10 dB, the Xone:23C meters stay around 0 dB most of the time for normal parts of most tracks I listened to, whereas the signal increases to at most +6 dB for especially loud parts in a song, or when some effects are added.

If you are using any music player for playing audio on the mixer not through ASIO, but through the normal audio driver of your operating system, I found that a master volume of about 60 % to 70 % is sufficiently low enough for not clamping the signal. If this is set to 100 %, as it usually is, you are already in the reds. Bad.

USB audio from the mixer to the computer.

The USB soundcard in the Xone:23C provides two output stereo channels (from the computer into the mixer) and to input stereo channels (from the mixer into the computer). The usage of the output channels is obvious: get sound into the mixer. Each of the two input channels plays a special role, this information is rather hidden in the manual. The mixer has an analogue stereo RCA record output, for capturing the main mix into an analogue recording device. USB input channels 1 and 2 are the same, just digitally. Hence, you can easily use your computer to record the master output of the Xone:23C, with no additional hardware and through the same USB cable that is connecting the mixer to your computer anyway. This is great.

The mixer also has an analogue effects unit stereo output. USB input channels 3 and 4 are the same, just digitally. Hence, you can use software for capturing this input (e.g. Ableton), generate a corresponding effect output, and feed this one back into the FX input of the Xone:23C. The latter, however, requires additional hardware (another sound device that generates an analogue signal), because there is no dignal FX input.

Recording only works through ASIO so far

There seems to be one caveat with the USB recording function, at least on Windows. The Xone:23C presents a Line-In WDM recording device, for recording the master mix. However, I was not able to access this device with another software simultaneously playing back through ASIO. Playback and recording only seem to work simultaneously through the ASIO interface.

Audacity (and many other popular open source tools) does not support ASIO (ASIO is a proprietary interface and GPL-licensed software must legally not be binary-distributed with ASIO support built-in). On the other hand, Audacity could record through the Xone:23C Line-In WDM device. However, as stated above, this cannot be accessed if e.g. Traktor at the same time feeds the Xone:23C with audio data through ASIO. In other words, Audacity can not be used for recording the master mix through the Xone:23C WDM Line-In device, while having Traktor playback through the Xone:23C ASIO interface. Opening the WDM device in this scenario results in an error, saying that the device cannot be accessed. What works, indeed, is recording via the Xone:23C ASIO driver through e.g. Traktor or other commercial software.

Recording the master mix from within Traktor, however, is not totally straight-forward. One needs to define an external input source for a normal track deck (e.g. deck A). This input source must be the channels 1+2 from the Xone:23C ASIO input. As long as you do not switch deck A to be of type “live deck”, this input effectively is a no-op input (it does not end up in the output again). Now, you can switch to external recording mode, and choose deck A as input source. Don’t worry, deck A still behaves as a normal track deck, it is just mis-used for this workaround.

Issues with playback on one of my platforms

I have tested the Xone:23C’s internal ASIO sound hardware with two laptops. Both have Windows 7 Professional installed. One is 64 bit architecture and operating system, the other is 32 bit architecture and operating system. I have installed the ASIO drivers from here, specifically the 32 bit version for the 32 bit OS/laptop, and the 64 bit version for the 64 bit OS/laptop. On the 64 bit system, the audio chain (playback software -> ASIO driver -> USB audio interface) behaves as expected. On the 32 bit system I have observed infrequent crackling sounds in the output.

The 32 bit system is a fresh and clean Windows operating system install, and the driver is the “Xone:23C Windows 32bit Driver V2.9.65″. I tried different setups, all without success. Important examples that I tried:

  • Foobar audio player to Xone:23C audio WDM device with small and large buffer sizes
  • Traktor 2.6.8 output to Xone:23C ASIO driver, with small and large buffer sizes
  • Traktor 2.6.8 output to ASIO4ALL driver, with small and large buffer sizes

In all cases, the crackling appears and seems to be independent of the buffer size. The crackling is not very prominent, it appears roughly every 10 seconds, and is rather quiet. I tried different USB ports, re-installing the driver, and a couple of other things, but could not get rid of the cracks. The same Xone:23C attached to the 64 bit machine works perfectly. My 32 bit laptop has an Intel P8800 CPU, i.e. it is definitely not too weak, and playback from Foobar right to the WDM device does not require much CPU power at all. It could be a problem with the 32 bit driver (I have submitted a support ticket to A&H), but it could also be a certain quirk of this specific platform, where one of the drivers (e.g. ACPI or USB) is leading to high latencies. I have to further investigate. It would be great if you could report whether you got the Xone:23C USB audio properly working on a 32 bit Windows system.