You have probably heard about this major security issue in GnuTLS, publicly announced on March 3, 2014, with the following words in a patch note on the GnuTLS mailinglist:

This fixes is an important (and at the same time embarrassing) bug
discovered during an audit for Red Hat. Everyone is urged to upgrade.

The official security advisory describes the issue in these general terms:

A vulnerability was discovered that affects the certificate verification functions of all gnutls versions. A specially crafted certificate could bypass certificate validation checks. The vulnerability was discovered during an audit of GnuTLS for Red Hat.

Obviously, media and tech bloggers pointed out the significance of this issue. If you are interested in some technical detail, I would like to recommend a well-written article on LWN on the topic: A longstanding GnuTLS certificate validation botch. As it turns out, the bug was introduced by a code change that re-factored the error/success communication between functions. Eventually, spoken generally, the problem is that two communication partners went out of sync: when the sender sent ‘Careful, error!’, the recipient actually understood ‘Cool, success.’. Bah. We are used to modern, test-driven development culture. Consequently, most of us immediately think “WTF, don’t they test their code?”.

An automated test suite should have immediately spotted that invalid commit, right. But wait a second, that malicious commit was pushed in year 2000, the language we are talking about is C, and unit testing for C is not exactly established. Given that — did you really, honestly, expect a C code base that reaches back more than a decade to be under surveillance of ideal unit-tests, by modern standards? No? Me neither (although I would have expected a security-relevant library such as GnuTLS to be under a significant 3rd party test coverage — does everybody trust the authors?).

We seem to excuse or at least acknowledge and tolerate that old system software written in C is not well-tested by modern standards of test-driven development. For sure, there is modern software out there applying ideal testing strategies — but having only a few users. At the same time old software is circulating, used by millions, but not applying modern testing strategies. But why is that? And should we tolerate this? There was an interesting discussion about this topic, right underneath the above-mentioned LWN article. I’d like to quote one comment that I particularly agree to, although it is mostly asking questions than providing answers:

> In addition to the culture of limited testing you alluded to,
> I think there are some language issues here as well

Yes, true. But I wonder if discussing type systems is also a
distraction from the more pressing issue here? After all, even
with all the help of Haskell’s type system, you *will* still
have bugs.

It seems to me that the lack of rigorous testing was:
(a) The most immediate cause of these bugs
(b) More common in projects written in C

I find it frustrating that discussions of these issues continually
drift towards language wars, rather than towards modern ideas about
unit testing, software composability, test-driven development, and
code coverage tracking.

Aren’t these the more pressing questions?
(1) Where are the GnuTLS unit tests, so I can review and add more?
(2) Where is the new regression test covering this bug?
(3) What is the command to run a code coverage tool on the test
suite, so that I can see what coverage is missing?

Say what you will about “toy” languages, but that is what would
happen in any halfway mature Ruby or Python or Javascript project,
and I’m happy to provide links to back that up.

Say what you will about the non-systems languages on the JVM, but
that is also what would happen in any halfway mature Scala, Java,
or Clojure project.

It’s only in C, the systems language in which so many of these
vital libraries are written, that this is not the case. Isn’t it
time to ask why?

Someone answered, and I think this view makes sense:

For example, I suspect that the reason “C culture” seems impervious to adopting the lessons of test-driven development has a lot to do with the masses of developers who are interested in it, by following your advice, are moving to other languages and practicing it there.

In other words, by complecting the issue of unit testing and test coverage with the choice of language, are we not actively *contributing* to the continuing absence of these ideas from C culture, and thus from the bulk of our existing systems?

Food for thought, at least, I hope!

I agree: the effort for improved testing of old, but essential, C libraries must come from the open source community. Someone has to do it.