A rational explanation for the dress color confusion and hype

Introduction

Wired reported, SPIEGEL ONLINE reported, Adobe contributed, Twitter and Facebook went crazy like hell — it’s like half of the world discusses the colors of a dress today. One thing upfront: no sensation here, at all. The level of “virality” of this story and its scientific impact are not related. Still, there are various important aspects about this topic which I want to bring into order and contribute to the discussion.

I think the most revealing insight of this development is not about color perception — it is about

  • how easily people can get biased (how many of us are not able to grasp an independent thought).
  • how many of us are not able to isolate the actual facts and to neutrally reflect them.
  • how difficult it is to grasp seemingly simple concepts.

Evidently, these points are not new. This is all human, and the natural cure to the listed problems is science. Now I want you to relax, and approach this topic scientifically, to a certain degree at least. In this context it is important to clarify: what does “scientifically” mean? Let me quote Steven D. Schafersman:

Science is a method of discovering reliable knowledge about nature. There are other methods of discovering and learning knowledge about nature, but science is the only method that results in the acquisition of reliable knowledge. Reliable knowledge is knowledge that has a high probability of being true because its veracity has been justified by a reliable method.

This is vague, but this really is the main idea. It is up to us to fulfill this criterion, and obviously this requires some knowledge about how things really work.

Wired misses the point

There are many levels of “science” that can be applied here, and for sure others tried. Wired, for instance, claim that they have taken a scientific approach, but they did not do a good (and complete) job from my point of view: they argue why people have different perceptions, and that is fine, but known. Then they discuss the colors in the image without explaining what it is that the observer of the image actually sees. Eventually, they jump into the following conclusion:

At least we can all agree on one thing: The people who see the dress as white are utterly, completely wrong.

How dare you. Point missed! Providing insights in this topic is not done by telling people how their brain should work. Providing insight is done by telling them what it actually is that enters their eyes. Beyond that boundary there is no binary distinction between right and wrong (nor between black and white, pun).

So, the Wired article on the topic does not construct a convincing chain of causality. The one thing they got right is that colors are interpreted by brain depending on the environment.

Two concepts got mixed up, constructively interfering towards a super-magic phenomenon

Colors are interpreted by the brain, that is nothing new. Don’t get me wrong, this is an impressive and super-interesting fact. From a neuroscientific point of view, there is endless deepness in this fact: this is an interface allowing to investigate how parts of human brain work. This can be considered “magic”, because we still do not know exactly how the brain works in this regard. This is concept one:

1. Colors are interpreted by brain, depending on environment. Not fully understandable, but appreciable.

And then the dress story came, suggesting the following concept to people:

2. The dress is either white/gold or blue/black. One of these solutions is correct. The correct solution can be inferred from the photograph. Some people see the correct solution, others are wrong.

This is wrong and problematic in different regards, as I will explain this further below. The dress story got so much attention, largely because these two concepts were mixed. One of the concept obviously is right. The other concept is, unfortunately, not obviously wrong to many people. In combination, both concepts suggest even more magic than there really is. So the debate got heated up.

We do not need the Wired article cited above to explain in many words how the first concept works. It is XKCD who does the perfect job of punching this very fact right into our faces:

XKCD nails (at least a part of) it

http://xkcd.com/1492/

http://xkcd.com/1492/

The RGB colors of both dresses are the same. The effect (that the perception depends on the environment) is obvious. This was also obvious to Randall Munroe, the genius behind XKCD. And, I suppose, it was not this very obvious fact why he created the cartoon. I think the most important message of his cartoon number 1492 is, as so often, hidden in the image title on xkcd.com/1492 (seen only when keeping the mouse pointer a while on top of the image):

This white-balance illusion hit so hard because it felt like someone had been playing through the Monty Hall scenario and opened their chosen door, only to find there was unexpectedly disagreement over whether the thing they'd revealed was a goat or a car.

Please, chew on that, for a minute. Randall suggests the following scenario: What is behind the door? Car or goat, car or goat, car or goat, car or goat? Now you open the door. And you might say: oh, I am not sure whether this is it a car or a goat. And you think you now have to decide between both, because car is great, and goat is bad and everything else is not important. You have lost capability to see and realize that it is neither of both. People are so focused on seeing either of car or goat that they lose their ability to judge neutrally what they see.

This is Randall’s way of criticizing how people are asked about this very dress issue and about how media fails in attempting to resolve this issue. Media is creating a bias in this discussion, which clearly prevents a neutral analysis. In fact, what happened is a phenomenon that can be described as social self-amplification, converging towards a tremendous simplification of the problem, and towards the wrong questions being asked, eventually yielding the seemingly only viable options of white/gold vs. blue/black.

The essential ingredients for understanding the issue

Rollback. By now it is clear that we should forget about white/gold vs. blue/black, and focus on what we actually see. This might read trivial, but is the primary insight for treating this topic on a scientific level.

What we actually see is, from my “scientific” experience, built of four major components:

  • the working principle of a camera for taking a photograph
  • the flow of information from the camera towards a digital picture
  • the flow of information from the digital picture over the screen into your eyes
  • the flow of information within your brain

Hence, there are multiple levels of information and information flow involved. The actual color information in the picture is just a part of the “global picture”. However, it is the global picture which one needs to have in mind when one wants to rationally grasp WTF this is about. So, when Adobe tweeted

For those seeing #WhiteandGold in #TheDress (http://bit.ly/1APDFay ), @HopeTaylorPhoto ends the debate.

they made a fool of themselves and approached the problem at a wrong level, not looking at the entire flow of information.

The essential concepts for assessing the dress color problem, distributed over four levels of information flow, are not that complex and shall be discussed within the next few paragraphs.

The role of camera and digital modification reduces the problem

Depending on the exposure and white balance settings, a camera records very different images of any given scene. You do not need to exactly understand what these two things mean. You just need to appreciate that color and brightness as recorded by the camera might be very different from what a human being would have observed in position of the camera. Simple, right? Then, once the image has landed on a computer, it can be edited. In any way. The first two points from the list above boil down to the following conclusion: a photograph that you did not take yourself and that you did not digitally edit yourself might show anything far from what the actual, real scene looked like to humans. Still simple, so far.

I think the most important conclusion from here is: based on a random image circulating the Internet, it is pointless to discuss which color the dress really has. This might sound nitpicky to you, but in fact this relaxes the problem a lot. The important insight from here is that asking for the real color of the dress is an ill-posed problem. It is a waste of time to even think about the real color of the dress. A valid question left is: what color does the dress have in the image?

Take the colors for granted, just for a moment

I fear that my last point might still might appear a little nitpicky. But it is essential, so I’ll use a second approach: there is no point in interpreting this photograph with respect to the real lightning. The flow of information from the real scene towards the digital photograph implicates a loss of information. This is irreversible. Furthermore, the information that is contained in the image underwent transformations whose details we do not know (remember: we did not take and edit this photo by ourselves). Hence, the remaining information can not be used to make resilient conclusions about the lightning in the real scene.

The point is: if you want to approach this scientifically, forget about the real scene, really. I mean, you can do whatever you want, but if you incorporate the real scene of the dress photograph in your thought process, then you are making unsupported assumptions and leave the scientific regime.

When one follows this logic, the only valid way to answer this question (“what color does the dress have in the image?”) is to quantify the colors in the image. Quantification usually happens through “measurement”, which is an important concept in science. During a measurement, a certain quantity is determined. The measurement yields a number, and the number has common units. Measuring should exclude experimental errors as far as possible. Naturally, a measurement transforms one type of information into another. Now, you might just use a calibrated screen and a calibrated camera and then display the image on the screen and then point the camera to the screen and then measure colors. Or you stop over-interpreting this very simple problem and realize that in this case there is no need to transform information another time. The information we are interested in (the colors in the image) are already there. In the digital image file. Just read them out. There is no point in performing another measurement.

So, now we look at the colors encoded in the digital image file. Others have done this for us. This is one of the most comprehensive results I have found (in a Reddit discussion):

dress-colors-categorized

Given the dress photograph as data source, we now have (scientifically) obtained a conclusion (you might want to name it a “fact”): The two main colors of the dress in the image are blueish and brownish (the RGB unit system is precise, its transformation to “words” at least follows a reproducible system, by neutral standards).

The last step of information flow: error-prone extrapolation

The last piece of information flow is within the brain of the one observing the image. Clearly, there is an interesting relation between the distribution of colors in the image (not in the scene, in the image), and human perception/imagination of the two main colors of the dress in the real scene. I appreciate that when forced to imagine how the dress might have looked like in reality, certain people argue that they think it was white/golden or blue/black. Fine with me, and I believe them: Clearly, this particular image has a weird light constellation, so that it requires a huge amount of extrapolation towards imagining the real lightning in the real scene. A huge amount of extrapolation based on bad data yields an unstable result, even when using a computer and a numerical approach. Now, this is brain. Much more complex, many more degrees of freedom and clearly a larger potential to end up with different outcomes when extrapolating. Same person, different attempts, different outcomes. Different persons, different outcomes. Really, this is not surprising, and there is not right or wrong, because the underlying data is bad.

I asked myself: which colors do I see in the picture? Honestly, “blueish” and “brownish” were the colors I saw. I actively denied to think further than that, because I knew that it does not make sense to over-interpret that picture. Because so much information was lost on the way from the real scene towards the image on my screen. Because, obviously, this is a bad photograph. You know that when you work with photographs.

Wrapping up

The discussed confusion is due to a “real” quantity (something that is physically measurable) being measured under unclear conditions, manipulated to an unknown amount, and then interpreted by human brain. The interpretation may differ from person to person, depending on the question and depending on how the person trained his/her brain (without knowing) throughout life. As clarified in the article, there is no right or wrong about what people say what they see here. It is just that every honest answer to the question “Which colors does the dress have in the image” is the result of either

  • a neutral color analysis based on what is in the image (without extrapolation)
  • or a complex extrapolation thought process (subconscious or not) with the goal to identify the real scene lightning.

The latter is, as argued, an ill-posed problem and people are weighting different aspects of this thought process differently, which is why the outcome is different. Media turned this whole story into something that looks super magical, because asking whether he/she thinks if the dress is white/gold or blue/black and suggesting that only one solution is correct is very different from asking “which colors do you see?” and manipulates people.

How to set up a 64 bit version of NumPy on Windows

A short note on a complex topic. Feel free to shoot questions at me in the comments.

There are no official NumPy 64 bit builds available for Windows. In fact, 64 bit Windows is not officially supported by NumPy. So, if you are serious about your project, you need to either consider building on top of Unix-like platforms and inherit external quality assurance, or (on Windows) you need to anticipate issues of various kinds, and do extensive testing on your own. One of the reasons is that there is no adequate (open source, reliable, feature-rich) tool chain for creating proper 64 bit builds of NumPy in Windows (further references: numpy mailing list thread, Intel forums). Nevertheless, in many cases a working solution are the non-official builds provided by Christoph Gohlke, created with Intel’s commercial compiler suite. It is up to you to understand the license impacts and whether you want or can use these builds. I love to use these builds.

The following steps show a very simple way to get NumPy binaries for the AMD64 architecture installed on top of CPython 3(.4). These instructions are valid only for Python installed with an official CPython installer, obtained from python.org.

1) Install CPython for AMD64 arch

Download a 64 bit MSI installer file from python.org. The crucial step is to get an installer for the AMD64 (x86-64) architecture, usually called “Windows x86-64 MSI installer”. I have chosen python-3.4.2.amd64.msi. Run the setup.

2) Upgrade pip

Recent versions of Python 3 ship with pip, but you should use the newest version for proper wheel support. Open cmd.exe, and run

C:\> pip install pip --upgrade

Verify:

C:\> pip --version
pip 6.0.8 from C:\Python34\lib\site-packages (python 3.4)

The latter verifies that this pip i) is up-to-date, and ii) belongs to our target CPython version (multiple versions of CPython can be installed on any given system, and the correspondence between pip and a certain Python build is sometimes not obvious).

Note: The CPython installer should properly adjust your PATH environment variable so that python as well as pip entered at the command line correspond to what has been installed by the installer. It is however possible that you have somehow lost control of your environment by installing too many different things in an unreasonable order. In that case, you might have to manually adjust your PATH so that it priorizes the exetuables in C:\Python34\Scripts (or wherever you have installed your 64 bit Python version to).

3) Download wheel of NumPy build for AMD64 on Windows

Navigate to lfd.uci.edu/~gohlke/pythonlibs/#numpy and select a build for your Python version and for AMD64. I chose numpy‑1.9.2rc1+mkl‑cp34‑none‑win_amd64.whl.

4) Install the wheel via pip

On the command line, navigate to the directory where you have downloaded the wheel file to. Install it:

C:\Users\user\Desktop>pip install "numpy-1.9.2rc1+mkl-cp34-none-win_amd64.whl"
Processing c:\users\user\desktop\numpy-1.9.2rc1+mkl-cp34-none-win_amd64.whl
Installing collected packages: numpy
 
Successfully installed numpy-1.9.2rc1

The simplicity of this approach is kind of new. Actually, this simplicity is why wheels have been designed in the first place! Installing pre-built binaries with pip has not been possible with the “old” egg package format. So, older tutorials/descriptions of this kind might point to MSI installers or dubious self-extracting installers. These times are over now, and this is also is the major reason why I am writing this blog post.

5) Verify

>>> import numpy
>>> numpy.__version__
'1.9.2rc1'

Great.

Third-party Python distributions

I do not want to leave unmentioned that out there are very nice third party Python distributions (i.e. not provided by the Python Software Foundation) that include commercially supported and properly tested NumPy/SciPy builds for 64 bit Windows platforms. Most of these third party vendors have a commercial background, and dictate their own licenses with respect to the usage of their Python distribution. For non-commercial purposes, most of them can be used for free. The following distributions provide a working solution:

All of these three distributions are recommendable from a technical point of view (I cannot tell whether their license models / restrictions are an issue for you or not). They all come as 64 bit builds. I am not entirely sure if Enthought and ActiveState build NumPy against Intel’s Math Kernel Library. In case of Anaconda, this definitely is not the case in the free version — this is something that can be explicitly obtained, for 29 $ (it’s called the “MKL Optimizations” package).

Gibbs’ foundation of the canonical ensemble

I enjoy digging in historical physics publications, and want to share a document with you that might greatly simplify putting the concepts of statistical mechanics into context. Gibbs himself did an enlightening job in providing this context, in the preface to his essay titled “Elementary principles in statistical mechanics — the rational foundation of thermodynamics”, published in 1902. It is the very same essay that coined the term “canonical ensemble”, so if you ever wondered since when this term is actually used and who has defined it for the first time, this essay will be quite insightful for you.

gibbs_stat_mech_1902_title

Thanks to Google’s literature scanning program, named essay is freely available (not subject to copyright restrictions) on archive.org. According to the download counter, it has been viewed/downloaded only 540 times (at the time of writing this blog post), which is not a whole lot compared to the importance of the document. It can be obtained in various file formats, including PDF.

I recommend giving the preface a read. What follows now are a couple of quotations that I found to be a linear combination of enlightening and funny.

In the very beginning, Gibbs argues that thermodynamics is adjusted to the human capabilities of observation:

The laws of thermodynamics, as empirically determined, express the approximate and probable behavior of systems of a great number of particles, or, more precisely, they express the laws of mechanics for such systems as they appear to beings who have not the fineness of perception to enable them to appreciate quantities of the order of magnitude of those which relate to single particles, and who cannot repeat their experiments often enough to obtain any but the most probable results.

Also, Gibbs humbly appreciates the difficulty and complexity of describing natural phenomenons in accurate terms and resigns from making attempts in this direction. Instead, he proposes to change perspective, and praises the conceptual advantage of working on a theory which does not claim to have predictive capabilities. He stresses the beauty and simplicity of a self-consistent theory, and warns about introducing bias:

Difficulties of this kind have deterred the author from attempting to explain the mysteries of nature, and have forced him to be contented with the more modest aim of deducing some of the more obvious propositions relating to the statistical branch of mechanics. Here, there can be no mistake in regard to the agreement of the hypotheses with the facts of nature, for nothing is assumed in that respect. The only error into which one can fall, is the want of agreement between the premises and the conclusions, and this, with care, one may hope, in the main, to avoid.

Gibbs proceeds introducing the single chapters of his essay, and eventually comes up with the “canonical ensemble” terminus:

In the fourth and following chapters we return to the consideration of statistical equilibrium, and confine our attention to conservative systems. We consider especially ensembles of systems in which the index (or logarithm) of probability of phase is a linear function of the energy. This distribution, on account of its unique importance in the theory of statistical equilibrium, I have ventured to call canonical, and the divisor of energy, the modulus of distribution. The moduli of ensembles have properties analogous to temperature, in that equality of the moduli is a condition of equilibrium with respect to exchange of energy, when such exchange is made possible.

Gibbs stresses that a major outcome of the self-consistent statistical mechanics framework is that under certain conditions expectation values of ensemble properties may yield the thermodynamic quantities temperature and entropy:

We meet with other quantities, in the development of the subject, which, when the number of degrees of freedom is very great, coincide sensibly with the modulus, and with the average index of probability, taken negatively, in a canonical ensemble, and which, therefore, may also be regarded as corresponding to temperature and entropy. […] In Chapter XIV, this subject of thermodynamic analogies is discussed somewhat at length.

Gibbs concludes his preface with a sharp distinction between the theoretical concepts of statistical mechanics and thermodynamics on the one hand, and the properties of matter on the other hand. That one really is cryptic and beautiful at the same time:

Finally, in Chapter XV, we consider the modification of the preceding results which is necessary when we consider systems composed of a number of entirely similar particles, or, it may be, of a number of particles of several kinds, all of each kind being entirely similar to each other, and when one of the variations to be considered is that of the numbers of the particles of the various kinds which are contained in a system. This supposition would naturally have been introduced earlier, if our object had been simply the expression of the laws of nature. It seemed desirable, however, to separate sharply the purely thermodynamic laws from those special modifications which belong rather to the theory of the properties of matter.

Reading such texts — communicating nothing but the pure intent of the original creators — provides a perspective that may greatly simplify understanding and applying difficult concepts.

WP-GeSHi-Highlight 1.2.3 released

I have released version 1.2.3 of WP-GeSHi-Highlight. WP-GeSHi-Highlight is a popular code syntax highlighting plugin for WordPress, based on the established PHP highlighting library GeSHi.

The only change in this realease:

  • The bundled GeSHi library was updated to version 1.0.8.12.

So far, there are no official release notes for GeSHi 1.0.8.12. However, the commit history clarifies that this update involves various language file improvements as well as newcomers. A list of those languages that are affected by the update (manually crafted):

  • PHP
  • Rust
  • TCL
  • CSS
  • Haskell
  • PostScript
  • QML
  • NSIS
  • Nimrod
  • Oxygene
  • LSL2
  • StandardML
  • Oxygene

With this update in place, WP-GeSHi-Highlight supports syntax highlighting for 240 languages.

Documentation, FAQ, and comments can be found at gehrcke.de/wp-geshi-highlight.

The most popular (serious) country code top-level domain still is .DE

To my surprise and according to DENIC, .DE was still the most popular (serious*) country code top-level domain (ccTLD) by December 2014:

http://www.denic.de/en/background/statistics/international-domain-statistics/

Popularity of top-level domains by December 2014. Image source: DENIC.

Right, .DE ranks only #3 in that graph. However, the popular .COM is not a country code top-level domain — although widespread in the U.S., it has no geographical implications. Furthermore, while .TK technically is a country code domain, it is very different from regular ones as discussed by the end of this article.

Rising China

Surprisingly, .CN (China’s ccTLD) still did not overtake Germany’s .DE: In a 2013 report, CENTR (the European country code TLD organisation) revealed an especially large growth for .CN within six months only:

ccTLD growth rate

Growth of ccTLD popularity within six months in 2012/2013. Image source: CENTR report 2013.

This growth rate lead to .CN actually overtake .UK at some point within 2014, as can be inferred from the May 2013 and the Sept 2014 reports by CENTR:

.CN overtook .UK

.CN overtook .UK by 2014. Source: CENTR

Keeping that pace, it would not take long until .CN overtakes .DE, too. But does the popularity of .CN constantly grow that fast? No. According to this report, it grew by only 1.8 % between May and August 2014. From these data it can be concluded that .CN’s growth rate underlies strong fluctuations. It is still significantly larger than DE’s growth rate, but according to DENIC the absolute difference between .DE and .CN domain registrations was still about 4.5 million by the end of 2014. That is, .CN probably will overtake .DE, but it will rather happen within the next couple of years than tomorrow.

Liechtenstein: even more domains than letter boxes?

An interesting fact: normalized by population, Liechtenstein holds a record: it has 1.76 ccTLD’s registered per citizen, compared to 0.19 in case of Germany (CENTR report from September 2014).

Why .TK is not comparable to .DE, .UK, and .CN (*)

From the figures you inferred that .TK is the largest country-related zone with about 26 million reported domain names as of 2014. Quite a phenomenon you should know about! This domain is based on a fundamentally different business model than traditional domains such as .DE. I have two quotes for you, describing the situation:

DENIC’s statement:

.tk is the country-code domain of Tokelau, a small group of islands and a territory of New Zealand. Domains under .tk are offered free of charge for an initial period. Once expired, they are used for advertising purposes by the registry operator. […] No clear geographical relationship exists between .tk domains and the territory of Tokelau.

Comlaude’s statement:

Tokelau […] is a territory of New Zealand in the South Pacific Ocean consisting of three atolls with a combined land area of 10 km^2. The owner of all free .tk domains is the registry. If you want to own the domain, you must pay. Pricing is from $6.95 per year with a minimum two-year registration period; although domains containing one, two or three characters or those with branded, generic or common words are considered premium and are priced higher. As a result, one sixth of the Tokelau economy consists of income from .tk domain names.

Long-term development towards steady-state

In another extensive report from 2013, CENTR states that the growth rate of country code top-level domain names has halved between 2008 and 2013. That is, the Internet is slowly progressing towards saturation and steady-state with respect to domain names:

The average increase in the latter half of 2013 of 264,000 domains per month is half the average increase of 528,000 evidenced in the first part of 2008.

Interesting: most of the data used in CENTR reports seems to stem from a single source — a commercial “provider of data and analysis for the domain name industry”: zooknic.com

Are you aware of additional interesting insights derived from the development of top-level domain popularity?