I just spent hours trying to build numpy 1.8.0 with the most recent Intel MKL, Intel C++ Composer, and Intel Fortran Composer (Version 2013 SP1, from January 2014). numpy built fine, imported fine, and basic math worked fine. But one of the earlier tests (when running numpy.test()) triggered this error:

MKL FATAL ERROR: Cannot load libmkl_mc3.so or libmkl_def.so.

This error message is obviously not generated by the system, but written out by some MKL code. I double-checked that LD_LIBRARY_PATH was set correctly, and that these libraries were actually available. MKL itself seems to be confused about how to access these libraries.

There is not too much to find about this error — the two most important resources being

• An Intel forum thread, where even the Intel engineers are struggling in identifying the problem. The user there solved his issue by accident, there was no logical solution.
• and a StackOverflow answer saying “Intel people claim this was MKL library bug.“

More people have this kind of problem:

• http://software.intel.com/en-us/forums/topic/487928
• https://groups.google.com/forum/#!msg/qutip/7_ug3Oq8wfw/5Ps_kys37BEJ
• http://software.intel.com/en-us/forums/topic/392223

The Intel guys have a general “solution” to this class of problem: http://software.intel.com/en-us/articles/mkl-fatal-error-cannot-load-neither-xxxx-with-intel-mkl-10x — this article provides insights into how things should work, but did not really help in my scenario.

Some of the workarounds that magically solve this issue for some people involve explicit declaration of a list of libraries to link to (via mkl_libs in site.cfg in case of numpy), but that did not help in my case. According to http://software.intel.com/en-us/articles/numpy-scipy-with-mkl and other Intel resources, it should be the best idea to rely on mkl_rt for dynamic dependency resolution, so this is what I wanted to do. And most importantly the mkl_rt method should, according to Intel, prevent this error from happening (reference). This was also propagated in a forum thread in 2010 (here):

“To who’m concerns,

The mkl dynamic library issue will be fixed thoroughly in MKL 10.3 and future version. A new librar mkl_rt was introduced. Now to build the custom MKL library based on dynamic MKL, one can easilyuse the library libmkl_rt.so instead of -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core

The new mkl version will be integrated to next intel Compiler version too, which targeted to be release in Nov.”

Yeah, that’s ironic. I do not want to say that it is impossible to get things running using recent Intel compiler suites, this statement would be wrong. But it is definitely tedious! See https://github.com/GalSim-developers/GalSim/issues/261. One quote from that thread:

Actually, this might be harder than I thought. I googled the error you were having and found a thread that ends with this comment:

“I remember now that I had the same problem recently – it is a
fundamental incompatibility between MKL and Python way of loading shared
libraries through dlopen. AFAIK, there is no solution to this problem,
except for using the static libraries.”

I’m not sure what they mean about using static libraries though, since python usually can’t handle static libraries.

For sure, the conclusion after this bit of research is that this error message is the result of a serious issue, and the direct solution to it is not well documented. There is a simpler way to tackle this:

I went a few steps back and built with the 12.1.3 suite (MKL, icc, ifort). It just works (and suggests that there are actual issues with newer releases of MKL/icc/ifort).

I describe the build process in another blog post here.