Compile NumPy with ifort, MSVC and MKL - DLL load failed

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Compile NumPy with ifort, MSVC and MKL - DLL load failed

Pietro Fontana

Hi all,

I've been trying to compile NumPy from source on Windows 10, with MSVC compiler and Intel MKL. Whenever I link to MKL it fails at loading DLLs.
I am running Windows 10.0.18363 with Microsoft Visual Studio 2019 (16.8.5) and Intel MKL 2017.8.275.

I managed to reproduce the issue with a minimal setup, using latest Python and NumPy.

  1. Download latest Python (3.9.1) and latest NumPy (1.20.1) source.
  2. Open a VS command prompt, unpack Python source, build with PCbuild\build.bat
  3. Run mklvars.bat intel64 to get the right environment variables set.
  4. Add the Intel compilers (needed for ifort) to PATH:

set PATH=C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017\windows\bin\intel64;%PATH%

  1. Create a virtual env, copy a few files from the Python build and activate the virtual env:

copy Python\PCbuild\amd64\python39.dll venv\Scripts
copy Python\PC\pyconfig.h venv\Include

  1. Build NumPy from source and install: pip install . -v
  2. Try to import NumPy: python -c "import numpy"

The error message appears as follows:

Traceback (most recent call last):
  File "C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
    from . import multiarray
  File "C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
    from . import overrides
  File "C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: DLL load failed while importing _multiarray_umath: The specified module could not be found.
 
During handling of the above exception, another exception occurred:
 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\path\numpy_clean_env\venv\lib\site-packages\numpy\__init__.py", line 145, in <module>
    from . import core
  File "C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
    raise ImportError(msg)
ImportError:
[… useful suggestions that however did not lead to a solution…]
Original error was: DLL load failed while importing _multiarray_umath: The specified module could not be found.
 

The MKL libraries are picked up during compilation since it returns:

FOUND:
        libraries = ['mkl_rt']
        library_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib\\intel64']
        define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
        include_dirs = ['C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\include', 'C:\\Program Files (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib']

 

I tried to analyze the DLL resolution on _multiarray_umath.pyd with Dependencies (the newer version of Dependency Walker) but it seems that the MKL DLL loads fine. There are some DLLs that appear as not correctly loaded, but as far as I understand it is caused by the inspection software limit with Windows API sets (api-ms-win-core-*, ext-ms-onecore-*, ext-ms-win-*, and similar), not by actual problems with this DLLs, so I think the system is correctly setup.

 

If I skip the initialization of MKL environment variables, then the MKL libraries are not picked and NumPy is compiled to a functional state.

In the past this setup used to work with Python 3.6, VS2015 and a similar version of Intel MKL.
I was able to reproduce the issue with NumPy 1.16.2, 1.17 and 1.20.1; with Python 3.8.6 and Python 3.9.1; with Intel MKL 2017 and oneAPI 2020.

Am I missing any obvious step to succeed in this adventure?

 


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compile NumPy with ifort, MSVC and MKL - DLL load failed

Christoph Gohlke
Hello,

On 2/15/2021 7:38 AM, Pietro Fontana wrote:

> Hi all,
>
> I've been trying to compile NumPy from source on Windows 10, with MSVC
> compiler and Intel MKL. Whenever I link to MKL it fails at loading DLLs.
> I am running Windows 10.0.18363 with Microsoft Visual Studio 2019
> (16.8.5) and Intel MKL 2017.8.275.
>
> I managed to reproduce the issue with a minimal setup, using latest
> Python and NumPy.
>
>  1. Download latest Python (3.9.1) and latest NumPy (1.20.1) source.
>  2. Open a VS command prompt, unpack Python source, build with
>     PCbuild\build.bat
>  3. Run mklvars.bat intel64 to get the right environment variables set.
>  4. Add the Intel compilers (needed for ifort) to PATH:
>
> set PATH=C:\Program Files
> (x86)\IntelSWTools\compilers_and_libraries_2017\windows\bin\intel64;%PATH%
>
>  5. Create a virtual env, copy a few files from the Python build and
>     activate the virtual env:
>
> copy Python\PCbuild\amd64\python39.dll venv\Scripts
> copy Python\PC\pyconfig.h venv\Include
>
>  6. Build NumPy from source and install: pip install . -v
>  7. Try to import NumPy: python -c "import numpy"
>
> The error message appears as follows:
>
> |Traceback (most recent |call||last|):|
>
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py"|, line
> |22|, |in|<|module|>|
>
> ||from|. |import|multiarray|
>
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\multiarray.py"|,
> line |12|, |in|<|module|>|
>
> ||from|. |import|overrides|
>
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\overrides.py"|,
> line |7|, |in|<|module|>|
>
> ||from|numpy.core._multiarray_umath |import|(|
>
> |ImportError: DLL |load||failed||while|importing _multiarray_umath: The specified |module|could |not|be found.|
>
> ||
>
> |During handling |of|the above |exception|, another |exception|occurred:|
>
> ||
>
> |Traceback (most recent |call||last|):|
>
> ||File||"<string>"|, line |1|, |in|<|module|>|
>
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\__init__.py"|, line |145|, |in|<|module|>|
>
> ||from|. |import|core|
>
> ||File||"C:\path\numpy_clean_env\venv\lib\site-packages\numpy\core\__init__.py"|, line
> |48|, |in|<|module|>|
>
> ||raise|ImportError(msg)|
>
> |ImportError:|
>
> |[… useful suggestions that however did not lead to a solution…]|
>
> |Original |error|was: DLL |load||failed||while|importing _multiarray_umath: The specified |module|could |not|be found.|
>
>  
>
> The MKL libraries are picked up during compilation since it returns:
>
> |FOUND:|
>
> |        libraries = [|'mkl_rt'|]|
>
> |        library_dirs = [|'C:\\Program Files
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib\\intel64'|]|
>
> |        define_macros = [(|'SCIPY_MKL_H'|, |None|), (|'HAVE_CBLAS'|, |None|)]|
>
> |        include_dirs = [|'C:\\Program Files
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl'|, |'C:\\Program Files
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\include'|, |'C:\\Program Files
> (x86)\\IntelSWTools\\compilers_and_libraries\\windows\\mkl\\lib'|]|
>
> I tried to analyze the DLL resolution on |_multiarray_umath.pyd| with
> Dependencies (the newer version of Dependency Walker) but it seems that
> the MKL DLL loads fine. There are some DLLs that appear as not correctly
> loaded, but as far as I understand it is caused by the inspection
> software limit with Windows API sets (api-ms-win-core-*,
> ext-ms-onecore-*, ext-ms-win-*, and similar), not by actual problems
> with this DLLs, so I think the system is correctly setup.
>
> If I skip the initialization of MKL environment variables, then the MKL
> libraries are not picked and NumPy is compiled to a functional state.
>
> In the past this setup used to work with Python 3.6, VS2015 and a
> similar version of Intel MKL.
> I was able to reproduce the issue with NumPy 1.16.2, 1.17 and 1.20.1;
> with Python 3.8.6 and Python 3.9.1; with Intel MKL 2017 and oneAPI 2020.
>
> Am I missing any obvious step to succeed in this adventure?
>
>

Python >= 3.8 will no longer use PATH for resolving dependencies of
extension modules. Use os.add_dll_directory(mkl_bin_path)
<https://docs.python.org/3/library/os.html#os.add_dll_directory> in all
your scripts before importing numpy or add the call to a
_distributor_init.py file in the numpy package directory.

Christoph
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compile NumPy with ifort, MSVC and MKL - DLL load failed

Pietro Fontana
Hi, thank you very much for pointing me at this. I managed not to find this bit of information despite spending quite some time on the issue.

Cheers, Pietro
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion