Compiling NumPy on Windows for Python 3.3 with MSVC 2010

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Peter Cock
Changing title to reflect the fact this thread is now about using
the Microsoft compiler rather than mingw32 as in the old thread.

On Sat, Nov 10, 2012 at 11:04 PM, Peter Cock <[hidden email]> wrote:

> On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <[hidden email]> wrote:
>>
>>
>>
>> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <[hidden email]>
>> wrote:
>>>
>>> Dear all,
>>>
>>> Since the NumPy 1.7.0b2 release didn't include a Windows
>>> (32 bit) installer for Python 3.3, I am considering compiling it
>>> myself for local testing. What compiler is recommended?
>>
>>
>> Either MSVC or MinGW 3.4.5. For the latter see
>> https://github.com/certik/numpy-vendor
>
> Thanks Ralf,
>
> I was trying with MSVC 9.0 installed, but got this cryptic error:
>
>    C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build
>    ...
>    error: Unable to find vcvarsall.bat
>
> After sprinkling distutils with debug statements, I found it was
> looking for MSVC v10 (not numpy's fault but the error is most
> unhelpful).
>
> Presumably "Microsoft Visual C++ 2010 Express Edition" is
> the appropriate thing to download?
> http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express
>

I would have tried this earlier, but it required Windows XP SP3
and there was something amiss with the permissions on my
work Windows machine preventing that update. Solved now,
and this file now exists:

C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat

I've tried my branch (focussed on mingw32 fixes), and numpy-1.7.0b2
but both fail at the same point. I did remove the old build directory first.

C:\Downloads\numpy-1.7.0b2>c:\python33\python setup.py build
Converting to Python3 via 2to3...
<snip>
Could not locate executable efc
don't know how to compile Fortran code on platform 'nt'
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo
/Ox /MD /W3 /GS- /DNDEBUG -Inumpy\core\src\private -Inumpy\core\src
-Inumpy\core -Inumpy\
core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath
-Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include
-Ic:\python33\include /Tc_configtest.c /Fo_configtest.obj
Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /nologo
/INCREMENTAL:NO _configtest.obj /OUT:_configtest.exe
/MANIFESTFILE:_configtest.exe.manifest
Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
mt.exe -nologo -manifest _configtest.exe.manifest
-outputresource:_configtest.exe;1
Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe

_configtest.exe.manifest : general error c1010070: Failed to load and
parse the manifest. The system cannot find the file specified.

failure.
removing: _configtest.c _configtest.obj
Traceback (most recent call last):
  File "setup.py", line 214, in <module>
    setup_package()
  File "setup.py", line 207, in setup_package
    configuration=configuration )
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186
, in setup
    return old_setup(**new_attr)
  File "c:\python33\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
    self.run_command(cmd)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py",
 line 37, in run
    old_build.run(self)
  File "c:\python33\lib\distutils\command\build.py", line 126, in run
    self.run_command(cmd_name)
  File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "c:\python33\lib\distutils\dist.py", line 936, in run_command
    cmd_obj.run()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 152, in run
    self.build_sources()
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 163, in build_sources
    self.build_library_sources(*libname_info)
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 298, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
py", line 385, in generate_sources
    source = func(extension, build_dir)
  File "numpy\core\setup.py", line 648, in get_mathlib_info
    raise RuntimeError("Broken toolchain: cannot link a simple C program")
RuntimeError: Broken toolchain: cannot link a simple C program

It appears a similar issue was raised before:
http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html

Any tips?

Peter
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Christoph Gohlke
On 11/14/2012 10:10 AM, Peter Cock wrote:

> Changing title to reflect the fact this thread is now about using
> the Microsoft compiler rather than mingw32 as in the old thread.
>
> On Sat, Nov 10, 2012 at 11:04 PM, Peter Cock <[hidden email]> wrote:
>> On Sat, Nov 10, 2012 at 5:47 PM, Ralf Gommers <[hidden email]> wrote:
>>>
>>>
>>>
>>> On Tue, Nov 6, 2012 at 6:49 PM, Peter Cock <[hidden email]>
>>> wrote:
>>>>
>>>> Dear all,
>>>>
>>>> Since the NumPy 1.7.0b2 release didn't include a Windows
>>>> (32 bit) installer for Python 3.3, I am considering compiling it
>>>> myself for local testing. What compiler is recommended?
>>>
>>>
>>> Either MSVC or MinGW 3.4.5. For the latter see
>>> https://github.com/certik/numpy-vendor
>>
>> Thanks Ralf,
>>
>> I was trying with MSVC 9.0 installed, but got this cryptic error:
>>
>>     C:\Downloads\numpy-1.7.0b2 > C:\python33\python setup.py build
>>     ...
>>     error: Unable to find vcvarsall.bat
>>
>> After sprinkling distutils with debug statements, I found it was
>> looking for MSVC v10 (not numpy's fault but the error is most
>> unhelpful).
>>
>> Presumably "Microsoft Visual C++ 2010 Express Edition" is
>> the appropriate thing to download?
>> http://www.microsoft.com/visualstudio/eng/downloads#d-2010-express
>>
>
> I would have tried this earlier, but it required Windows XP SP3
> and there was something amiss with the permissions on my
> work Windows machine preventing that update. Solved now,
> and this file now exists:
>
> C:\Program Files\Microsoft Visual Studio 10.0\VC\vcvarsall.bat
>
> I've tried my branch (focussed on mingw32 fixes), and numpy-1.7.0b2
> but both fail at the same point. I did remove the old build directory first.
>
> C:\Downloads\numpy-1.7.0b2>c:\python33\python setup.py build
> Converting to Python3 via 2to3...
> <snip>
> Could not locate executable efc
> don't know how to compile Fortran code on platform 'nt'
> C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo
> /Ox /MD /W3 /GS- /DNDEBUG -Inumpy\core\src\private -Inumpy\core\src
> -Inumpy\core -Inumpy\
> core\src\npymath -Inumpy\core\src\multiarray -Inumpy\core\src\umath
> -Inumpy\core\src\npysort -Inumpy\core\include -Ic:\python33\include
> -Ic:\python33\include /Tc_configtest.c /Fo_configtest.obj
> Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
> C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /nologo
> /INCREMENTAL:NO _configtest.obj /OUT:_configtest.exe
> /MANIFESTFILE:_configtest.exe.manifest
> Found executable C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
> mt.exe -nologo -manifest _configtest.exe.manifest
> -outputresource:_configtest.exe;1
> Found executable C:\Program Files\Microsoft SDKs\Windows\v7.0A\bin\mt.exe
>
> _configtest.exe.manifest : general error c1010070: Failed to load and
> parse the manifest. The system cannot find the file specified.
>
> failure.
> removing: _configtest.c _configtest.obj
> Traceback (most recent call last):
>    File "setup.py", line 214, in <module>
>      setup_package()
>    File "setup.py", line 207, in setup_package
>      configuration=configuration )
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\core.py", line 186
> , in setup
>      return old_setup(**new_attr)
>    File "c:\python33\lib\distutils\core.py", line 148, in setup
>      dist.run_commands()
>    File "c:\python33\lib\distutils\dist.py", line 917, in run_commands
>      self.run_command(cmd)
>    File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>      cmd_obj.run()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build.py",
>   line 37, in run
>      old_build.run(self)
>    File "c:\python33\lib\distutils\command\build.py", line 126, in run
>      self.run_command(cmd_name)
>    File "c:\python33\lib\distutils\cmd.py", line 313, in run_command
>      self.distribution.run_command(command)
>    File "c:\python33\lib\distutils\dist.py", line 936, in run_command
>      cmd_obj.run()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 152, in run
>      self.build_sources()
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 163, in build_sources
>      self.build_library_sources(*libname_info)
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 298, in build_library_sources
>      sources = self.generate_sources(sources, (lib_name, build_info))
>    File "C:\Downloads\numpy-1.7.0b2\build\py3k\numpy\distutils\command\build_src.
> py", line 385, in generate_sources
>      source = func(extension, build_dir)
>    File "numpy\core\setup.py", line 648, in get_mathlib_info
>      raise RuntimeError("Broken toolchain: cannot link a simple C program")
> RuntimeError: Broken toolchain: cannot link a simple C program
>
> It appears a similar issue was raised before:
> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>
> Any tips?
>
> Peter

Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to
`mfinfo = None`.

http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648

Christoph
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Peter Cock
On Wed, Nov 14, 2012 at 7:35 PM, Christoph Gohlke <[hidden email]> wrote:

>> ...
>> RuntimeError: Broken toolchain: cannot link a simple C program
>>
>> It appears a similar issue was raised before:
>> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>>
>> Any tips?
>>
>> Peter
>
> Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to
> `mfinfo = None`.
>
> http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648
>
> Christoph

Hi Christoph,

That was very precise advice and seems to solve this. Presumably
you've faced something like this before. Is there an open issue for
this in Python itself?

Line 648 didn't seem sensible (and I guess the tip has changed), but
I tried replacing this bit of Python33\Lib\distutils\msvc9compiler.py
which was near by:

            # embed the manifest
            # XXX - this is somewhat fragile - if mt.exe fails, distutils
            # will still consider the DLL up-to-date, but it will not have a
            # manifest.  Maybe we should link to a temp file?  OTOH, that
            # implies a build environment error that shouldn't go undetected.
            mfinfo = self.manifest_get_embed_info(target_desc, ld_args)

with your suggestion of 'mfinfo = None', and this did seem enough
to get NumPy to compile with Python 3.3 and MSCV v10. I could
then build and test a library using NumPy (C and Python APIs), but
I've not yet installed nose to run NumPy's own tests.

Looking at the code for the manifest_get_embed_info method,
I don't see any obvious 9 vs 10 issues like the problems I hit
before. However there are some regular expressions in the
method _remove_visual_c_ref which it calls which look more
likely - looking for two digits when perhaps it needs to be three
under MSVC 10...

As a general point, if MSVC 10 is sufficiently different from 9,
does it make sense to introduce distutils/msvc10compiler.py
(subclassing/reusing most of distutils/msvc9compiler.py) in
Python itself? Or in NumPy's distutils?

Thanks,

Peter
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Christoph Gohlke
On 11/15/2012 6:24 AM, Peter Cock wrote:

> On Wed, Nov 14, 2012 at 7:35 PM, Christoph Gohlke <[hidden email]> wrote:
>>> ...
>>> RuntimeError: Broken toolchain: cannot link a simple C program
>>>
>>> It appears a similar issue was raised before:
>>> http://mail.scipy.org/pipermail/numpy-discussion/2012-June/062866.html
>>>
>>> Any tips?
>>>
>>> Peter
>>
>> Try changing line 648 in Python33\Lib\distutils\msvc9compiler.py to
>> `mfinfo = None`.
>>
>> http://hg.python.org/cpython/file/tip/Lib/distutils/msvc9compiler.py#l648
>>
>> Christoph
>
> Hi Christoph,
>
> That was very precise advice and seems to solve this. Presumably
> you've faced something like this before. Is there an open issue for
> this in Python itself?
>
> Line 648 didn't seem sensible (and I guess the tip has changed), but
> I tried replacing this bit of Python33\Lib\distutils\msvc9compiler.py
> which was near by:
>
>              # embed the manifest
>              # XXX - this is somewhat fragile - if mt.exe fails, distutils
>              # will still consider the DLL up-to-date, but it will not have a
>              # manifest.  Maybe we should link to a temp file?  OTOH, that
>              # implies a build environment error that shouldn't go undetected.
>              mfinfo = self.manifest_get_embed_info(target_desc, ld_args)
>
> with your suggestion of 'mfinfo = None', and this did seem enough
> to get NumPy to compile with Python 3.3 and MSCV v10. I could
> then build and test a library using NumPy (C and Python APIs), but
> I've not yet installed nose to run NumPy's own tests.
>
> Looking at the code for the manifest_get_embed_info method,
> I don't see any obvious 9 vs 10 issues like the problems I hit
> before. However there are some regular expressions in the
> method _remove_visual_c_ref which it calls which look more
> likely - looking for two digits when perhaps it needs to be three
> under MSVC 10...
>
> As a general point, if MSVC 10 is sufficiently different from 9,
> does it make sense to introduce distutils/msvc10compiler.py
> (subclassing/reusing most of distutils/msvc9compiler.py) in
> Python itself? Or in NumPy's distutils?
>
> Thanks,
>
> Peter
>

Naturally the file would be named msvc10compiler.py but the name may be
kept for compatibility reasons. AFAIK msvc10 does not use manifests any
longer for the CRT dependencies and all the code handling msvc9
manifests could be removed for Python 3.3. I have been building
extensions for Python 3.3 with msvc10 and this distutils patch for some
months and did not notice any issues.

Christoph
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Peter Cock
On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <[hidden email]> wrote:
>
> Naturally the file would be named msvc10compiler.py but the name may be
> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
> longer for the CRT dependencies and all the code handling msvc9
> manifests could be removed for Python 3.3. I have been building
> extensions for Python 3.3 with msvc10 and this distutils patch for some
> months and did not notice any issues.
>

Sounds Python 3.3 needs a fix then - have you reported this?
If not, could you report it (since you know far more about the
Windows build system than I do)?

If it will be fixed in Python itself, then perhaps a manual hack like
this will be enough for NumPy in the short term. Otherwise, maybe
numpy needs to include its own copy of msvc9compiler.py (or
msvc10compiler.py)?

Thanks,

Peter
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Christoph Gohlke
On 11/16/2012 1:28 AM, Peter Cock wrote:

> On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <[hidden email]> wrote:
>>
>> Naturally the file would be named msvc10compiler.py but the name may be
>> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
>> longer for the CRT dependencies and all the code handling msvc9
>> manifests could be removed for Python 3.3. I have been building
>> extensions for Python 3.3 with msvc10 and this distutils patch for some
>> months and did not notice any issues.
>>
>
> Sounds Python 3.3 needs a fix then - have you reported this?
> If not, could you report it (since you know far more about the
> Windows build system than I do)?
>
> If it will be fixed in Python itself, then perhaps a manual hack like
> this will be enough for NumPy in the short term. Otherwise, maybe
> numpy needs to include its own copy of msvc9compiler.py (or
> msvc10compiler.py)?
>
> Thanks,
>
> Peter

Could be related to <http://bugs.python.org/issue16296>.

Christoph
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Compiling NumPy on Windows for Python 3.3 with MSVC 2010

Peter Cock
On Fri, Nov 16, 2012 at 10:08 AM, Christoph Gohlke <[hidden email]> wrote:

> On 11/16/2012 1:28 AM, Peter Cock wrote:
>> On Thu, Nov 15, 2012 at 6:15 PM, Christoph Gohlke <[hidden email]> wrote:
>>>
>>> Naturally the file would be named msvc10compiler.py but the name may be
>>> kept for compatibility reasons. AFAIK msvc10 does not use manifests any
>>> longer for the CRT dependencies and all the code handling msvc9
>>> manifests could be removed for Python 3.3. I have been building
>>> extensions for Python 3.3 with msvc10 and this distutils patch for some
>>> months and did not notice any issues.
>>>
>>
>> Sounds Python 3.3 needs a fix then - have you reported this?
>> If not, could you report it (since you know far more about the
>> Windows build system than I do)?
>>
>> If it will be fixed in Python itself, then perhaps a manual hack like
>> this will be enough for NumPy in the short term. Otherwise, maybe
>> numpy needs to include its own copy of msvc9compiler.py (or
>> msvc10compiler.py)?
>>
>> Thanks,
>>
>> Peter
>
> Could be related to <http://bugs.python.org/issue16296>.
>
> Christoph

Thanks Christoph, you're probably right this is linked to
http://bugs.python.org/issue16296

Note here's an example of the manifest file, obtained from a hack
to Python 3.3's distutitls/msvc9compiler.py - looks like there are
no MSVC version numbers in here that we would need to
worry about:

build\temp.win32-3.3\Release\numpy\core\src\_dummy.pyd.manifest
<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level='asInvoker' uiAccess='false' />
      </requestedPrivileges>
    </security>
  </trustInfo>
</assembly>

I tried the patch from http://bugs.python.org/issue16296
applied by hand (in place of Christoph's one line change),
and it also seemed to work. I wanted to double check this,
so started by reverting to an unmodified copy of Python 3.3.

I just removed Python 3.3, and reinstalled it afresh using
python-3.3.0.msi, then updated to the latest commit on the
master branch of numpy, as it happens Ralf merging my
fixes to get mingw32  to compile numpy Python 3.3:
724da615902b9feb140cb6f7307ff1b1c2596a40

Now a clean numpy build under Python 3.3 with MSVC 10
"just worked", the error "Broken toolchain: cannot link a simple
C program" has gone. The comments in msvc9compiler.py did
mention this manifest stuff was fragile... but I am puzzled.

My hunch right now is that the order of installation of
MSVC 2010 and Python 3.3 could be important. Either
that, or something else changed on the numpy master
which had an impact?

Regards,

Peter
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion