Debugging NumPy development build in conda environment

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Debugging NumPy development build in conda environment

Lars Grüter
Dear devs,

If this is the wrong place for this kind of question I apologize and
feel free to ignore this or point me to the right place.

I'm using the conda package manager for my development environment which
used to work just fine with these steps:

> conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage
> source activate dev-numpy
> conda uninstall --force numpy numpy-base
> pip install -e .

This would yield me a nice isolated playground and I could use

> python setup.py build_ext --incplace -j 4

at anytime to rebuild the binaries. Unfortunately the last two commands
stopped working out of the blue with my existing environment and fail with

> RuntimeError: Broken toolchain: cannot link a simple C program

(full output in [1]) on a clean repository. No recent changes to my tool
chain come to mind which could help me pinpoint the problem. Because it
used to work just fine I have the annoying feeling that the fix is
something small and trivial and have the hope that it's immediately
obvious to someone here or that you can suggest a way to debug this. In
any case your help would be appreciated.

I'm aware that this may be an exotic dev setup. Using a virtual
environment instead of the conda one works just fine. The build script
seems to use a different linear algebra library in that case: blas from
my OS as opposed to mkl in the conda environment. This is not my
preferred solution but it would do if nothing comes of this.

Some additional facts and research I've done:

- Using conda environments with asv has stopped working as well due to
not finding any LA library at all.
- If I'm understanding [1] correctly the build script finds the mkl
library provided by the conda environment.
- Trying earlier versions of NumPy shows the same behavior. I tested the
tags v1.14.6, v1.15.4.
- However creating a development environment for SciPy and scikit-image
using this approach still works fine.
- Searching the internet or looking at NumPy's build guide didn't yield
me a solution.
- I'm on linux (Manjaro) if that is of any relevance.
- Miniconda was reinstalled in my home directory with the installation
script provided through Anaconda's website.
- Haven't yet tried this approach in a fresh virtual machine.

I hope that I haven't overlooked something obvious. In any case thank
you for your time and attention.

Cheers, Lars

[1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Debugging NumPy development build in conda environment

Hameer Abbasi
On Friday, Jan 04, 2019 at 2:44 PM, Lars Grueter <[hidden email]> wrote:
Dear devs,

If this is the wrong place for this kind of question I apologize and
feel free to ignore this or point me to the right place.

I'm using the conda package manager for my development environment which
used to work just fine with these steps:

conda create -n dev-numpy python=3.7 numpy numpydoc cython pytest ipython coverage
source activate dev-numpy
conda uninstall --force numpy numpy-base
pip install -e .

This would yield me a nice isolated playground and I could use

python setup.py build_ext --incplace -j 4

at anytime to rebuild the binaries. Unfortunately the last two commands
stopped working out of the blue with my existing environment and fail with

RuntimeError: Broken toolchain: cannot link a simple C program

(full output in [1]) on a clean repository. No recent changes to my tool
chain come to mind which could help me pinpoint the problem. Because it
used to work just fine I have the annoying feeling that the fix is
something small and trivial and have the hope that it's immediately
obvious to someone here or that you can suggest a way to debug this. In
any case your help would be appreciated.

I'm aware that this may be an exotic dev setup. Using a virtual
environment instead of the conda one works just fine. The build script
seems to use a different linear algebra library in that case: blas from
my OS as opposed to mkl in the conda environment. This is not my
preferred solution but it would do if nothing comes of this.

Some additional facts and research I've done:

- Using conda environments with asv has stopped working as well due to
not finding any LA library at all.
- If I'm understanding [1] correctly the build script finds the mkl
library provided by the conda environment.
- Trying earlier versions of NumPy shows the same behavior. I tested the
tags v1.14.6, v1.15.4.
- However creating a development environment for SciPy and scikit-image
using this approach still works fine.
- Searching the internet or looking at NumPy's build guide didn't yield
me a solution.
- I'm on linux (Manjaro) if that is of any relevance.
- Miniconda was reinstalled in my home directory with the installation
script provided through Anaconda's website.
- Haven't yet tried this approach in a fresh virtual machine.

I hope that I haven't overlooked something obvious. In any case thank
you for your time and attention.

Cheers, Lars

[1] https://gist.github.com/lagru/c6ac16b9984492850fe1174d6418b7b5
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

Hi Lars,

What I’m used to doing is simply the following (inside the NumPy root):

conda create -n numpy-dev python=3[.x] pytest
conda activate numpy-dev
python setup.py build_ext --inplace -j 4
pip install -e .

Often, when I run into issues with the build, I simply do this:

[backup site.cfg]
git clean -xfd
[restore site.cfg]
python setup.py build_ext --inplace -j 4
pip install -e .

In your case, it might be the second that helps. If it’s not finding any LA library, you can try compiling with the site.cfg that points to MKL.

Cheers and Best Regards,
Hameer Abbasi

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Debugging NumPy development build in conda environment

mattip
In reply to this post by Lars Grüter

On 4/1/19 3:34 pm, Lars Grueter wrote:
> Unfortunately the last two commands
> stopped working out of the blue with my existing environment and fail with
>
>> RuntimeError: Broken toolchain: cannot link a simple C program


The compiler error is a few lines above this (copied below). When I have
gotten that kind of output, it is because I am using a 32 bit compiler
on 64 bit or the opposite.



C compiler: gcc -pthread -B
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat -Wl,--sysroot=/
-Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC

compile options: '-Inumpy/core/src/common -Inumpy/core/src -Inumpy/core
-Inumpy/core/src/npymath -Inumpy/core/src/multiarray
-Inumpy/core/src/umath -Inumpy/core/src/npysort
-I/home/lg/.miniconda3/envs/dev-numpy/include/python3.7m -c'
gcc: _configtest.c
gcc -pthread -B /home/lg/.miniconda3/envs/dev-numpy/compiler_compat
-Wl,--sysroot=/ _configtest.o -o _configtest
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o:
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o:
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o:
unable to initialize decompress status for section .debug_info
/home/lg/.miniconda3/envs/dev-numpy/compiler_compat/ld: _configtest.o:
unable to initialize decompress status for section .debug_info
_configtest.o: file not recognized: file format not recognized



Can you compile anything with that gcc?


Matti

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Debugging NumPy development build in conda environment

Lars Grüter
On 04/01/2019 14:55, Matti Picus wrote:

> On 4/1/19 3:34 pm, Lars Grueter wrote:
>> Unfortunately the last two commands
>> stopped working out of the blue with my existing environment and fail
>> with
>>
>>> RuntimeError: Broken toolchain: cannot link a simple C program
>
>
> The compiler error is a few lines above this (copied below). When I have
> gotten that kind of output, it is because I am using a 32 bit compiler
> on 64 bit or the opposite.
>
> [...]
>
> Can you compile anything with that gcc?

This problem even persists on a new system.

I noticed that conda includes libgcc-ng [1] in every environment with
Python. Is it possible that this library (or another one) is
incompatible with the GCC version installed on my system? That would
also explain why changing the available LA libraries has not effect and
why the setup works fine using plain virtual environments created with venv.

Is someone else successfully maintaining a development setup with conda
on an up-to-date Arch-based system or even Manjaro?

Best regards,
Lars

[1] http://docs.continuum.io/anaconda/packages/py3.7_linux-64/
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Debugging NumPy development build in conda environment

Lars Grüter
In reply to this post by Hameer Abbasi
Sorry, I just noticed that I didn't send the message below to the
mailing list but replied directly to Hameer. I've resent it for the
archives below.

@Hameer, sorry for the spam. :)

Best regards,
Lars

On 04/01/2019 20:06, Lars Grueter wrote:> Hi Hameer,

>
> thanks for the fast answer! I'm even more confused now. If any of my
> assumptions below are wrong please point them out.
>
> On 04.01.19 14:53, Hameer Abbasi wrote:
>> Hi Lars,
>>
>> What I’m used to doing is simply the following (inside the NumPy root):
>>
>>     conda create -n numpy-dev python=3[.x] pytest
>>     conda activate numpy-dev
>>     python setup.py build_ext --inplace -j 4
>
> This fails right here telling me that it doesn't find any LA library
> such as ATLAS, BLIS, MKL, etc. (assuming I read the output correctly).
> The error message is however different from the workflow I described in
> my first mail. It's missing the sections à la
>
> lapack_mkl_info:
> customize UnixCCompiler
>   FOUND:
>     libraries = ['mkl_rt', 'pthread']
>
> I checked my OS's package manager and found blas, cblas and lapack.
> Aren't these LA libraries? Am I still missing an OS version of either
> ATLAS, BLIS or MKL?
>
> https://docs.scipy.org/doc/numpy/user/building.html#disabling-atlas-and-other-accelerated-libraries
> tells me it should be possible to build NumPy without these libraries.
> This has no effect and won't build.
>
> But if that's the case why is the build working inside an environment
> created with virtualenv? Is it building without those libraries?
>
>> Often, when I run into issues with the build, I simply do this:
>>
>>     [backup site.cfg]
>>     git clean -xfd
>>     [restore site.cfg]
>>     python setup.py build_ext --inplace -j 4
>>     pip install -e . >
>> In your case, it might be the second that helps. If it’s not finding any
>> LA library, you can try compiling with the site.cfg that points to MKL.
>
> I didn't yet need my own site.cfg. I tried creating one myself right now
> but regardless of how I try to point it to the conda envs lib and
> include directories it has now effect on the output of the setup.py
> command. The build script already finds these paths by default. E.g.:
>
> [mkl]
> library_dirs = /home/lg/.miniconda3/envs/dev-numpy/lib
> include_dirs = /home/lg/.miniconda3/envs/dev-numpy/include
> mkl_libs = mkl_rt
> lapack_libs =
>
> I'm not sure what else to provide in this file.
>
> Thanks again and best regards,
> Lars

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion