Issue with setup_requires and 1.16 release candidates

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue with setup_requires and 1.16 release candidates

Thomas Robitaille
Hi all,

Back in December I started getting failures in continuous integration
as well as reports of failures from users of installation issues for a
couple of packages. The problem can be easily reproduced in a Docker
container with:

   FROM ubuntu:16.04
   RUN apt-get update
   RUN apt-get install -y python3 python3-dev python3-pip python3-wheel
   RUN pip3 install fast-histogram
   RUN python3 -c 'import fast_histogram'

Doing this results in the following traceback:

   ImportError: No module named 'numpy.core._multiarray_umath'
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py",
line 1, in <module>
       from .histogram import *
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py",
line 7, in <module>
       from ._histogram_core import (_histogram1d,
   ImportError: numpy.core.multiarray failed to import

I've seen similar issues with other packages too. The key is that the
fast-histogram package defines:

   setup_requires=['numpy']

in the setup.py (since the package has a C extension that uses the
Numpy C API) and numpy is needed before the install_requires
dependencies are installed:

   https://github.com/astrofrog/fast-histogram/blob/master/setup.py

Now this normally works fine, but the issues I saw started when the
first 1.16 RC was made available, when installing into an environment
in which numpy is not already installed.

My understanding is that setup_requires is honored by easy_install
(even if installing the main package with pip), which doesn't ignore
pre-releases. Thus, the package is built against the 1.16 RC but then
1.15 is installed due to:

   install_requires=['numpy']

which is honored by pip.

I think that the correct solution is to make sure that:

   [build-system]
   requires = ["setuptools", "wheel", "numpy"]

is added to the pyproject.toml file (as per PEP 518). This then works
properly with recent versions of pip (>10 I think). I think removing
setup_requires then makes sense because it'd be better to have an
error with older versions of pip that numpy is not installed rather
than having the package built against the wrong version.

I know this is a temporary issue in the sense that it will go away
once the final version of 1.16 is out, but I just wanted to share this
as a heads-up in case you get reports of issues from other people, and
also to check whether there are any other solutions/workarounds to be
aware of? (to avoid a similar situation in future).

Thanks,
Tom
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Issue with setup_requires and 1.16 release candidates

ralfgommers


On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille <[hidden email]> wrote:
Hi all,

Back in December I started getting failures in continuous integration
as well as reports of failures from users of installation issues for a
couple of packages. The problem can be easily reproduced in a Docker
container with:

   FROM ubuntu:16.04
   RUN apt-get update
   RUN apt-get install -y python3 python3-dev python3-pip python3-wheel
   RUN pip3 install fast-histogram
   RUN python3 -c 'import fast_histogram'

Doing this results in the following traceback:

   ImportError: No module named 'numpy.core._multiarray_umath'
   Traceback (most recent call last):
     File "<string>", line 1, in <module>
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py",
line 1, in <module>
       from .histogram import *
     File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py",
line 7, in <module>
       from ._histogram_core import (_histogram1d,
   ImportError: numpy.core.multiarray failed to import

I've seen similar issues with other packages too. The key is that the
fast-histogram package defines:

   setup_requires=['numpy']

in the setup.py (since the package has a C extension that uses the
Numpy C API) and numpy is needed before the install_requires
dependencies are installed:

   https://github.com/astrofrog/fast-histogram/blob/master/setup.py

Now this normally works fine, but the issues I saw started when the
first 1.16 RC was made available, when installing into an environment
in which numpy is not already installed.

My understanding is that setup_requires is honored by easy_install
(even if installing the main package with pip), which doesn't ignore
pre-releases. Thus, the package is built against the 1.16 RC but then
1.15 is installed due to:

   install_requires=['numpy']

which is honored by pip.

Oh fun. Thanks for explaining this!


I think that the correct solution is to make sure that:

   [build-system]
   requires = ["setuptools", "wheel", "numpy"]

is added to the pyproject.toml file (as per PEP 518).

To clarify: you're talking about fast-histogram's pyproject.toml file right?

This then works
properly with recent versions of pip (>10 I think). I think removing
setup_requires then makes sense because it'd be better to have an
error with older versions of pip that numpy is not installed rather
than having the package built against the wrong version.

This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way.

Cheers,
Ralf



I know this is a temporary issue in the sense that it will go away
once the final version of 1.16 is out, but I just wanted to share this
as a heads-up in case you get reports of issues from other people, and
also to check whether there are any other solutions/workarounds to be
aware of? (to avoid a similar situation in future).

Thanks,
Tom
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Issue with setup_requires and 1.16 release candidates

Thomas Robitaille
Hi Ralf,

On Sun, 6 Jan 2019 at 22:06, Ralf Gommers <[hidden email]> wrote:

>
>
>
> On Sun, Jan 6, 2019 at 12:45 PM Thomas Robitaille <[hidden email]> wrote:
>>
>> Hi all,
>>
>> Back in December I started getting failures in continuous integration
>> as well as reports of failures from users of installation issues for a
>> couple of packages. The problem can be easily reproduced in a Docker
>> container with:
>>
>>    FROM ubuntu:16.04
>>    RUN apt-get update
>>    RUN apt-get install -y python3 python3-dev python3-pip python3-wheel
>>    RUN pip3 install fast-histogram
>>    RUN python3 -c 'import fast_histogram'
>>
>> Doing this results in the following traceback:
>>
>>    ImportError: No module named 'numpy.core._multiarray_umath'
>>    Traceback (most recent call last):
>>      File "<string>", line 1, in <module>
>>      File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py",
>> line 1, in <module>
>>        from .histogram import *
>>      File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py",
>> line 7, in <module>
>>        from ._histogram_core import (_histogram1d,
>>    ImportError: numpy.core.multiarray failed to import
>>
>> I've seen similar issues with other packages too. The key is that the
>> fast-histogram package defines:
>>
>>    setup_requires=['numpy']
>>
>> in the setup.py (since the package has a C extension that uses the
>> Numpy C API) and numpy is needed before the install_requires
>> dependencies are installed:
>>
>>    https://github.com/astrofrog/fast-histogram/blob/master/setup.py
>>
>> Now this normally works fine, but the issues I saw started when the
>> first 1.16 RC was made available, when installing into an environment
>> in which numpy is not already installed.
>>
>> My understanding is that setup_requires is honored by easy_install
>> (even if installing the main package with pip), which doesn't ignore
>> pre-releases. Thus, the package is built against the 1.16 RC but then
>> 1.15 is installed due to:
>>
>>    install_requires=['numpy']
>>
>> which is honored by pip.
>
>
> Oh fun. Thanks for explaining this!
>
>>
>> I think that the correct solution is to make sure that:
>>
>>    [build-system]
>>    requires = ["setuptools", "wheel", "numpy"]
>>
>> is added to the pyproject.toml file (as per PEP 518).
>
>
> To clarify: you're talking about fast-histogram's pyproject.toml file right?

Yes that's right. Just for the record, here's the pull request to
fast-histogram to implement the fix:

https://github.com/astrofrog/fast-histogram/pull/33

Cheers,
Tom

>
>> This then works
>> properly with recent versions of pip (>10 I think). I think removing
>> setup_requires then makes sense because it'd be better to have an
>> error with older versions of pip that numpy is not installed rather
>> than having the package built against the wrong version.
>
>
> This is a pretty hacky solution, needed because pip doesn't support setup_requires, but yes it should work and I can't think of a better way.
>
> Cheers,
> Ralf
>
>
>>
>> I know this is a temporary issue in the sense that it will go away
>> once the final version of 1.16 is out, but I just wanted to share this
>> as a heads-up in case you get reports of issues from other people, and
>> also to check whether there are any other solutions/workarounds to be
>> aware of? (to avoid a similar situation in future).
>>
>> Thanks,
>> Tom
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion