setuptools/distutils merger & numpy.distutils

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

setuptools/distutils merger & numpy.distutils

ralfgommers
Hi all,

I spent some time looking at the fallout of the setuptools 50.0 release. There's quite a few small issues, those can/should all be worked around by pinning setuptools to a lower version.

The root cause and main longer-term issue is that numpy.distutils extends and monkeypatches distutils, which mostly was fine because distutils moved super slowly and had a decent QA process. Now with setuptools, any patch goes into master and gets released to the whole wide world without any testing. The summary of that and how to deal with it I posted on https://github.com/pypa/setuptools/issues/2372 for discussion.

Cheers,
Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

ralfgommers
Hi all,

Preliminary conclusion: let's pin all packages to setuptools < 50.0 and wait for 6-9 months till the dust settles. Things may still change; the reception of PEP 632 [1] hasn't been a "great idea, let's just rip it out without a plan" one so far.

Cheers,
Ralf



On Wed, Sep 2, 2020 at 11:33 AM Ralf Gommers <[hidden email]> wrote:
Hi all,

I spent some time looking at the fallout of the setuptools 50.0 release. There's quite a few small issues, those can/should all be worked around by pinning setuptools to a lower version.

The root cause and main longer-term issue is that numpy.distutils extends and monkeypatches distutils, which mostly was fine because distutils moved super slowly and had a decent QA process. Now with setuptools, any patch goes into master and gets released to the whole wide world without any testing. The summary of that and how to deal with it I posted on https://github.com/pypa/setuptools/issues/2372 for discussion.

Cheers,
Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

David Cournapeau
I will comment with more details on GH, but I think jraco's suggestion makes a lot of sense. I would be willing to spend some time to merge upstream (i.e. setuptools now :) ) some of our changes in numpy.distutils.

Assuming the numpy.distutils codebase has not changed much in the last 10 years, my sense is that a lot of the features that relied on monkey patching can be merged upstream,  fortran support being one notable exception.

I remember trying to put in mingw64 support in distutils upstream 10 years ago, and the pace of communication was so slow that I gave up. But it was technically straightforward.

David

On Sat, Sep 5, 2020 at 8:13 PM Ralf Gommers <[hidden email]> wrote:
Hi all,

Preliminary conclusion: let's pin all packages to setuptools < 50.0 and wait for 6-9 months till the dust settles. Things may still change; the reception of PEP 632 [1] hasn't been a "great idea, let's just rip it out without a plan" one so far.

Cheers,
Ralf



On Wed, Sep 2, 2020 at 11:33 AM Ralf Gommers <[hidden email]> wrote:
Hi all,

I spent some time looking at the fallout of the setuptools 50.0 release. There's quite a few small issues, those can/should all be worked around by pinning setuptools to a lower version.

The root cause and main longer-term issue is that numpy.distutils extends and monkeypatches distutils, which mostly was fine because distutils moved super slowly and had a decent QA process. Now with setuptools, any patch goes into master and gets released to the whole wide world without any testing. The summary of that and how to deal with it I posted on https://github.com/pypa/setuptools/issues/2372 for discussion.

Cheers,
Ralf

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

Daniele Nicolodi
On 06/09/2020 07:06, David Cournapeau wrote:
> Assuming the numpy.distutils codebase has not changed much in the last
> 10 years, my sense is that a lot of the features that relied on monkey
> patching can be merged upstream,  fortran support being one notable
> exception.

This may be bigger endeavor, but wouldn't it be possible to extend
setuptools interfaces in a way that plugging in the fortran support does
not require monkey patching or accessing the implementation internals?

Such an extension could also be used to extend setuptools to handle
extensions written in other languages such as Rust.

By the way, how is Cython affected by this? Cython also distributes an
extension to (the module formerly know as) distutils to transparently
compile extensions written in Cython. Could the efforts toward
multi-language support in distutils be coordinated with the Cython
maintainers?

Cheers,
Dan
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

ralfgommers
In reply to this post by David Cournapeau


On Sun, Sep 6, 2020 at 6:07 AM David Cournapeau <[hidden email]> wrote:
I will comment with more details on GH, but I think jraco's suggestion makes a lot of sense. I would be willing to spend some time to merge upstream (i.e. setuptools now :) ) some of our changes in numpy.distutils.

Thanks David!


Assuming the numpy.distutils codebase has not changed much in the last 10 years, my sense is that a lot of the features that relied on monkey patching can be merged upstream,  fortran support being one notable exception.

There has been a lot of churn and a few new features, but the structure hasn't changed much so I think your assessment is right.

Cheers,
Ralf


I remember trying to put in mingw64 support in distutils upstream 10 years ago, and the pace of communication was so slow that I gave up. But it was technically straightforward.

David

On Sat, Sep 5, 2020 at 8:13 PM Ralf Gommers <[hidden email]> wrote:
Hi all,

Preliminary conclusion: let's pin all packages to setuptools < 50.0 and wait for 6-9 months till the dust settles. Things may still change; the reception of PEP 632 [1] hasn't been a "great idea, let's just rip it out without a plan" one so far.

Cheers,
Ralf



On Wed, Sep 2, 2020 at 11:33 AM Ralf Gommers <[hidden email]> wrote:
Hi all,

I spent some time looking at the fallout of the setuptools 50.0 release. There's quite a few small issues, those can/should all be worked around by pinning setuptools to a lower version.

The root cause and main longer-term issue is that numpy.distutils extends and monkeypatches distutils, which mostly was fine because distutils moved super slowly and had a decent QA process. Now with setuptools, any patch goes into master and gets released to the whole wide world without any testing. The summary of that and how to deal with it I posted on https://github.com/pypa/setuptools/issues/2372 for discussion.

Cheers,
Ralf

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

ralfgommers
In reply to this post by Daniele Nicolodi


On Sun, Sep 6, 2020 at 9:50 AM Daniele Nicolodi <[hidden email]> wrote:
On 06/09/2020 07:06, David Cournapeau wrote:
> Assuming the numpy.distutils codebase has not changed much in the last
> 10 years, my sense is that a lot of the features that relied on monkey
> patching can be merged upstream,  fortran support being one notable
> exception.

This may be bigger endeavor, but wouldn't it be possible to extend
setuptools interfaces in a way that plugging in the fortran support does
not require monkey patching or accessing the implementation internals?

Probably not, since distutils really doesn't have much of a design, or separation between API and implementation.


Such an extension could also be used to extend setuptools to handle
extensions written in other languages such as Rust.

By the way, how is Cython affected by this? Cython also distributes an
extension to (the module formerly know as) distutils to transparently
compile extensions written in Cython. Could the efforts toward
multi-language support in distutils be coordinated with the Cython
maintainers?

Cython.Distutils is a few hundred lines of code, numpy.distutils is 20,000 lines of code. I don't think Cython will have much problems adapting.

Cheers,
Ralf


Cheers,
Dan
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

Daniele Nicolodi
On 06/09/2020 11:28, Ralf Gommers wrote:
> On Sun, Sep 6, 2020 at 9:50 AM Daniele Nicolodi <[hidden email]>
>     This may be bigger endeavor, but wouldn't it be possible to extend
>     setuptools interfaces in a way that plugging in the fortran support does
>     not require monkey patching or accessing the implementation internals?
>
> Probably not, since distutils really doesn't have much of a design, or
> separation between API and implementation.

I was under the impression that one of the reasons why distutils is
being deprecated in favor of setuptools is to change this and evolve the
code into a better form, not to just move the code around.

>     By the way, how is Cython affected by this? Cython also distributes an
>     extension to (the module formerly know as) distutils to transparently
>     compile extensions written in Cython. Could the efforts toward
>     multi-language support in distutils be coordinated with the Cython
>     maintainers?
>
> Cython.Distutils is a few hundred lines of code, numpy.distutils is
> 20,000 lines of code. I don't think Cython will have much problems adapting.

I'm well aware of that, however I was only referring to Fortran support
here, which may be something valuable to merge upstream. And Cython also
injects the support for a different language into distutils/setuptools,
thus maybe a common approach could be envisioned, maybe more robust than
the current one. I haven't looked at the code, though.

Cheers,
Dan
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: setuptools/distutils merger & numpy.distutils

David Cournapeau


On Sun, Sep 6, 2020 at 6:40 PM Daniele Nicolodi <[hidden email]> wrote:
On 06/09/2020 11:28, Ralf Gommers wrote:
> On Sun, Sep 6, 2020 at 9:50 AM Daniele Nicolodi <[hidden email]>
>     This may be bigger endeavor, but wouldn't it be possible to extend
>     setuptools interfaces in a way that plugging in the fortran support does
>     not require monkey patching or accessing the implementation internals?
>
> Probably not, since distutils really doesn't have much of a design, or
> separation between API and implementation.

I was under the impression that one of the reasons why distutils is
being deprecated in favor of setuptools is to change this and evolve the
code into a better form, not to just move the code around.

The conclusion of most people who do know that codebase is that you can't fix it through evolutionary design. It needs to be redone from the ground up, and few people are interested in doing that work.

David

>     By the way, how is Cython affected by this? Cython also distributes an
>     extension to (the module formerly know as) distutils to transparently
>     compile extensions written in Cython. Could the efforts toward
>     multi-language support in distutils be coordinated with the Cython
>     maintainers?
>
> Cython.Distutils is a few hundred lines of code, numpy.distutils is
> 20,000 lines of code. I don't think Cython will have much problems adapting.

I'm well aware of that, however I was only referring to Fortran support
here, which may be something valuable to merge upstream. And Cython also
injects the support for a different language into distutils/setuptools,
thus maybe a common approach could be envisioned, maybe more robust than
the current one. I haven't looked at the code, though.

Cheers,
Dan
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion