Proposal to accept NEP-18, __array_function__ protocol

classic Classic list List threaded Threaded
62 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

einstein.edison
> On 29. Aug 2018, at 11:44, Matti Picus <[hidden email]> wrote:
>
> On 29/08/18 10:37, Nathaniel Smith wrote:
>> it's easy to imagine scenarios where the
>> people being broken aren't the ones who had a chance to read the docs
>> – e.g. if a major package starts relying on __array_function__, then
>> it's all*their*  users who we'd be breaking, even though they had
>> nothing to do with it.
> This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages.

One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion.

Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.

>
> Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP?
> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Matthew Rocklin
>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?

I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis.  Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts).  Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out.

This is different from how Numpy has operated in the past, but that might be ok.

>> 2. is it plausible that we'll find ourselves in that situation?

> Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime

Yup.  Same for Dask.  We're pretty accustomed to this.

On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <[hidden email]> wrote:
> On 29. Aug 2018, at 11:44, Matti Picus <[hidden email]> wrote:
>
> On 29/08/18 10:37, Nathaniel Smith wrote:
>> it's easy to imagine scenarios where the
>> people being broken aren't the ones who had a chance to read the docs
>> – e.g. if a major package starts relying on __array_function__, then
>> it's all*their*  users who we'd be breaking, even though they had
>> nothing to do with it.
> This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages.

One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion.

Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.

>
> Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP?
> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Marten van Kerkwijk
HI All,

On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13.

One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.)

All the best,

Marten

On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <[hidden email]> wrote:
>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?

I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis.  Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts).  Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out.

This is different from how Numpy has operated in the past, but that might be ok.

>> 2. is it plausible that we'll find ourselves in that situation?

> Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime

Yup.  Same for Dask.  We're pretty accustomed to this.

On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <[hidden email]> wrote:
> On 29. Aug 2018, at 11:44, Matti Picus <[hidden email]> wrote:
>
> On 29/08/18 10:37, Nathaniel Smith wrote:
>> it's easy to imagine scenarios where the
>> people being broken aren't the ones who had a chance to read the docs
>> – e.g. if a major package starts relying on __array_function__, then
>> it's all*their*  users who we'd be breaking, even though they had
>> nothing to do with it.
> This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages.

One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion.

Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.

>
> Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP?
> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Matthew Rocklin
> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,

My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings  attached to it.   In that case astropy might just wait before adopting it until that label was dropped.  Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely.

This would give the ecosystem an opportunity to try something and modify it after having some experience without promising to support it in a fully backwards compatible way forever.

On Wed, Aug 29, 2018 at 9:46 AM Marten van Kerkwijk <[hidden email]> wrote:
HI All,

On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain, i.e., that it will continue to be possible to override, say, concatenate. It is not a big deal if the actual implementation changes (say, `__array_concatenate__` is introduced) and we need to do some version-dependent magic to ensure that our users do not notice the change; we did the same with `__array_ufunc__` - the wrap/prepare machinery will be gone only in the next version of astropy, when our minimum version for numpy reaches 1.13.

One thing perhaps worth remembering that what really matters is not so much `__array_function__` but the set of functions that actually implement the override. It would seem unlikely this would be the complete numpy API on the first go; perhaps it is an idea to be somewhat specific about which part we start with? My vote would go towards array manipulation routines like concatenate. I would also suggest to avoid those functions for which ufunc implementations would seem quite possible (i.e., avoid things like median, etc.)

All the best,

Marten

On Wed, Aug 29, 2018 at 8:31 AM Matthew Rocklin <[hidden email]> wrote:
>> 1. if we do find ourselves in a situation where changing this would break lots of users, will we consider ourselves beholden to them?

I think that it would be useful for Numpy's continued evolution to develop the ability to include code on a provisional basis.  Other projects do this and they just have big bold "Experimental" notes everywhere that a user might go to learn about the functionality (docs, docstrings, blogposts).  Some users will definitely get burned, yes, but the alternative is to burn all other users a little bit by moving too slowly and not trying things out.

This is different from how Numpy has operated in the past, but that might be ok.

>> 2. is it plausible that we'll find ourselves in that situation?

> Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime

Yup.  Same for Dask.  We're pretty accustomed to this.

On Wed, Aug 29, 2018 at 7:01 AM Hameer Abbasi <[hidden email]> wrote:
> On 29. Aug 2018, at 11:44, Matti Picus <[hidden email]> wrote:
>
> On 29/08/18 10:37, Nathaniel Smith wrote:
>> it's easy to imagine scenarios where the
>> people being broken aren't the ones who had a chance to read the docs
>> – e.g. if a major package starts relying on __array_function__, then
>> it's all*their*  users who we'd be breaking, even though they had
>> nothing to do with it.
> This is a packaging problem. This proposal is intended for use by other "major packages", not so much for end-users. We would have much more trouble if we were proposing a broad change to something like indexing or the random number module (see those NEPs). If we break one of those major packages, it is on them to pin the version of NumPy they can work with. In my opinion very few end users will be implementing their own ndarray classes with `__array_function__`. While we will get issue reports, we can handle them much as we do the MKL or OpenBLAS ones - pinpoint the problem and urge users to complain to those packages.

One thing that might help here is nightly or continuous CI builds of the NumPy wheels. This would be good, as we could test it in CI, and fix it when it comes up. But I guess that’s another discussion.

Personally, for as long as this protocol is experimental, I’ll add a warning in the docs of sparse as well; saying this might disappear anytime.

>
> Other than adding a warning, I am not sure what the concrete proposal is here. To not accept the NEP?
> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Marten van Kerkwijk
On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <[hidden email]> wrote:
> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,

My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings  attached to it.   In that case astropy might just wait before adopting it until that label was dropped.  Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely.

Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain.

I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that -  the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.)

-- Marten

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Allan, Daniel
Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain. 

That's fair enough, but I think Matt's point still stands. Any given project can weigh the benefits of early adoption against the risks, considering its need for a new feature and its resources available to deal with possible future changes. An "Experimental" designation gives numpy a little space to innovate and lets individual projects opt in. Without that space, numpy has to take on all the burden of continuity instead of sharing that burden with the community of projects that opt in to early adoption.

Best,
Dan


Daniel B. Allan, Ph.D
Associate Computational Scientist, Brookhaven National Lab
(631) 344-3281 (no voicemail set up)

From: NumPy-Discussion <numpy-discussion-bounces+dallan=[hidden email]> on behalf of Marten van Kerkwijk <[hidden email]>
Sent: Wednesday, August 29, 2018 10:43:55 AM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] Proposal to accept NEP-18, __array_function__ protocol
 
On Wed, Aug 29, 2018 at 9:53 AM Matthew Rocklin <[hidden email]> wrote:
> On the backwards compatibility: from an astropy perspective, I would expect that the introduction of `__array_function__` implies a guarantee that the *functionality* it provides will remain,

My guess is that you wouldn't have this expectation if Numpy released this feature with explicit "Experimental" warnings  attached to it.   In that case astropy might just wait before adopting it until that label was dropped.  Projects that were more risk-friendly would then take on the burden of trying it out, working out some kinks, and then hopefully in a version or two the "Experimental" label would be dropped and astropy would step in and adopt it more safely.

Well, I guess I'll be proving Nathaniel right: I would *definitely* start using __array_function__ in astropy - not being able to concatenate Quantity and other instances which use it has been a long-standing pain.

I do think that even for an experimental feature one should be allowed to expect that there will continue to be a way provided by numpy to access the same functionality, i.e., once we allow people to do `np.concatenate(list-of-mimics)`, I think we should consider ourselves committed to providing some way to continue doing that -  the experimental tag should only imply that we are not committed to the precise method with which that can be achieved. (Of course, I can be a bit more cavalier than most, since I am hopeful I can help ensure that some method will continue to be present; indeed, the same happened for __array_ufunc__ - I had already written a __numpy_ufunc__ implementation in Quantity and when that stalled in numpy, I picked up and finished the __array_ufunc__ code that Nathan had written.)

-- Marten

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Marten van Kerkwijk
Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented).

-- Marten

p.s. And, yes, do count me in for helping to back up that commitment!


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Nathaniel Smith
In reply to this post by mattip
On Wed, Aug 29, 2018, 02:44 Matti Picus <[hidden email]> wrote:
On 29/08/18 10:37, Nathaniel Smith wrote:
> it's easy to imagine scenarios where the
> people being broken aren't the ones who had a chance to read the docs
> – e.g. if a major package starts relying on __array_function__, then
> it's all*their*  users who we'd be breaking, even though they had
> nothing to do with it.
This is a packaging problem. This proposal is intended for use by other
"major packages", not so much for end-users. We would have much more
trouble if we were proposing a broad change to something like indexing
or the random number module (see those NEPs). If we break one of those
major packages, it is on them to pin the version of NumPy they can work
with. In my opinion very few end users will be implementing their own
ndarray classes with `__array_function__`. While we will get issue
reports, we can handle them much as we do the MKL or OpenBLAS ones -
pinpoint the problem and urge users to complain to those packages.

Other than adding a warning, I am not sure what the concrete proposal is
here. To not accept the NEP?

The proposal is just that while the NEP is considered experimental and provisional, we should use some kind of technical measures to discourage use in a non-experimental settings. We want to stay in control of when it escapes the lab, and docs alone, or trivially disableable messages, aren't a very effective way to do that.

-n

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
In reply to this post by Marten van Kerkwijk
On Wed, Aug 29, 2018 at 9:34 AM Marten van Kerkwijk <[hidden email]> wrote:
Absolutely fine to have to deal with future chances - my main point is that by accepting the NEP, I think numpy is committing to provide some way to override whatever functions __array_function__ is introduced for, i.e., we cannot reasonably go back to not providing any way to override such a function (but by marking the feature experimental, we *can* change the way the override of any given function is implemented).

I'm not entirely sure that we're ready to make this commitment yet, but I think this ties into our decision about what sort of user opt-ins to require.

I'd like to structure this decision in terms of the types of breaking changes we contemplate possibly making in this future with regards to this protocol:

1. Abandoning this approach of allowing overrides for NumPy's high level API entirely, in favor of requiring the use of a dedicated namespace like numpy.api. This might arise if we decide that the overhead of checking for __array_function__ attributes is too much, or if it's simply too confusing to allow NumPy functions to be overloaded in arbitrary ways.
2. Changing the general __array_function__ protocol in a breaking way, e.g., to eliminate the "types" argument.
3. Replacing __array_function__ with another override mechanism, either (a) in general (e.g., __array_function_v2__ without the "types" argument) or (b) for specific functions (e.g., switching to a more specific protocol like __array_ufunc__).
4. Removing the possibility of overriding specific functions altogether, because we want to require using a more generic interface (e.g., to require overriding np.stack implicitly via np.expand_dims/np.concatenate instead).

Possible changes (1) or (2) would be good reasons for requiring end-users to make an intentional choice to enable __array_function__. Either of these changes would break everyone who uses the protocol. Concern about change (1) would be a particularly good reason to require an opt-in, because it's concerned about impact on users who don't use any new functionality.

Possible changes (3) or (4) would be annoying to users, but in my mind would not justify requiring an explicit opt-in. Both could be achieved without immediately breaking users by making use of a standard deprecation cycle, i.e., by still calling __array_function__ but issuing a FutureWarning. The main reason for requiring the explicit opt-in would be if we don't want to need to bother with a deprecation cycle.

To be clear, with the exception of change (3)(b), I don't think any of these are particularly likely or desirable outcomes. So in my personal opinion, I don't think we need the explicit opt-in. But I'm willing to make to go ahead with requiring the opt-in if that's what it takes to get everyone on board -- as long as we don't make the opt-in too onerous in a way that imposes hard limits on this protocol can be used. (I would be OK with an explicit Python API for enabling this feature that raises a FutureWarning when it's called, without any API for turning it off.)

Best,
Stephan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
In reply to this post by Nathaniel Smith
On Wed, Aug 29, 2018 at 1:38 AM Nathaniel Smith <[hidden email]> wrote:
The official Tensorflow wheels flat out lie about being manylinux
compatible, and the Tensorflow team has never talked to anyone about
how to fix this, they just upload them to PyPI and leave others get to
deal with the fallout [1]. That may well be what's best for their
users, I don't know. But stuff like this is normal, it happens all the
time, and if someone does it with __array_function__ then we have no
leverage.

Yes, this is a good example of how library authors sometimes break rules imposed by their dependencies when not enforced by technical means.

I'd like to think the TensorFlow team was making an intentional choice here and is willing to live with the consequence of their decision. Unfortunately right now many of those consequences are imposed on others, so this was certainly an inconsiderate choice. I don't know if they were aware of these costs that they are imposing on the rest of the Python ecosystem. I don't see any mention in PEP-513 about the consequences of lying about manylinux compatibility -- even though arguably such warnings should not be necessary.

One lesson we might take from this is documentation is essential: we should have more prominent warnings in NEP-18 (given that it's also likely to serve as a form of user-facing docs) about how experimental this protocol is, and about the specific types of changes we anticipate in the future (e.g., see my other email).

I also hope that Python packaging maintainers would not hesitate to break this abuse of the manylinux1 tag, e.g., if they wanted to add compatibility checks on wheels when uploaded to PyPI.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Charles R Harris
In reply to this post by Stephan Hoyer-2


On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":
http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


Skipping over all the discussion following this for brevity. I find myself in general agreement with Stephan and think we should go ahead and merge this. A few points:
  1. I don't think we should have an opt in environment variable, just put `__array_function__` out there with the understanding that it might change with experience. I don't think scores, let alone thousands, of folks are going to rush to take advantage of this, most likely the developers of the proposal will be the early adopters.
  2. I have a preference for implementing high level functions rather than low level, and fewer rather than more, with the choice driven by need. I think that will limit the amount of interdependence tangle and, in the long run, might serve to define an unofficial NumPy API.
  3. NumPy cannot manage all the projects that make use of the new option to keep them in sync, we have neither the time nor experience to do so, and historic attempts along those lines tended to die in obscurity. To the extent that we do so, it comes down to 2.
  4. I don't think this conflicts with Nathaniel's proposal, the main disagreement seems to over how to proceed and the selection of functions, see 2.
  5. The `__array_function_types__` idea looks interesting.
Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Charles R Harris


On Sun, Sep 2, 2018 at 12:58 PM Charles R Harris <[hidden email]> wrote:


On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":
http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


Skipping over all the discussion following this for brevity. I find myself in general agreement with Stephan and think we should go ahead and merge this. A few points:
  1. I don't think we should have an opt in environment variable, just put `__array_function__` out there with the understanding that it might change with experience. I don't think scores, let alone thousands, of folks are going to rush to take advantage of this, most likely the developers of the proposal will be the early adopters.
  2. I have a preference for implementing high level functions rather than low level, and fewer rather than more, with the choice driven by need. I think that will limit the amount of interdependence tangle and, in the long run, might serve to define an unofficial NumPy API.
  3. NumPy cannot manage all the projects that make use of the new option to keep them in sync, we have neither the time nor experience to do so, and historic attempts along those lines tended to die in obscurity. To the extent that we do so, it comes down to 2.
  4. I don't think this conflicts with Nathaniel's proposal, the main disagreement seems to over how to proceed and the selection of functions, see 2.
  5. The `__array_function_types__` idea looks interesting.

We might want to add an 'Experimental' status for NEPs, meaning accepted but subject to modification.

Chuck 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Charles R Harris
In reply to this post by Stephan Hoyer-2


On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":
http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


I've merged the PR. What next?

Chuck 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris <[hidden email]> wrote:
On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":
http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


I've merged the PR. What next?

Chuck 

I rolled back Chuck's merge of the "Acceptance" PR for this NEP because (1) it was not clear that if had reached consensus and (2) I still wanted to make a few changes based on this discussion.

I have now drafted these revisions to the NEP to clarify its stance around backwards compatibility, and the type of the "types" argument: https://github.com/numpy/numpy/pull/11943
 
I have *not* included any form of the explicit "end-user opt-in" requested by Nathaniel. I don't think it is warranted based on the scope of anticipated future changes; see my earlier post [1] for details. Nobody has seriously suggested that we would release this functionality and later eliminate the ability to override NumPy functions entirely in the future.

Of course, requiring an onerous explicit opt-in would be fine while this feature is initially under development, and until we are sure that checks for overriding arbitrary NumPy functions are fast enough to be generally viable. In particular, we should verify that __array_function__ does not meaningfully impact performance for major downstream components of the SciPy stack. But I think running standard benchmark suites (e.g., with ASV) and user testing of release candidates would be enough.

If you have still have major objections to this version of proposal, please raise them unambiguously -- preferably with an formal veto [2].

Best,
Stephan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Marten van Kerkwijk
Thanks for adding the clarifications. They read well to me, and I think make clear to a project like astropy what to expect (and I expect we'll be use it!).  -- Marten

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

einstein.edison
In reply to this post by Stephan Hoyer-2
On Thursday, Sep 13, 2018 at 7:30 PM, Stephan Hoyer <[hidden email]> wrote:
On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris <[hidden email]> wrote:
On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


I've merged the PR. What next?

Chuck 

I rolled back Chuck's merge of the "Acceptance" PR for this NEP because (1) it was not clear that if had reached consensus and (2) I still wanted to make a few changes based on this discussion.

I have now drafted these revisions to the NEP to clarify its stance around backwards compatibility, and the type of the "types" argument: https://github.com/numpy/numpy/pull/11943
 
I have *not* included any form of the explicit "end-user opt-in" requested by Nathaniel. I don't think it is warranted based on the scope of anticipated future changes; see my earlier post [1] for details. Nobody has seriously suggested that we would release this functionality and later eliminate the ability to override NumPy functions entirely in the future.

Of course, requiring an onerous explicit opt-in would be fine while this feature is initially under development, and until we are sure that checks for overriding arbitrary NumPy functions are fast enough to be generally viable. In particular, we should verify that __array_function__ does not meaningfully impact performance for major downstream components of the SciPy stack. But I think running standard benchmark suites (e.g., with ASV) and user testing of release candidates would be enough.

If you have still have major objections to this version of proposal, please raise them unambiguously -- preferably with an formal veto [2].

Best,
Stephan

_______________________________________________ 
NumPy-Discussion mailing list 
https://mail.python.org/mailman/listinfo/numpy-discussion 

+1 from me. The edits are readable and clean, and a good compromise.

Best Regards,
Hameer Abbasi

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
In reply to this post by Stephan Hoyer-2
On Thu, Sep 13, 2018 at 10:30 AM Stephan Hoyer <[hidden email]> wrote:
On Sat, Sep 8, 2018 at 7:12 PM Charles R Harris <[hidden email]> wrote:
On Wed, Aug 1, 2018 at 6:27 PM Stephan Hoyer <[hidden email]> wrote:
I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions":
http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP.

If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details.


I've merged the PR. What next?

Chuck 

I rolled back Chuck's merge of the "Acceptance" PR for this NEP because (1) it was not clear that if had reached consensus and (2) I still wanted to make a few changes based on this discussion.

I have now drafted these revisions to the NEP to clarify its stance around backwards compatibility, and the type of the "types" argument: https://github.com/numpy/numpy/pull/11943
 
I have *not* included any form of the explicit "end-user opt-in" requested by Nathaniel. I don't think it is warranted based on the scope of anticipated future changes; see my earlier post [1] for details. Nobody has seriously suggested that we would release this functionality and later eliminate the ability to override NumPy functions entirely in the future.

Of course, requiring an onerous explicit opt-in would be fine while this feature is initially under development, and until we are sure that checks for overriding arbitrary NumPy functions are fast enough to be generally viable. In particular, we should verify that __array_function__ does not meaningfully impact performance for major downstream components of the SciPy stack. But I think running standard benchmark suites (e.g., with ASV) and user testing of release candidates would be enough.

If you have still have major objections to this version of proposal, please raise them unambiguously -- preferably with an formal veto [2].

Again, if anyone (yes, I'm looking at you, Nathaniel!) still has objections please speak up soon.

If I don't hear anything, I'm going submit a PR to mark this proposal as "Accepted" tomorrow (one week after these latest revisions).

As Chuck noted, "Tentatively accepted" or "Experimental" would probably be a better status, but that currently isn't one of our options for NEPs. In any case, I would love to move on from debate to implementation!

As a side note, I recently noticed a CuPy pull request (https://github.com/cupy/cupy/pull/1650) to add __array_function__. It's a testament to our proposed interface that the full PR is 9 lines of code without tests.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Nathaniel Smith
In reply to this post by Stephan Hoyer-2
On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer <[hidden email]> wrote:
> I have now drafted these revisions to the NEP to clarify its stance around
> backwards compatibility, and the type of the "types" argument:
> https://github.com/numpy/numpy/pull/11943

Okay, so this is a pretty substantial change! Before, the NEP's stance
was "we might change anything, at any time, without any warning",
which of course makes it easier to accept the NEP (since we can always
back out), but was also so different from our normal rules that it
seemed important to make sure people weren't using it without
realizing. Now it actually makes a commitment: to not regress on what
functions can be overloaded (though the details might change), and
commits to an abbreviated-but-nonzero deprecation process when we
change things. I get the impression that this is closer to what the
authors were intending in the first place, so that's good! I would
probably have kept the noisy warning and zero commitments for one
release anyway, because IMO it's not a big deal and it rarely hurts to
hedge bets and gather data. But on reflection, I think I am OK with
this level of commitment if that's what y'all want to go for. (After
all, it's not really any stronger than NEP 22's high-level plan.) So,
+0.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
On Thu, Sep 20, 2018 at 2:33 AM Nathaniel Smith <[hidden email]> wrote:
On Thu, Sep 13, 2018 at 10:30 AM, Stephan Hoyer <[hidden email]> wrote:
> I have now drafted these revisions to the NEP to clarify its stance around
> backwards compatibility, and the type of the "types" argument:
> https://github.com/numpy/numpy/pull/11943

Okay, so this is a pretty substantial change! Before, the NEP's stance
was "we might change anything, at any time, without any warning",
which of course makes it easier to accept the NEP (since we can always
back out), but was also so different from our normal rules that it
seemed important to make sure people weren't using it without
realizing. Now it actually makes a commitment: to not regress on what
functions can be overloaded (though the details might change), and
commits to an abbreviated-but-nonzero deprecation process when we
change things. I get the impression that this is closer to what the
authors were intending in the first place, so that's good! I would
probably have kept the noisy warning and zero commitments for one
release anyway, because IMO it's not a big deal and it rarely hurts to
hedge bets and gather data. But on reflection, I think I am OK with
this level of commitment if that's what y'all want to go for. (After
all, it's not really any stronger than NEP 22's high-level plan.) So,
+0.

Nathaniel -- thanks for your critical reviews here, and your open-mindedness!

I've gone ahead and merged the PR to mark the NEP as accepted. Let's get started on the fun part of implementation!

Cheers,
Stephan

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Proposal to accept NEP-18, __array_function__ protocol

Stephan Hoyer-2
On Thu, Sep 20, 2018 at 2:05 PM Stephan Hoyer <[hidden email]> wrote:
I've gone ahead and merged the PR to mark the NEP as accepted. Let's get started on the fun part of implementation!

Cheers,
Stephan

I have started implementing NEP-18 in a pull request: https://github.com/numpy/numpy/pull/12005

I propose a multi-step process for implementation:
1. Write a correctly working version of the core __array_function__ machinery in pure Python, adapted from the example implementation in the NEP.
2. Rewrite bottlenecks in C.
3. Implement overrides for various NumPy functions.

I think the first step is mostly complete, but I'm particularly looking for help with the second and third steps, which should be able to happen incrementally and in parallel.

Cheers,
Stephan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
1234