Deprecate numpy.dual?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Deprecate numpy.dual?

Warren Weckesser-2
In response to some work on improving the documentation of `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard suggested that the documentation of the module `numpy.dual` should also be improved.  When I mentioned this suggestion in the community meeting on December 11, it was suggested that we should probably deprecate `numpy.dual`.

I think some current NumPy developers (myself included at the time the topic came up) are unfamiliar with the history and purpose of this module, so I spent some time reading code and github issues and wrote up some notes.  These notes are available at

    https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md

If you are not familiar with `numpy.dual`, you might find those notes useful.

Now that I know a bit more about `numpy.dual`, I'm not sure it should be deprecated.  It provides a hook for other libraries to selectively replace the use of the exposed functions in internal NumPy code, so if a library has a better version of, say, `linalg.eigh`, it can configure `numpy.dual` to use its version. Then, for example, NumPy multivariate normal distribution code could benefit from the use of that library's version of `eigh`.

The NumPy documentation of `numpy.dual` refers specifically to SciPy, but it could be used by any library.  Does anyone know if any other libraries use `register_func` to put their functions into the `numpy.dual` namespace?

SciPy currently registers some functions, but there is an open issue in which it is proposed that SciPy no longer register its functions with `numpy.dual`:

    https://github.com/scipy/scipy/issues/10441

This email is to start the discussion of the future of `numpy.dual`.
Some of the options:

1. Quietly continue the status quo.
2. Deprecate `numpy.dual`.
3. Spend time improving the documentation of this feature, and
   perhaps even expand the functions that are supported.

What do you think?  For those who were involved in the creation of `numpy.dual`: is it working out like you expected?  If not, is it worthwhile maintaining it?

Warren


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate numpy.dual?

Sebastian Berg
On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote:

> In response to some work on improving the documentation of
> `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard
> suggested that the documentation of the module `numpy.dual` should
> also be improved.  When I mentioned this suggestion in the community
> meeting on December 11, it was suggested that we should probably
> deprecate `numpy.dual`.
>
> I think some current NumPy developers (myself included at the time
> the topic came up) are unfamiliar with the history and purpose of
> this module, so I spent some time reading code and github issues and
> wrote up some notes.  These notes are available at
>
>    
> https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md
>
> If you are not familiar with `numpy.dual`, you might find those notes
> useful.
>
> Now that I know a bit more about `numpy.dual`, I'm not sure it should
> be deprecated.  It provides a hook for other libraries to selectively
> replace the use of the exposed functions in internal NumPy code, so
> if a library has a better version of, say, `linalg.eigh`, it can
> configure `numpy.dual` to use its version. Then, for example, NumPy
> multivariate normal distribution code could benefit from the use of
> that library's version of `eigh`.
That is in principle true, but I do not think we use `dual` at all
internally right now in numpy, and I doubt there is more than a hand
full uses out there.

Dual is an override mechanism for functionality on ndarrays implemented
also by numpy.

In either case, I still tend towards deprecation. It seems to have
issues and the main use case probably was to improve the situation when
NumPy was compiled without an optimized BLAS/LAPACK. That probably was
a common problem at some point, but I am not sure it is still an issue.

Overriding functionality with faster implementations is of course a
valid use-case and maybe `dual` is not a bad solution to the problem
[0]. But I think we should discuss this more generally with other
options. IMO deprecating this practically unused thing now does not
mean we cannot do something similar in the future.

- Sebastian


[0] It has its own namespace, so is opt-in for the end user. You can
only support a single backend at a time, although I am not sure that
matters too much. If overrides provide a function to override, it is
explicit to the end user as to what gets executed as well.


> The NumPy documentation of `numpy.dual` refers specifically to SciPy,
> but it could be used by any library.  Does anyone know if any other
> libraries use `register_func` to put their functions into the
> `numpy.dual` namespace?
>
> SciPy currently registers some functions, but there is an open issue
> in which it is proposed that SciPy no longer register its functions
> with `numpy.dual`:
>
>     https://github.com/scipy/scipy/issues/10441
>
> This email is to start the discussion of the future of `numpy.dual`.
> Some of the options:
>
> 1. Quietly continue the status quo.
> 2. Deprecate `numpy.dual`.
> 3. Spend time improving the documentation of this feature, and
>    perhaps even expand the functions that are supported.
>
> What do you think?  For those who were involved in the creation of
> `numpy.dual`: is it working out like you expected?  If not, is it
> worthwhile maintaining it?
>
> Warren
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate numpy.dual?

Warren Weckesser-2
On 1/3/20, Sebastian Berg <[hidden email]> wrote:

> On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote:
>> In response to some work on improving the documentation of
>> `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard
>> suggested that the documentation of the module `numpy.dual` should
>> also be improved.  When I mentioned this suggestion in the community
>> meeting on December 11, it was suggested that we should probably
>> deprecate `numpy.dual`.
>>
>> I think some current NumPy developers (myself included at the time
>> the topic came up) are unfamiliar with the history and purpose of
>> this module, so I spent some time reading code and github issues and
>> wrote up some notes.  These notes are available at
>>
>>
>> https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md
>>
>> If you are not familiar with `numpy.dual`, you might find those notes
>> useful.
>>
>> Now that I know a bit more about `numpy.dual`, I'm not sure it should
>> be deprecated.  It provides a hook for other libraries to selectively
>> replace the use of the exposed functions in internal NumPy code, so
>> if a library has a better version of, say, `linalg.eigh`, it can
>> configure `numpy.dual` to use its version. Then, for example, NumPy
>> multivariate normal distribution code could benefit from the use of
>> that library's version of `eigh`.
>
> That is in principle true, but I do not think we use `dual` at all
> internally right now in numpy, and I doubt there is more than a hand
> full uses out there.

In the notes, I listed the internal uses of `numpy.dual` within numpy
that I found:

1. In the code that generates random variates from the multivariate normal
   distribution, one of `svd`, `eigh` or `cholesky` are used from `numpy.dual`.
2. In `matrixlib/defmatrix.py`, the `.I` property of the `matrix` class
   uses either `inv` or `pinv` from `numpy.dual` to compute its value.
3. The window function `numpy.kaiser` uses `numpy.dual.i0`.


>
> Dual is an override mechanism for functionality on ndarrays implemented
> also by numpy.
>
> In either case, I still tend towards deprecation. It seems to have
> issues and the main use case probably was to improve the situation when
> NumPy was compiled without an optimized BLAS/LAPACK. That probably was
> a common problem at some point, but I am not sure it is still an issue.
>
> Overriding functionality with faster implementations is of course a
> valid use-case and maybe `dual` is not a bad solution to the problem
> [0]. But I think we should discuss this more generally with other
> options. IMO deprecating this practically unused thing now does not
> mean we cannot do something similar in the future.

It probably makes sense to have the general discussion before
deprecating `numpy.dual`--there is a (slim?) chance that `numpy.dual`
will turn out to be the best option.

Warren


>
> - Sebastian
>
>
> [0] It has its own namespace, so is opt-in for the end user. You can
> only support a single backend at a time, although I am not sure that
> matters too much. If overrides provide a function to override, it is
> explicit to the end user as to what gets executed as well.
>
>
>> The NumPy documentation of `numpy.dual` refers specifically to SciPy,
>> but it could be used by any library.  Does anyone know if any other
>> libraries use `register_func` to put their functions into the
>> `numpy.dual` namespace?
>>
>> SciPy currently registers some functions, but there is an open issue
>> in which it is proposed that SciPy no longer register its functions
>> with `numpy.dual`:
>>
>>     https://github.com/scipy/scipy/issues/10441
>>
>> This email is to start the discussion of the future of `numpy.dual`.
>> Some of the options:
>>
>> 1. Quietly continue the status quo.
>> 2. Deprecate `numpy.dual`.
>> 3. Spend time improving the documentation of this feature, and
>>    perhaps even expand the functions that are supported.
>>
>> What do you think?  For those who were involved in the creation of
>> `numpy.dual`: is it working out like you expected?  If not, is it
>> worthwhile maintaining it?
>>
>> Warren
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate numpy.dual?

Robert Kern-2
On Fri, Jan 3, 2020 at 11:55 AM Warren Weckesser <[hidden email]> wrote:
In the notes, I listed the internal uses of `numpy.dual` within numpy
that I found:

1. In the code that generates random variates from the multivariate normal
   distribution, one of `svd`, `eigh` or `cholesky` are used from `numpy.dual`.

This was always a bad idea (originally mine, FWIW). The variability in arithmetic introduced by using possibly-different LAPACK versions depending on whether or not scipy is installed (and how it was built) is not worth the very minor speed advantages. The covariance matrices are typically small. And where they are large (e.g. Gaussian processes), you should probably be doing your own decompositions instead of using `multivariate_normal()`.

--
Robert Kern

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Deprecate numpy.dual?

ralfgommers
In reply to this post by Warren Weckesser-2


On Fri, Jan 3, 2020 at 5:54 PM Warren Weckesser <[hidden email]> wrote:
On 1/3/20, Sebastian Berg <[hidden email]> wrote:
> On Fri, 2020-01-03 at 07:11 -0500, Warren Weckesser wrote:
>> In response to some work on improving the documentation of
>> `numpy.linalg` and how it compares to `scipy.linalg`, Kevin Sheppard
>> suggested that the documentation of the module `numpy.dual` should
>> also be improved.  When I mentioned this suggestion in the community
>> meeting on December 11, it was suggested that we should probably
>> deprecate `numpy.dual`.
>>
>> I think some current NumPy developers (myself included at the time
>> the topic came up) are unfamiliar with the history and purpose of
>> this module, so I spent some time reading code and github issues and
>> wrote up some notes.  These notes are available at
>>
>>
>> https://github.com/WarrenWeckesser/numpy-notes/blob/master/numpy-dual.md
>>
>> If you are not familiar with `numpy.dual`, you might find those notes
>> useful.
>>
>> Now that I know a bit more about `numpy.dual`, I'm not sure it should
>> be deprecated.  It provides a hook for other libraries to selectively
>> replace the use of the exposed functions in internal NumPy code, so
>> if a library has a better version of, say, `linalg.eigh`, it can
>> configure `numpy.dual` to use its version. Then, for example, NumPy
>> multivariate normal distribution code could benefit from the use of
>> that library's version of `eigh`.
>
> That is in principle true, but I do not think we use `dual` at all
> internally right now in numpy, and I doubt there is more than a hand
> full uses out there.

In the notes, I listed the internal uses of `numpy.dual` within numpy
that I found:

1. In the code that generates random variates from the multivariate normal
   distribution, one of `svd`, `eigh` or `cholesky` are used from `numpy.dual`.

Agree with Robert's comment on this: not a good idea.

2. In `matrixlib/defmatrix.py`, the `.I` property of the `matrix` class
   uses either `inv` or `pinv` from `numpy.dual` to compute its value.

Easy to get rid of, and not very interesting.

3. The window function `numpy.kaiser` uses `numpy.dual.i0`.

SciPy doesn't even register its special.i0. We should just get rid of all this.
 


>
> Dual is an override mechanism for functionality on ndarrays implemented
> also by numpy.
>
> In either case, I still tend towards deprecation. It seems to have
> issues and the main use case probably was to improve the situation when
> NumPy was compiled without an optimized BLAS/LAPACK. That probably was
> a common problem at some point, but I am not sure it is still an issue.

That's not an issue anymore, it's been years since that was popping up occasionally.

>
> Overriding functionality with faster implementations is of course a
> valid use-case and maybe `dual` is not a bad solution to the problem
> [0]. But I think we should discuss this more generally with other
> options. IMO deprecating this practically unused thing now does not
> mean we cannot do something similar in the future.

+1


It probably makes sense to have the general discussion before
deprecating `numpy.dual`--there is a (slim?) chance that `numpy.dual`
will turn out to be the best option.

I think it's *very* safe to say that's not the case. Let's just be pragmatic and not hedge on slim options.

Cheers,
Ralf

 

Warren


>
> - Sebastian
>
>
> [0] It has its own namespace, so is opt-in for the end user. You can
> only support a single backend at a time, although I am not sure that
> matters too much. If overrides provide a function to override, it is
> explicit to the end user as to what gets executed as well.
>
>
>> The NumPy documentation of `numpy.dual` refers specifically to SciPy,
>> but it could be used by any library.  Does anyone know if any other
>> libraries use `register_func` to put their functions into the
>> `numpy.dual` namespace?
>>
>> SciPy currently registers some functions, but there is an open issue
>> in which it is proposed that SciPy no longer register its functions
>> with `numpy.dual`:
>>
>>     https://github.com/scipy/scipy/issues/10441
>>
>> This email is to start the discussion of the future of `numpy.dual`.
>> Some of the options:
>>
>> 1. Quietly continue the status quo.
>> 2. Deprecate `numpy.dual`.
>> 3. Spend time improving the documentation of this feature, and
>>    perhaps even expand the functions that are supported.
>>
>> What do you think?  For those who were involved in the creation of
>> `numpy.dual`: is it working out like you expected?  If not, is it
>> worthwhile maintaining it?
>>
>> Warren
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion