Add sliding_window_view method to numpy

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Add sliding_window_view method to numpy

Zimmermann Klaus
Hello,

I would like to draw the attention of this list to PR #17394 [1] that
adds the implementation of a sliding window view to numpy.

Having a sliding window view in numpy is a longstanding open issue (cf
#7753 [2] from 2016). A brief summary of the discussions surrounding it
can be found in the description of the PR.

This PR implements a sliding window view based on stride tricks.
Following the discussion in issue #7753, a first implementation was
provided by Fanjin Zeng in PR #10771. After some discussion, that PR
stalled and I picked up the issue in the present PR #17394. It is based
on the first implementation, but follows the changed API as suggested by
Eric Wieser.

Code reviews have been provided by Bas van Beek, Stephen Hoyer, and Eric
Wieser. Sebastian Berg added the "62 - Python API" label.


Do you think this is suitable for inclusion in numpy?

Do you consider the PR ready?

Do you have suggestions or requests?


Thanks for your time and consideration!
Klaus


[1] https://github.com/numpy/numpy/pull/17394
[2] https://github.com/numpy/numpy/issues/7753
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Sebastian Berg
On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
Hello,

I would like to draw the attention of this list to PR #17394 [1] that
adds the implementation of a sliding window view to numpy.


Hi,

thanks for working on this and driving going forward. I like the choice of a minimal API.  I have pasted the doc-string (html, hope that works fine) below to allow a quicker idea of what is being proposed.  To me it looks good! (I wonder if we need `subok`, but I guess we probably do.)

Cheers,

Sebastian


numpy.sliding_window_view

numpy.sliding_window_view(xwindow_shapeaxis=None*subok=Falsewriteable=False)

Create a sliding window view into the array with the given window shape.

Creates a sliding window view of the N dimensional array with the given window shape. Window slides across each dimension of the array and extract a subsets of the array at any window position.

Parameters
x : array_like

Array to create the sliding window view from.

window_shape : int or tuple of int

Size of window over each axis that takes part in the sliding window. If axis is not present, must have same length as the number of input array dimensions. Single integers i are treated as if they were the tuple (i,).

axis : int or tuple of int, optional

Axis or axes along which the sliding window is applied. By default, the sliding window is applied to all axes and window_shape[i] will refer to axis i of x. If axis is given as a tuple of intwindow_shape[i] will refer to the axis axis[i] of x. Single integers i are treated as if they were the tuple (i,).

subok : bool, optional

If True, sub-classes will be passed-through, otherwise the returned array will be forced to be a base-class array (default).

writeable : bool, optional

When true, allow writing to the returned view. The default is false, as this should be used with caution: the returned view contains the same memory location multiple times, so writing to one location will cause others to change.

Returns
view : ndarray

Sliding window view of the array. The sliding window dimensions are inserted at the end, and the original dimensions are trimmed as required by the size of the sliding window.

That is, view.shape = x_shape_trimmed + window_shape, where x_shape_trimmed is x.shape with every entry reduced by one less than the corresponding window size.

See also

lib.stride_tricks.as_strided

Create a view into the array with the given shape and strides.

broadcast_to

broadcast an array to a given shape.

Notes

For some cases there may be more efficient approaches to calculate transformations across multi-dimensional arrays, for instance scipy.signal.fftconvolve, where combining the iterating step with the calculation itself while storing partial results can result in significant speedups.

Examples

>>> x = np.arange(6)
>>> x.shape
(6,)
>>> v = np.sliding_window_view(x, 3)
>>> v.shape
(4, 3)
>>> v
array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4],
       [3, 4, 5]])

This also works in more dimensions, e.g.

>>> i, j = np.ogrid[:3, :4]
>>> x = 10*i + j
>>> x.shape
(3, 4)
>>> x
array([[ 0,  1,  2,  3],
       [10, 11, 12, 13],
       [20, 21, 22, 23]])
>>> shape = (2,2)
>>> v = np.sliding_window_view(x, shape)
>>> v.shape
(2, 3, 2, 2)
>>> v
array([[[[ 0,  1],
         [10, 11]],
        [[ 1,  2],
         [11, 12]],
        [[ 2,  3],
         [12, 13]]],
       [[[10, 11],
         [20, 21]],
        [[11, 12],
         [21, 22]],
        [[12, 13],
         [22, 23]]]])

The axis can be specified explicitly:

>>> v = np.sliding_window_view(x, 3, 0)
>>> v.shape
(1, 4, 3)
>>> v
array([[[ 0, 10, 20],
        [ 1, 11, 21],
        [ 2, 12, 22],
        [ 3, 13, 23]]])

The same axis can be used several times. In that case, every use reduces the corresponding original dimension:

>>> v = np.sliding_window_view(x, (2, 3), (1, 1))
>>> v.shape
(3, 1, 2, 3)
>>> v
array([[[[ 0,  1,  2],
         [ 1,  2,  3]]],
       [[[10, 11, 12],
         [11, 12, 13]]],
       [[[20, 21, 22],
         [21, 22, 23]]]])

Combining with stepped slicing (::step), this can be used to take sliding views which skip elements:

>>> x = np.arange(7)
>>> np.sliding_window_view(x, 5)[:, ::2]
array([[0, 2, 4],
       [1, 3, 5],
       [2, 4, 6]])

or views which move by multiple elements

>>> x = np.arange(7)
>>> np.sliding_window_view(x, 3)[::2, :]
array([[0, 1, 2],
       [2, 3, 4],
       [4, 5, 6]])





Having a sliding window view in numpy is a longstanding open issue (cf
#7753 [2] from 2016). A brief summary of the discussions surrounding it
can be found in the description of the PR.

This PR implements a sliding window view based on stride tricks.
Following the discussion in issue #7753, a first implementation was
provided by Fanjin Zeng in PR #10771. After some discussion, that PR
stalled and I picked up the issue in the present PR #17394. It is based
on the first implementation, but follows the changed API as suggested by
Eric Wieser.

Code reviews have been provided by Bas van Beek, Stephen Hoyer, and Eric
Wieser. Sebastian Berg added the "62 - Python API" label.


Do you think this is suitable for inclusion in numpy?

Do you consider the PR ready?

Do you have suggestions or requests?


Thanks for your time and consideration!
Klaus


_______________________________________________
NumPy-Discussion mailing list



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Juan Nunez-Iglesias-2
Looks gorgeous, thank you to all who worked on the implementation, API, and review, and thank you Sebastian for saving me a click! ūüėā

On 13 Oct 2020, at 2:25 am, Sebastian Berg <[hidden email]> wrote:

On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
Hello,

I would like to draw the attention of this list to PR #17394 [1] that
adds the implementation of a sliding window view to numpy.


Hi,

thanks for working on this and driving going forward. I like the choice of a minimal API.  I have pasted the doc-string (html, hope that works fine) below to allow a quicker idea of what is being proposed.  To me it looks good! (I wonder if we need `subok`, but I guess we probably do.)

Cheers,

Sebastian


numpy.sliding_window_view

numpy.sliding_window_view(xwindow_shapeaxis=None*subok=Falsewriteable=False)

Create a sliding window view into the array with the given window shape.

Creates a sliding window view of the N dimensional array with the given window shape. Window slides across each dimension of the array and extract a subsets of the array at any window position.

Parameters
x : array_like
Array to create the sliding window view from.
window_shape : int or tuple of int
Size of window over each axis that takes part in the sliding window. If axis is not present, must have same length as the number of input array dimensions. Single integers i are treated as if they were the tuple (i,).
axis : int or tuple of int, optional
Axis or axes along which the sliding window is applied. By default, the sliding window is applied to all axes and window_shape[i] will refer to axis i of x. If axis is given as a tuple of intwindow_shape[i] will refer to the axis axis[i] of x. Single integers i are treated as if they were the tuple (i,).
subok : bool, optional
If True, sub-classes will be passed-through, otherwise the returned array will be forced to be a base-class array (default).
writeable : bool, optional
When true, allow writing to the returned view. The default is false, as this should be used with caution: the returned view contains the same memory location multiple times, so writing to one location will cause others to change.
Returns
view : ndarray
Sliding window view of the array. The sliding window dimensions are inserted at the end, and the original dimensions are trimmed as required by the size of the sliding window.
That is, view.shape = x_shape_trimmed + window_shape, where x_shape_trimmed is x.shape with every entry reduced by one less than the corresponding window size.
See also
lib.stride_tricks.as_strided
Create a view into the array with the given shape and strides.
broadcast_to
broadcast an array to a given shape.

Notes

For some cases there may be more efficient approaches to calculate transformations across multi-dimensional arrays, for instance scipy.signal.fftconvolve, where combining the iterating step with the calculation itself while storing partial results can result in significant speedups.

Examples

>>> x = np.arange(6)
>>> x.shape
(6,)
>>> v = np.sliding_window_view(x, 3)
>>> v.shape
(4, 3)
>>> v
array([[0, 1, 2],
       [1, 2, 3],
       [2, 3, 4],
       [3, 4, 5]])

This also works in more dimensions, e.g.

>>> i, j = np.ogrid[:3, :4]
>>> x = 10*i + j
>>> x.shape
(3, 4)
>>> x
array([[ 0,  1,  2,  3],
       [10, 11, 12, 13],
       [20, 21, 22, 23]])
>>> shape = (2,2)
>>> v = np.sliding_window_view(x, shape)
>>> v.shape
(2, 3, 2, 2)
>>> v
array([[[[ 0,  1],
         [10, 11]],
        [[ 1,  2],
         [11, 12]],
        [[ 2,  3],
         [12, 13]]],
       [[[10, 11],
         [20, 21]],
        [[11, 12],
         [21, 22]],
        [[12, 13],
         [22, 23]]]])

The axis can be specified explicitly:

>>> v = np.sliding_window_view(x, 3, 0)
>>> v.shape
(1, 4, 3)
>>> v
array([[[ 0, 10, 20],
        [ 1, 11, 21],
        [ 2, 12, 22],
        [ 3, 13, 23]]])

The same axis can be used several times. In that case, every use reduces the corresponding original dimension:

>>> v = np.sliding_window_view(x, (2, 3), (1, 1))
>>> v.shape
(3, 1, 2, 3)
>>> v
array([[[[ 0,  1,  2],
         [ 1,  2,  3]]],
       [[[10, 11, 12],
         [11, 12, 13]]],
       [[[20, 21, 22],
         [21, 22, 23]]]])

Combining with stepped slicing (::step), this can be used to take sliding views which skip elements:

>>> x = np.arange(7)
>>> np.sliding_window_view(x, 5)[:, ::2]
array([[0, 2, 4],
       [1, 3, 5],
       [2, 4, 6]])

or views which move by multiple elements

>>> x = np.arange(7)
>>> np.sliding_window_view(x, 3)[::2, :]
array([[0, 1, 2],
       [2, 3, 4],
       [4, 5, 6]])





Having a sliding window view in numpy is a longstanding open issue (cf
#7753 [2] from 2016). A brief summary of the discussions surrounding it
can be found in the description of the PR.

This PR implements a sliding window view based on stride tricks.
Following the discussion in issue #7753, a first implementation was
provided by Fanjin Zeng in PR #10771. After some discussion, that PR
stalled and I picked up the issue in the present PR #17394. It is based
on the first implementation, but follows the changed API as suggested by
Eric Wieser.

Code reviews have been provided by Bas van Beek, Stephen Hoyer, and Eric
Wieser. Sebastian Berg added the "62 - Python API" label.


Do you think this is suitable for inclusion in numpy?

Do you consider the PR ready?

Do you have suggestions or requests?


Thanks for your time and consideration!
Klaus


_______________________________________________
NumPy-Discussion mailing list


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Zimmermann Klaus
In reply to this post by Sebastian Berg
Hi,

thanks, Sebastian! Since `sliding_window_view` is essentially a small
function determining the parameters with which to call `as_strided`, it
makes sense to me to keep the essential `as_strided` parameters, `subok`
among them. But if somebody with a deeper insight into numpy is
convinced this should go, I have no problem with it.

Cheers
Klaus


On 12/10/2020 17:25, Sebastian Berg wrote:

> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>> Hello,
>>
>> I would like to draw the attention of this list to PR #17394 [1] that
>> adds the implementation of a sliding window view to numpy.
>>
>
> Hi,
>
> thanks for working on this and driving going forward. I like the choice
> of a minimal API.  I have pasted the doc-string (html, hope that works
> fine) below to allow a quicker idea of what is being proposed.  To me it
> looks good! (I wonder if we need `subok`, but I guess we probably do.)
>
> Cheers,
>
> Sebastian
>
>
>   numpy.sliding_window_view<https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.sliding_window_view.html#numpy-sliding-window-view>
>
>
>   |numpy.||sliding_window_view|(/x/,¬†/window_shape/,¬†/axis=None/,¬†/*/,¬†/subok=False/,¬†/writeable=False/)<https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.sliding_window_view.html#numpy.sliding_window_view>
>
>       Create a sliding window view into the array with the given window
>       shape.
>
>       Creates a sliding window view of the N dimensional array with the
>       given window shape. Window slides across each dimension of the
>       array and extract a subsets of the array at any window position.
>
>       Parameters
>
>           x :¬†array_like
>
>               Array to create the sliding window view from.
>
>           window_shape¬†:¬†int or tuple of int
>
>               Size of window over each axis that takes part in the
>               sliding window. If¬†/axis/¬†is not present, must have same
>               length as the number of input array dimensions. Single
>               integers¬†/i/¬†are treated as if they were the tuple¬†/(i,)/.
>
>           axis¬†:¬†int or tuple of int, optional
>
>               Axis or axes along which the sliding window is applied. By
>               default, the sliding window is applied to all axes
>               and¬†/window_shape[i]/¬†will refer to axis¬†/i/¬†of¬†/x/.
>               If¬†/axis/¬†is given as a¬†/tuple of
>               int/,¬†/window_shape[i]/¬†will refer to the
>               axis¬†/axis[i]/¬†of¬†/x/. Single integers¬†/i/¬†are treated as
>               if they were the tuple¬†/(i,)/.
>
>           subok¬†:¬†bool, optional
>
>               If True, sub-classes will be passed-through, otherwise the
>               returned array will be forced to be a base-class array
>               (default).
>
>           writeable¬†:¬†bool, optional
>
>               When true, allow writing to the returned view. The default
>               is false, as this should be used with caution: the
>               returned view contains the same memory location multiple
>               times, so writing to one location will cause others to change.
>
>       Returns
>
>           view¬†:¬†ndarray
>
>               Sliding window view of the array. The sliding window
>               dimensions are inserted at the end, and the original
>               dimensions are trimmed as required by the size of the
>               sliding window.
>
>               That is,¬†|view.shape¬†=¬†x_shape_trimmed¬†+¬†window_shape|,
>               where¬†|x_shape_trimmed|¬†is¬†|x.shape|¬†with every entry
>               reduced by one less than the corresponding window size.
>
>       See also
>
>       |lib.stride_tricks.as_strided|
>       <https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.lib.stride_tricks.as_strided.html#numpy.lib.stride_tricks.as_strided>
>
>           Create a view into the array with the given shape and strides.
>
>       |broadcast_to|
>       <https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.broadcast_to.html#numpy.broadcast_to>
>
>           broadcast an array to a given shape.
>
>       Notes
>
>       For some cases there may be more efficient approaches to calculate
>       transformations across multi-dimensional arrays, for
>       instance¬†|scipy.signal.fftconvolve|
>       <https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.fftconvolve.html#scipy.signal.fftconvolve>,
>       where combining the iterating step with the calculation itself
>       while storing partial results can result in significant speedups.
>
>       Examples
>
>       >>> x = np.arange(6)
>       >>> x.shape
>       (6,)
>       >>> v = np.sliding_window_view(x, 3)
>       >>> v.shape
>       (4, 3)
>       >>> v
>       array([[0, 1, 2],
>       [1, 2, 3],
>       [2, 3, 4],
>       [3, 4, 5]])
>
>       This also works in more dimensions, e.g.
>
>       >>> i, j = np.ogrid[:3, :4]
>       >>> x = 10*i + j
>       >>> x.shape
>       (3, 4)
>       >>> x
>       array([[ 0, 1, 2, 3],
>       [10, 11, 12, 13],
>       [20, 21, 22, 23]])
>       >>> shape = (2,2)
>       >>> v = np.sliding_window_view(x, shape)
>       >>> v.shape
>       (2, 3, 2, 2)
>       >>> v
>       array([[[[ 0, 1],
>       [10, 11]],
>       [[ 1, 2],
>       [11, 12]],
>       [[ 2, 3],
>       [12, 13]]],
>       [[[10, 11],
>       [20, 21]],
>       [[11, 12],
>       [21, 22]],
>       [[12, 13],
>       [22, 23]]]])
>
>       The axis can be specified explicitly:
>
>       >>> v = np.sliding_window_view(x, 3, 0)
>       >>> v.shape
>       (1, 4, 3)
>       >>> v
>       array([[[ 0, 10, 20],
>       [ 1, 11, 21],
>       [ 2, 12, 22],
>       [ 3, 13, 23]]])
>
>       The same axis can be used several times. In that case, every use
>       reduces the corresponding original dimension:
>
>       >>> v = np.sliding_window_view(x, (2, 3), (1, 1))
>       >>> v.shape
>       (3, 1, 2, 3)
>       >>> v
>       array([[[[ 0, 1, 2],
>       [ 1, 2, 3]]],
>       [[[10, 11, 12],
>       [11, 12, 13]]],
>       [[[20, 21, 22],
>       [21, 22, 23]]]])
>
>       Combining with stepped slicing (/::step/), this can be used to
>       take sliding views which skip elements:
>
>       >>> x = np.arange(7)
>       >>> np.sliding_window_view(x, 5)[:, ::2]
>       array([[0, 2, 4],
>       [1, 3, 5],
>       [2, 4, 6]])
>
>       or views which move by multiple elements
>
>       >>> x = np.arange(7)
>       >>> np.sliding_window_view(x, 3)[::2, :]
>       array([[0, 1, 2],
>       [2, 3, 4],
>       [4, 5, 6]])
>
>
>
>
>
>> Having a sliding window view in numpy is a longstanding open issue (cf
>> #7753 [2] from 2016). A brief summary of the discussions surrounding it
>> can be found in the description of the PR.
>>
>> This PR implements a sliding window view based on stride tricks.
>> Following the discussion in issue #7753, a first implementation was
>> provided by Fanjin Zeng in PR #10771. After some discussion, that PR
>> stalled and I picked up the issue in the present PR #17394. It is based
>> on the first implementation, but follows the changed API as suggested by
>> Eric Wieser.
>>
>> Code reviews have been provided by Bas van Beek, Stephen Hoyer, and Eric
>> Wieser. Sebastian Berg added the "62 - Python API" label.
>>
>>
>> Do you think this is suitable for inclusion in numpy?
>>
>> Do you consider the PR ready?
>>
>> Do you have suggestions or requests?
>>
>>
>> Thanks for your time and consideration!
>> Klaus
>>
>>
>> [1] https://github.com/numpy/numpy/pull/17394
>> <https://github.com/numpy/numpy/pull/17394>
>> [2] https://github.com/numpy/numpy/issues/7753
>> <https://github.com/numpy/numpy/issues/7753>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email] <mailto:[hidden email]>
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>> <https://mail.python.org/mailman/listinfo/numpy-discussion>
>>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Zimmermann Klaus
In reply to this post by Juan Nunez-Iglesias-2
Hi,

thanks, Juan, you are very kind!

Let's hope we can get this in soon :)

Cheers
Klaus

On 13/10/2020 01:18, Juan Nunez-Iglesias wrote:

> Looks gorgeous, thank you to all who worked on the implementation, API,
> and review, and thank you Sebastian for saving me a click! ūüėā
>
>> On 13 Oct 2020, at 2:25 am, Sebastian Berg <[hidden email]
>> <mailto:[hidden email]>> wrote:
>>
>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>>> Hello,
>>>
>>> I would like to draw the attention of this list to PR #17394 [1] that
>>> adds the implementation of a sliding window view to numpy.
>>>
>>
>> Hi,
>>
>> thanks for working on this and driving going forward. I like the
>> choice of a minimal API.  I have pasted the doc-string (html, hope
>> that works fine) below to allow a quicker idea of what is being
>> proposed.  To me it looks good! (I wonder if we need `subok`, but I
>> guess we probably do.)
>>
>> Cheers,
>>
>> Sebastian
>>
>>
>>   numpy.sliding_window_view<https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.sliding_window_view.html#numpy-sliding-window-view>
>>
>>
>>   |numpy.||sliding_window_view|(/x/,¬†/window_shape/,¬†/axis=None/,¬†/*/,¬†/subok=False/,¬†/writeable=False/)<https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.sliding_window_view.html#numpy.sliding_window_view>
>>
>>       Create a sliding window view into the array with the given
>>       window shape.
>>
>>       Creates a sliding window view of the N dimensional array with
>>       the given window shape. Window slides across each dimension of
>>       the array and extract a subsets of the array at any window position.
>>
>>       Parameters
>>
>>           x :¬†array_like
>>               Array to create the sliding window view from.
>>           window_shape¬†:¬†int or tuple of int
>>               Size of window over each axis that takes part in the
>>               sliding window. If¬†/axis/¬†is not present, must have same
>>               length as the number of input array dimensions. Single
>>               integers¬†/i/¬†are treated as if they were the tuple¬†/(i,)/.
>>           axis¬†:¬†int or tuple of int, optional
>>               Axis or axes along which the sliding window is applied.
>>               By default, the sliding window is applied to all axes
>>               and¬†/window_shape[i]/¬†will refer to axis¬†/i/¬†of¬†/x/.
>>               If¬†/axis/¬†is given as a¬†/tuple of
>>               int/,¬†/window_shape[i]/¬†will refer to the
>>               axis¬†/axis[i]/¬†of¬†/x/. Single integers¬†/i/¬†are treated
>>               as if they were the tuple¬†/(i,)/.
>>           subok¬†:¬†bool, optional
>>               If True, sub-classes will be passed-through, otherwise
>>               the returned array will be forced to be a base-class
>>               array (default).
>>           writeable¬†:¬†bool, optional
>>               When true, allow writing to the returned view. The
>>               default is false, as this should be used with caution:
>>               the returned view contains the same memory location
>>               multiple times, so writing to one location will cause
>>               others to change.
>>
>>       Returns
>>
>>           view¬†:¬†ndarray
>>               Sliding window view of the array. The sliding window
>>               dimensions are inserted at the end, and the original
>>               dimensions are trimmed as required by the size of the
>>               sliding window.
>>               That is,¬†|view.shape¬†=¬†x_shape_trimmed¬†+¬†window_shape|,
>>               where¬†|x_shape_trimmed|¬†is¬†|x.shape|¬†with every entry
>>               reduced by one less than the corresponding window size.
>>
>>       See also
>>
>>       |lib.stride_tricks.as_strided|
>>       <https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.lib.stride_tricks.as_strided.html#numpy.lib.stride_tricks.as_strided>
>>           Create a view into the array with the given shape and strides.
>>       |broadcast_to|
>>       <https://16171-908607-gh.circle-artifacts.com/0/doc/build/html/reference/generated/numpy.broadcast_to.html#numpy.broadcast_to>
>>           broadcast an array to a given shape.
>>
>>       Notes
>>
>>       For some cases there may be more efficient approaches to
>>       calculate transformations across multi-dimensional arrays, for
>>       instance¬†|scipy.signal.fftconvolve|
>>       <https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.fftconvolve.html#scipy.signal.fftconvolve>,
>>       where combining the iterating step with the calculation itself
>>       while storing partial results can result in significant speedups.
>>
>>       Examples
>>
>>       >>> x = np.arange(6)
>>       >>> x.shape
>>       (6,)
>>       >>> v = np.sliding_window_view(x, 3)
>>       >>> v.shape
>>       (4, 3)
>>       >>> v
>>       array([[0, 1, 2],
>>       [1, 2, 3],
>>       [2, 3, 4],
>>       [3, 4, 5]])
>>
>>       This also works in more dimensions, e.g.
>>
>>       >>> i, j = np.ogrid[:3, :4]
>>       >>> x = 10*i + j
>>       >>> x.shape
>>       (3, 4)
>>       >>> x
>>       array([[ 0, 1, 2, 3],
>>       [10, 11, 12, 13],
>>       [20, 21, 22, 23]])
>>       >>> shape = (2,2)
>>       >>> v = np.sliding_window_view(x, shape)
>>       >>> v.shape
>>       (2, 3, 2, 2)
>>       >>> v
>>       array([[[[ 0, 1],
>>       [10, 11]],
>>       [[ 1, 2],
>>       [11, 12]],
>>       [[ 2, 3],
>>       [12, 13]]],
>>       [[[10, 11],
>>       [20, 21]],
>>       [[11, 12],
>>       [21, 22]],
>>       [[12, 13],
>>       [22, 23]]]])
>>
>>       The axis can be specified explicitly:
>>
>>       >>> v = np.sliding_window_view(x, 3, 0)
>>       >>> v.shape
>>       (1, 4, 3)
>>       >>> v
>>       array([[[ 0, 10, 20],
>>       [ 1, 11, 21],
>>       [ 2, 12, 22],
>>       [ 3, 13, 23]]])
>>
>>       The same axis can be used several times. In that case, every use
>>       reduces the corresponding original dimension:
>>
>>       >>> v = np.sliding_window_view(x, (2, 3), (1, 1))
>>       >>> v.shape
>>       (3, 1, 2, 3)
>>       >>> v
>>       array([[[[ 0, 1, 2],
>>       [ 1, 2, 3]]],
>>       [[[10, 11, 12],
>>       [11, 12, 13]]],
>>       [[[20, 21, 22],
>>       [21, 22, 23]]]])
>>
>>       Combining with stepped slicing (/::step/), this can be used to
>>       take sliding views which skip elements:
>>
>>       >>> x = np.arange(7)
>>       >>> np.sliding_window_view(x, 5)[:, ::2]
>>       array([[0, 2, 4],
>>       [1, 3, 5],
>>       [2, 4, 6]])
>>
>>       or views which move by multiple elements
>>
>>       >>> x = np.arange(7)
>>       >>> np.sliding_window_view(x, 3)[::2, :]
>>       array([[0, 1, 2],
>>       [2, 3, 4],
>>       [4, 5, 6]])
>>
>>
>>
>>
>>
>>> Having a sliding window view in numpy is a longstanding open issue (cf
>>> #7753 [2] from 2016). A brief summary of the discussions surrounding it
>>> can be found in the description of the PR.
>>>
>>> This PR implements a sliding window view based on stride tricks.
>>> Following the discussion in issue #7753, a first implementation was
>>> provided by Fanjin Zeng in PR #10771. After some discussion, that PR
>>> stalled and I picked up the issue in the present PR #17394. It is based
>>> on the first implementation, but follows the changed API as suggested by
>>> Eric Wieser.
>>>
>>> Code reviews have been provided by Bas van Beek, Stephen Hoyer, and Eric
>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>>>
>>>
>>> Do you think this is suitable for inclusion in numpy?
>>>
>>> Do you consider the PR ready?
>>>
>>> Do you have suggestions or requests?
>>>
>>>
>>> Thanks for your time and consideration!
>>> Klaus
>>>
>>>
>>> [1] https://github.com/numpy/numpy/pull/17394
>>> <https://github.com/numpy/numpy/pull/17394>
>>> [2] https://github.com/numpy/numpy/issues/7753
>>> <https://github.com/numpy/numpy/issues/7753>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> [hidden email] <mailto:[hidden email]>
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>> <https://mail.python.org/mailman/listinfo/numpy-discussion>
>>>
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email] <mailto:[hidden email]>
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>> <https://mail.python.org/mailman/listinfo/numpy-discussion>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Sebastian Berg
In reply to this post by Zimmermann Klaus
Hi all,

just a brief note that I merged this proposal:

    https://github.com/numpy/numpy/pull/17394

adding `np.sliding_window_view` into the 1.20 release of NumPy.

There was only one public API change, and that is that the `shape`
argument is now called `window_shape`.

This is still a good time for feedback in case you have a better idea
e.g. for the function or parameter names.

Cheers,

Sebastian



On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:

> Hello,
>
> I would like to draw the attention of this list to PR #17394 [1] that
> adds the implementation of a sliding window view to numpy.
>
> Having a sliding window view in numpy is a longstanding open issue
> (cf
> #7753 [2] from 2016). A brief summary of the discussions surrounding
> it
> can be found in the description of the PR.
>
> This PR implements a sliding window view based on stride tricks.
> Following the discussion in issue #7753, a first implementation was
> provided by Fanjin Zeng in PR #10771. After some discussion, that PR
> stalled and I picked up the issue in the present PR #17394. It is
> based
> on the first implementation, but follows the changed API as suggested
> by
> Eric Wieser.
>
> Code reviews have been provided by Bas van Beek, Stephen Hoyer, and
> Eric
> Wieser. Sebastian Berg added the "62 - Python API" label.
>
>
> Do you think this is suitable for inclusion in numpy?
>
> Do you consider the PR ready?
>
> Do you have suggestions or requests?
>
>
> Thanks for your time and consideration!
> Klaus
>
>
> [1] https://github.com/numpy/numpy/pull/17394
> [2] https://github.com/numpy/numpy/issues/7753
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

ralfgommers


On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <[hidden email]> wrote:
Hi all,

just a brief note that I merged this proposal:

    https://github.com/numpy/numpy/pull/17394

adding `np.sliding_window_view` into the 1.20 release of NumPy.

There was only one public API change, and that is that the `shape`
argument is now called `window_shape`.

This is still a good time for feedback in case you have a better idea
e.g. for the function or parameter names.

The old PR had this in the lib.stride_tricks namespace. Seeing it in the main namespace is unexpected and likely will lead to issues/questions, given that such an overlapping view is going to do behave in ways the average user will be surprised by. It may also lead to requests for other array/tensor libraries to implement this. I don't see any discussion on this in PR 17394, it looks like a decision by the PR author that no one commented on - reconsider that?

Cheers,
Ralf


 

Cheers,

Sebastian



On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
> Hello,
>
> I would like to draw the attention of this list to PR #17394 [1] that
> adds the implementation of a sliding window view to numpy.
>
> Having a sliding window view in numpy is a longstanding open issue
> (cf
> #7753 [2] from 2016). A brief summary of the discussions surrounding
> it
> can be found in the description of the PR.
>
> This PR implements a sliding window view based on stride tricks.
> Following the discussion in issue #7753, a first implementation was
> provided by Fanjin Zeng in PR #10771. After some discussion, that PR
> stalled and I picked up the issue in the present PR #17394. It is
> based
> on the first implementation, but follows the changed API as suggested
> by
> Eric Wieser.
>
> Code reviews have been provided by Bas van Beek, Stephen Hoyer, and
> Eric
> Wieser. Sebastian Berg added the "62 - Python API" label.
>
>
> Do you think this is suitable for inclusion in numpy?
>
> Do you consider the PR ready?
>
> Do you have suggestions or requests?
>
>
> Thanks for your time and consideration!
> Klaus
>
>
> [1] https://github.com/numpy/numpy/pull/17394
> [2] https://github.com/numpy/numpy/issues/7753
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Stephan Hoyer-2
On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <[hidden email]> wrote:


On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <[hidden email]> wrote:
Hi all,

just a brief note that I merged this proposal:

    https://github.com/numpy/numpy/pull/17394

adding `np.sliding_window_view` into the 1.20 release of NumPy.

There was only one public API change, and that is that the `shape`
argument is now called `window_shape`.

This is still a good time for feedback in case you have a better idea
e.g. for the function or parameter names.

The old PR had this in the lib.stride_tricks namespace. Seeing it in the main namespace is unexpected and likely will lead to issues/questions, given that such an overlapping view is going to do behave in ways the average user will be surprised by. It may also lead to requests for other array/tensor libraries to implement this. I don't see any discussion on this in PR 17394, it looks like a decision by the PR author that no one commented on - reconsider that?

Cheers,
Ralf

+1 let's keep this in the lib.stride_tricks namespace.
 


 

Cheers,

Sebastian



On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
> Hello,
>
> I would like to draw the attention of this list to PR #17394 [1] that
> adds the implementation of a sliding window view to numpy.
>
> Having a sliding window view in numpy is a longstanding open issue
> (cf
> #7753 [2] from 2016). A brief summary of the discussions surrounding
> it
> can be found in the description of the PR.
>
> This PR implements a sliding window view based on stride tricks.
> Following the discussion in issue #7753, a first implementation was
> provided by Fanjin Zeng in PR #10771. After some discussion, that PR
> stalled and I picked up the issue in the present PR #17394. It is
> based
> on the first implementation, but follows the changed API as suggested
> by
> Eric Wieser.
>
> Code reviews have been provided by Bas van Beek, Stephen Hoyer, and
> Eric
> Wieser. Sebastian Berg added the "62 - Python API" label.
>
>
> Do you think this is suitable for inclusion in numpy?
>
> Do you consider the PR ready?
>
> Do you have suggestions or requests?
>
>
> Thanks for your time and consideration!
> Klaus
>
>
> [1] https://github.com/numpy/numpy/pull/17394
> [2] https://github.com/numpy/numpy/issues/7753
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Sebastian Berg
On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:

> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <[hidden email]>
> wrote:
>
> >
> > On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
> > [hidden email]>
> > wrote:
> >
> > > Hi all,
> > >
> > > just a brief note that I merged this proposal:
> > >
> > >     https://github.com/numpy/numpy/pull/17394
> > >
> > > adding `np.sliding_window_view` into the 1.20 release of NumPy.
> > >
> > > There was only one public API change, and that is that the
> > > `shape`
> > > argument is now called `window_shape`.
> > >
> > > This is still a good time for feedback in case you have a better
> > > idea
> > > e.g. for the function or parameter names.
> > >
> >
> > The old PR had this in the lib.stride_tricks namespace. Seeing it
> > in the
> > main namespace is unexpected and likely will lead to
> > issues/questions,
> > given that such an overlapping view is going to do behave in ways
> > the
> > average user will be surprised by. It may also lead to requests for
> > other
> > array/tensor libraries to implement this. I don't see any
> > discussion on
> > this in PR 17394, it looks like a decision by the PR author that no
> > one
> > commented on - reconsider that?
> >
> > Cheers,
> > Ralf
> >
>
> +1 let's keep this in the lib.stride_tricks namespace.
>
I have no reservations against having it in the main namespace and am
happy either way (it can still be exposed later in any case). It is the
conservative choice and maybe it is an uncommon enough function that it
deserves being a bit hidden...

But I am curious, it sounds like you have both very strong
reservations, and I would like to understand them better.

The behaviour can be surprising, but that is why the default is a read-
only view.  I do not think it is worse than `np.broadcast_to` in this
regard. (It is nowhere near as dangerous as `as_strided`.)

It is true that it is specific to NumPy (memory model). So that is
maybe a good enough reason right now.  But I am not sure that stuffing
things into a pretty hidden `np.lib.*` namespaces is a great long term
solution either. There is very little useful functionality hidden away
in `np.lib.*` currently.

Cheers,

Sebastian

>
> >
> >
> >
> > > Cheers,
> > >
> > > Sebastian
> > >
> > >
> > >
> > > On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
> > > > Hello,
> > > >
> > > > I would like to draw the attention of this list to PR #17394
> > > > [1] that
> > > > adds the implementation of a sliding window view to numpy.
> > > >
> > > > Having a sliding window view in numpy is a longstanding open
> > > > issue
> > > > (cf
> > > > #7753 [2] from 2016). A brief summary of the discussions
> > > > surrounding
> > > > it
> > > > can be found in the description of the PR.
> > > >
> > > > This PR implements a sliding window view based on stride
> > > > tricks.
> > > > Following the discussion in issue #7753, a first implementation
> > > > was
> > > > provided by Fanjin Zeng in PR #10771. After some discussion,
> > > > that PR
> > > > stalled and I picked up the issue in the present PR #17394. It
> > > > is
> > > > based
> > > > on the first implementation, but follows the changed API as
> > > > suggested
> > > > by
> > > > Eric Wieser.
> > > >
> > > > Code reviews have been provided by Bas van Beek, Stephen Hoyer,
> > > > and
> > > > Eric
> > > > Wieser. Sebastian Berg added the "62 - Python API" label.
> > > >
> > > >
> > > > Do you think this is suitable for inclusion in numpy?
> > > >
> > > > Do you consider the PR ready?
> > > >
> > > > Do you have suggestions or requests?
> > > >
> > > >
> > > > Thanks for your time and consideration!
> > > > Klaus
> > > >
> > > >
> > > > [1] https://github.com/numpy/numpy/pull/17394
> > > > [2] https://github.com/numpy/numpy/issues/7753
> > > > _______________________________________________
> > > > NumPy-Discussion mailing list
> > > > [hidden email]
> > > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > > >
> > >
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > [hidden email]
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Sebastian Berg
On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:

> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
> > On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
> > [hidden email]>
> > wrote:
> >
> > > On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > just a brief note that I merged this proposal:
> > > >
> > > >     https://github.com/numpy/numpy/pull/17394
> > > >
> > > > adding `np.sliding_window_view` into the 1.20 release of NumPy.
> > > >
> > > > There was only one public API change, and that is that the
> > > > `shape`
> > > > argument is now called `window_shape`.
> > > >
> > > > This is still a good time for feedback in case you have a
> > > > better
> > > > idea
> > > > e.g. for the function or parameter names.
> > > >
> > >
> > > The old PR had this in the lib.stride_tricks namespace. Seeing it
> > > in the
> > > main namespace is unexpected and likely will lead to
> > > issues/questions,
> > > given that such an overlapping view is going to do behave in ways
> > > the
> > > average user will be surprised by. It may also lead to requests
> > > for
> > > other
> > > array/tensor libraries to implement this. I don't see any
> > > discussion on
> > > this in PR 17394, it looks like a decision by the PR author that
> > > no
> > > one
> > > commented on - reconsider that?
> > >
> > > Cheers,
> > > Ralf
> > >
> >
> > +1 let's keep this in the lib.stride_tricks namespace.
> >
>
> I have no reservations against having it in the main namespace and am
> happy either way (it can still be exposed later in any case). It is
> the
> conservative choice and maybe it is an uncommon enough function that
> it
> deserves being a bit hidden...

In any case, its the safe bet for NumPy 1.20 at least so I opened a PR:

    https://github.com/numpy/numpy/pull/17720

Name changes, etc. are also possible of course.

I still think it might be nice to find a better place for this type of
function that `np.lib.stride_tricks` though, but dunno...

- Sebastian



>
> But I am curious, it sounds like you have both very strong
> reservations, and I would like to understand them better.
>
> The behaviour can be surprising, but that is why the default is a
> read-
> only view.  I do not think it is worse than `np.broadcast_to` in this
> regard. (It is nowhere near as dangerous as `as_strided`.)
>
> It is true that it is specific to NumPy (memory model). So that is
> maybe a good enough reason right now.  But I am not sure that
> stuffing
> things into a pretty hidden `np.lib.*` namespaces is a great long
> term
> solution either. There is very little useful functionality hidden
> away
> in `np.lib.*` currently.
>
> Cheers,
>
> Sebastian
>
> > >
> > >
> > > > Cheers,
> > > >
> > > > Sebastian
> > > >
> > > >
> > > >
> > > > On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
> > > > > Hello,
> > > > >
> > > > > I would like to draw the attention of this list to PR #17394
> > > > > [1] that
> > > > > adds the implementation of a sliding window view to numpy.
> > > > >
> > > > > Having a sliding window view in numpy is a longstanding open
> > > > > issue
> > > > > (cf
> > > > > #7753 [2] from 2016). A brief summary of the discussions
> > > > > surrounding
> > > > > it
> > > > > can be found in the description of the PR.
> > > > >
> > > > > This PR implements a sliding window view based on stride
> > > > > tricks.
> > > > > Following the discussion in issue #7753, a first
> > > > > implementation
> > > > > was
> > > > > provided by Fanjin Zeng in PR #10771. After some discussion,
> > > > > that PR
> > > > > stalled and I picked up the issue in the present PR #17394.
> > > > > It
> > > > > is
> > > > > based
> > > > > on the first implementation, but follows the changed API as
> > > > > suggested
> > > > > by
> > > > > Eric Wieser.
> > > > >
> > > > > Code reviews have been provided by Bas van Beek, Stephen
> > > > > Hoyer,
> > > > > and
> > > > > Eric
> > > > > Wieser. Sebastian Berg added the "62 - Python API" label.
> > > > >
> > > > >
> > > > > Do you think this is suitable for inclusion in numpy?
> > > > >
> > > > > Do you consider the PR ready?
> > > > >
> > > > > Do you have suggestions or requests?
> > > > >
> > > > >
> > > > > Thanks for your time and consideration!
> > > > > Klaus
> > > > >
> > > > >
> > > > > [1] https://github.com/numpy/numpy/pull/17394
> > > > > [2] https://github.com/numpy/numpy/issues/7753
> > > > > _______________________________________________
> > > > > NumPy-Discussion mailing list
> > > > > [hidden email]
> > > > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > > > >
> > > >
> > > > _______________________________________________
> > > > NumPy-Discussion mailing list
> > > > [hidden email]
> > > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > > >
> > > _______________________________________________
> > > NumPy-Discussion mailing list
> > > [hidden email]
> > > https://mail.python.org/mailman/listinfo/numpy-discussion
> > >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Zimmermann Klaus
Hi all,


I have absolutely no problem keeping this out of the main namespace.

In fact I'd like to point out that it was not my idea. Rather, it was
proposed by Bas van Beek in the comments [1,2] and received a little
more scrutiny from Eric Wieser in [3].

The reason that it didn't receive the scrutiny it probably deserves is
that it got a bit mangled up with the array dispatch discussion; sorry
for that.

On the subject matter, I am also curious about the potential for
confusion. What other behavior could one expect from a sliding window
view with this shape?

As I said, I am completely fine with keeping this out of the main
namespace, but I agree with Sebastian's comment, that
`np.lib.stride_tricks` is perhaps not the best namespace. The reason
from my point of view is that stride tricks is really a technical (and
slightly ominous) name that might throw of more application oriented
programmers from finding and using this function. Thinking of my
scientist colleagues, I think those are exactly the kind of users that
could benefit from such a prototyping tool.

Cheers
Klaus



[1] https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
[2] https://github.com/numpy/numpy/pull/17394#discussion_r498215468
[3] https://github.com/numpy/numpy/pull/17394#discussion_r498724340

On 06/11/2020 01:39, Sebastian Berg wrote:

> On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:
>> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
>>> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
>>> [hidden email]>
>>> wrote:
>>>
>>>> On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
>>>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> just a brief note that I merged this proposal:
>>>>>
>>>>>     https://github.com/numpy/numpy/pull/17394
>>>>>
>>>>> adding `np.sliding_window_view` into the 1.20 release of NumPy.
>>>>>
>>>>> There was only one public API change, and that is that the
>>>>> `shape`
>>>>> argument is now called `window_shape`.
>>>>>
>>>>> This is still a good time for feedback in case you have a
>>>>> better
>>>>> idea
>>>>> e.g. for the function or parameter names.
>>>>>
>>>>
>>>> The old PR had this in the lib.stride_tricks namespace. Seeing it
>>>> in the
>>>> main namespace is unexpected and likely will lead to
>>>> issues/questions,
>>>> given that such an overlapping view is going to do behave in ways
>>>> the
>>>> average user will be surprised by. It may also lead to requests
>>>> for
>>>> other
>>>> array/tensor libraries to implement this. I don't see any
>>>> discussion on
>>>> this in PR 17394, it looks like a decision by the PR author that
>>>> no
>>>> one
>>>> commented on - reconsider that?
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>
>>> +1 let's keep this in the lib.stride_tricks namespace.
>>>
>>
>> I have no reservations against having it in the main namespace and am
>> happy either way (it can still be exposed later in any case). It is
>> the
>> conservative choice and maybe it is an uncommon enough function that
>> it
>> deserves being a bit hidden...
>
>
> In any case, its the safe bet for NumPy 1.20 at least so I opened a PR:
>
>     https://github.com/numpy/numpy/pull/17720
>
> Name changes, etc. are also possible of course.
>
> I still think it might be nice to find a better place for this type of
> function that `np.lib.stride_tricks` though, but dunno...
>
> - Sebastian
>
>
>
>>
>> But I am curious, it sounds like you have both very strong
>> reservations, and I would like to understand them better.
>>
>> The behaviour can be surprising, but that is why the default is a
>> read-
>> only view.  I do not think it is worse than `np.broadcast_to` in this
>> regard. (It is nowhere near as dangerous as `as_strided`.)
>>
>> It is true that it is specific to NumPy (memory model). So that is
>> maybe a good enough reason right now.  But I am not sure that
>> stuffing
>> things into a pretty hidden `np.lib.*` namespaces is a great long
>> term
>> solution either. There is very little useful functionality hidden
>> away
>> in `np.lib.*` currently.
>>
>> Cheers,
>>
>> Sebastian
>>
>>>>
>>>>
>>>>> Cheers,
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>>
>>>>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I would like to draw the attention of this list to PR #17394
>>>>>> [1] that
>>>>>> adds the implementation of a sliding window view to numpy.
>>>>>>
>>>>>> Having a sliding window view in numpy is a longstanding open
>>>>>> issue
>>>>>> (cf
>>>>>> #7753 [2] from 2016). A brief summary of the discussions
>>>>>> surrounding
>>>>>> it
>>>>>> can be found in the description of the PR.
>>>>>>
>>>>>> This PR implements a sliding window view based on stride
>>>>>> tricks.
>>>>>> Following the discussion in issue #7753, a first
>>>>>> implementation
>>>>>> was
>>>>>> provided by Fanjin Zeng in PR #10771. After some discussion,
>>>>>> that PR
>>>>>> stalled and I picked up the issue in the present PR #17394.
>>>>>> It
>>>>>> is
>>>>>> based
>>>>>> on the first implementation, but follows the changed API as
>>>>>> suggested
>>>>>> by
>>>>>> Eric Wieser.
>>>>>>
>>>>>> Code reviews have been provided by Bas van Beek, Stephen
>>>>>> Hoyer,
>>>>>> and
>>>>>> Eric
>>>>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>>>>>>
>>>>>>
>>>>>> Do you think this is suitable for inclusion in numpy?
>>>>>>
>>>>>> Do you consider the PR ready?
>>>>>>
>>>>>> Do you have suggestions or requests?
>>>>>>
>>>>>>
>>>>>> Thanks for your time and consideration!
>>>>>> Klaus
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/numpy/numpy/pull/17394
>>>>>> [2] https://github.com/numpy/numpy/issues/7753
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> [hidden email]
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> [hidden email]
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> [hidden email]
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> [hidden email]
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

ralfgommers


On Fri, Nov 6, 2020 at 9:51 AM Zimmermann Klaus <[hidden email]> wrote:
Hi all,


I have absolutely no problem keeping this out of the main namespace.

In fact I'd like to point out that it was not my idea. Rather, it was
proposed by Bas van Beek in the comments [1,2] and received a little
more scrutiny from Eric Wieser in [3].

Thanks, between two PRs with that many comments, I couldn't figure that out - just saw the commit that make the change.


The reason that it didn't receive the scrutiny it probably deserves is
that it got a bit mangled up with the array dispatch discussion; sorry
for that.

No worries at all. This is why we announce new features on the mailing list.


On the subject matter, I am also curious about the potential for
confusion. What other behavior could one expect from a sliding window
view with this shape?

As I said, I am completely fine with keeping this out of the main
namespace, but I agree with Sebastian's comment, that
`np.lib.stride_tricks` is perhaps not the best namespace.

I agree that that's not a great namespace. There's multiple issues with namespaces, we basically have three good ones (fft, linalg, random) and a bunch of other ones that range from questionable to terrible. See https://github.com/numpy/numpy/blob/master/numpy/tests/test_public_api.py#L127 for details.

This would be a good thing to work on - making the `numpy.lib` namespace not bleed into `numpy` via `import *` is one thing to do there, and there's many others. But given backwards compat constraints it's not easy.
 
The reason
from my point of view is that stride tricks is really a technical (and
slightly ominous) name that might throw of more application oriented
programmers from finding and using this function. Thinking of my
scientist colleagues, I think those are exactly the kind of users that
could benefit from such a prototyping tool.

That phrasing is one of a number of concerns. NumPy is normally not in the business of providing things that are okay as a prototyping tool, but are potentially extremely slow (as pointed out in the Notes section of the docstring). A function like that would basically not be the right tool for almost anything in, e.g., SciPy - it requires an iterative algorithm. In NumPy we don't prefer performance at all costs, but in general it's pretty decent rather than "Numba or Cython may gain you 100x here".

Other issues include:
2) It is very specific to NumPy's memory model (as pointed out by you and Sebastian) - just like the rest of stride_tricks
3) It has "view" in the name, which doesn't quite make sense for the main namespace (also connected to point 2 above).
4) The cost of putting something in the main namespace for other array/tensor libraries is large. Maybe other libraries, e.g. CuPy, Dask, TensorFlow, PyTorch, JAX, MXNet, aim to reimplement part or all of the main NumPy namespace as well as possible. This would trigger discussions and likely many person-weeks of work for others.
5) It's a useful function, but it's very much on the margins of NumPy's scope. It could easily have gone into, for example, scipy.signal. At this point the bar for functions going into the main namespace should be (and is) high.

All this taken together means it's not even a toss-up for me. If it were just one or two of these points, maybe. But given all the above, I'm pretty confident saying "it does not belong in the main namespace".

Cheers,
Ralf

 

Cheers
Klaus



[1] https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
[2] https://github.com/numpy/numpy/pull/17394#discussion_r498215468
[3] https://github.com/numpy/numpy/pull/17394#discussion_r498724340

On 06/11/2020 01:39, Sebastian Berg wrote:
> On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:
>> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
>>> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
>>> [hidden email]>
>>> wrote:
>>>
>>>> On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
>>>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> just a brief note that I merged this proposal:
>>>>>
>>>>>     https://github.com/numpy/numpy/pull/17394
>>>>>
>>>>> adding `np.sliding_window_view` into the 1.20 release of NumPy.
>>>>>
>>>>> There was only one public API change, and that is that the
>>>>> `shape`
>>>>> argument is now called `window_shape`.
>>>>>
>>>>> This is still a good time for feedback in case you have a
>>>>> better
>>>>> idea
>>>>> e.g. for the function or parameter names.
>>>>>
>>>>
>>>> The old PR had this in the lib.stride_tricks namespace. Seeing it
>>>> in the
>>>> main namespace is unexpected and likely will lead to
>>>> issues/questions,
>>>> given that such an overlapping view is going to do behave in ways
>>>> the
>>>> average user will be surprised by. It may also lead to requests
>>>> for
>>>> other
>>>> array/tensor libraries to implement this. I don't see any
>>>> discussion on
>>>> this in PR 17394, it looks like a decision by the PR author that
>>>> no
>>>> one
>>>> commented on - reconsider that?
>>>>
>>>> Cheers,
>>>> Ralf
>>>>
>>>
>>> +1 let's keep this in the lib.stride_tricks namespace.
>>>
>>
>> I have no reservations against having it in the main namespace and am
>> happy either way (it can still be exposed later in any case). It is
>> the
>> conservative choice and maybe it is an uncommon enough function that
>> it
>> deserves being a bit hidden...
>
>
> In any case, its the safe bet for NumPy 1.20 at least so I opened a PR:
>
>     https://github.com/numpy/numpy/pull/17720
>
> Name changes, etc. are also possible of course.
>
> I still think it might be nice to find a better place for this type of
> function that `np.lib.stride_tricks` though, but dunno...
>
> - Sebastian
>
>
>
>>
>> But I am curious, it sounds like you have both very strong
>> reservations, and I would like to understand them better.
>>
>> The behaviour can be surprising, but that is why the default is a
>> read-
>> only view.  I do not think it is worse than `np.broadcast_to` in this
>> regard. (It is nowhere near as dangerous as `as_strided`.)
>>
>> It is true that it is specific to NumPy (memory model). So that is
>> maybe a good enough reason right now.  But I am not sure that
>> stuffing
>> things into a pretty hidden `np.lib.*` namespaces is a great long
>> term
>> solution either. There is very little useful functionality hidden
>> away
>> in `np.lib.*` currently.
>>
>> Cheers,
>>
>> Sebastian
>>
>>>>
>>>>
>>>>> Cheers,
>>>>>
>>>>> Sebastian
>>>>>
>>>>>
>>>>>
>>>>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I would like to draw the attention of this list to PR #17394
>>>>>> [1] that
>>>>>> adds the implementation of a sliding window view to numpy.
>>>>>>
>>>>>> Having a sliding window view in numpy is a longstanding open
>>>>>> issue
>>>>>> (cf
>>>>>> #7753 [2] from 2016). A brief summary of the discussions
>>>>>> surrounding
>>>>>> it
>>>>>> can be found in the description of the PR.
>>>>>>
>>>>>> This PR implements a sliding window view based on stride
>>>>>> tricks.
>>>>>> Following the discussion in issue #7753, a first
>>>>>> implementation
>>>>>> was
>>>>>> provided by Fanjin Zeng in PR #10771. After some discussion,
>>>>>> that PR
>>>>>> stalled and I picked up the issue in the present PR #17394.
>>>>>> It
>>>>>> is
>>>>>> based
>>>>>> on the first implementation, but follows the changed API as
>>>>>> suggested
>>>>>> by
>>>>>> Eric Wieser.
>>>>>>
>>>>>> Code reviews have been provided by Bas van Beek, Stephen
>>>>>> Hoyer,
>>>>>> and
>>>>>> Eric
>>>>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>>>>>>
>>>>>>
>>>>>> Do you think this is suitable for inclusion in numpy?
>>>>>>
>>>>>> Do you consider the PR ready?
>>>>>>
>>>>>> Do you have suggestions or requests?
>>>>>>
>>>>>>
>>>>>> Thanks for your time and consideration!
>>>>>> Klaus
>>>>>>
>>>>>>
>>>>>> [1] https://github.com/numpy/numpy/pull/17394
>>>>>> [2] https://github.com/numpy/numpy/issues/7753
>>>>>> _______________________________________________
>>>>>> NumPy-Discussion mailing list
>>>>>> [hidden email]
>>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> NumPy-Discussion mailing list
>>>>> [hidden email]
>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>>
>>>> _______________________________________________
>>>> NumPy-Discussion mailing list
>>>> [hidden email]
>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>>>
>>>
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> [hidden email]
>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Zimmermann Klaus
Hi,

On 06/11/2020 15:58, Ralf Gommers wrote:

> On Fri, Nov 6, 2020 at 9:51 AM Zimmermann Klaus
> <[hidden email] <mailto:[hidden email]>> wrote:
>     I have absolutely no problem keeping this out of the main namespace.
>
>     In fact I'd like to point out that it was not my idea. Rather, it was
>     proposed by Bas van Beek in the comments [1,2] and received a little
>     more scrutiny from Eric Wieser in [3].
>
> Thanks, between two PRs with that many comments, I couldn't figure that
> out - just saw the commit that make the change.

Understandable, no worries.

>     On the subject matter, I am also curious about the potential for
>     confusion. What other behavior could one expect from a sliding window
>     view with this shape?
>
>     As I said, I am completely fine with keeping this out of the main
>     namespace, but I agree with Sebastian's comment, that
>     `np.lib.stride_tricks` is perhaps not the best namespace.
>
>
> I agree that that's not a great namespace. There's multiple issues with
> namespaces, we basically have three good ones (fft, linalg, random) and
> a bunch of other ones that range from questionable to terrible. See
> https://github.com/numpy/numpy/blob/master/numpy/tests/test_public_api.py#L127
> <https://github.com/numpy/numpy/blob/master/numpy/tests/test_public_api.py#L127>
> for details.
>
> This would be a good thing to work on - making the `numpy.lib` namespace
> not bleed into `numpy` via `import *` is one thing to do there, and
> there's many others. But given backwards compat constraints it's not easy.

I understand cleaning up all the namespaces is a giant task, so far, far
out of scope here. As said before, I also completely agree to keep it
out of the main namespace (though I will still argue below :P).

I was just wondering if, of the top your head, an existing, better fit
comes to mind?

>     The reason
>     from my point of view is that stride tricks is really a technical (and
>     slightly ominous) name that might throw of more application oriented
>     programmers from finding and using this function. Thinking of my
>     scientist colleagues, I think those are exactly the kind of users that
>     could benefit from such a prototyping tool.
>
>
> That phrasing is one of a number of concerns. NumPy is normally not in
> the business of providing things that are okay as a prototyping tool,
> but are potentially extremely slow (as pointed out in the Notes section
> of the docstring). A function like that would basically not be the right
> tool for almost anything in, e.g., SciPy - it requires an iterative
> algorithm. In NumPy we don't prefer performance at all costs, but in
> general it's pretty decent rather than "Numba or Cython may gain you
> 100x here".

I still think that the performance concern is a bit overblown. Yes,
application with large windows can need more FLOPs by an equally large
factor. But most such applications will use small to moderate windows.
Furthermore, this view focuses only on FLOPs. In my current field of
climate science (and many others), that is almost never the limiting
factor. Memory demands are far more problematic and incidentally, those
are more likely to increase in other methods that require the storage of
ancillary, temporary data.

> Other issues include:
> 2) It is very specific to NumPy's memory model (as pointed out by you
> and Sebastian) - just like the rest of stride_tricks
Not wrong, but on the other hand, that memory model is not exotic. C,
Fortran, and any number of other languages play very nicely with this,
just as important downstream libraries like dask.

> 3) It has "view" in the name, which doesn't quite make sense for the
> main namespace (also connected to point 2 above).
Ok.

> 4) The cost of putting something in the main namespace for other
> array/tensor libraries is large. Maybe other libraries, e.g. CuPy, Dask,
> TensorFlow, PyTorch, JAX, MXNet, aim to reimplement part or all of the
> main NumPy namespace as well as possible. This would trigger discussions
> and likely many person-weeks of work for others.
Agreed. Though I have to say that my whole motivation comes from
corresponding issues in dask that where specifically waiting for (the
older version of) this PR (see [1, 2,...]). But I understand that dask
is effectively much closer to the numpy memory model than, say, CuPy, so
don't take this to mean it should be in the main namespace.

> 5) It's a useful function, but it's very much on the margins of NumPy's
> scope. It could easily have gone into, for example, scipy.signal. At
> this point the bar for functions going into the main namespace should be> (and is) high.
I agree that the bar for the main namespace should be high!

> All this taken together means it's not even a toss-up for me. If it were
> just one or two of these points, maybe. But given all the above, I'm
> pretty confident saying "it does not belong in the main namespace".
Again, I am happy with that.


Thanks for your thoughts and work! I really appreciate it!

Cheers
Klaus

[1] https://github.com/dask/dask/issues/4659
[2] https://github.com/pydata/xarray/issues/3608
[3] https://github.com/pandas-dev/pandas/issues/26959


>
>
>     Cheers
>     Klaus
>
>
>
>     [1] https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
>     <https://github.com/numpy/numpy/pull/17394#issuecomment-700998618>
>     [2] https://github.com/numpy/numpy/pull/17394#discussion_r498215468
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498215468>
>     [3] https://github.com/numpy/numpy/pull/17394#discussion_r498724340
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498724340>
>
>     On 06/11/2020 01:39, Sebastian Berg wrote:
>     > On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:
>     >> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
>     >>> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
>     >>> [hidden email] <mailto:[hidden email]>>
>     >>> wrote:
>     >>>
>     >>>> On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
>     >>>> [hidden email] <mailto:[hidden email]>>
>     >>>> wrote:
>     >>>>
>     >>>>> Hi all,
>     >>>>>
>     >>>>> just a brief note that I merged this proposal:
>     >>>>>
>     >>>>>¬† ¬† ¬†https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >>>>>
>     >>>>> adding `np.sliding_window_view` into the 1.20 release of NumPy.
>     >>>>>
>     >>>>> There was only one public API change, and that is that the
>     >>>>> `shape`
>     >>>>> argument is now called `window_shape`.
>     >>>>>
>     >>>>> This is still a good time for feedback in case you have a
>     >>>>> better
>     >>>>> idea
>     >>>>> e.g. for the function or parameter names.
>     >>>>>
>     >>>>
>     >>>> The old PR had this in the lib.stride_tricks namespace. Seeing it
>     >>>> in the
>     >>>> main namespace is unexpected and likely will lead to
>     >>>> issues/questions,
>     >>>> given that such an overlapping view is going to do behave in ways
>     >>>> the
>     >>>> average user will be surprised by. It may also lead to requests
>     >>>> for
>     >>>> other
>     >>>> array/tensor libraries to implement this. I don't see any
>     >>>> discussion on
>     >>>> this in PR 17394, it looks like a decision by the PR author that
>     >>>> no
>     >>>> one
>     >>>> commented on - reconsider that?
>     >>>>
>     >>>> Cheers,
>     >>>> Ralf
>     >>>>
>     >>>
>     >>> +1 let's keep this in the lib.stride_tricks namespace.
>     >>>
>     >>
>     >> I have no reservations against having it in the main namespace and am
>     >> happy either way (it can still be exposed later in any case). It is
>     >> the
>     >> conservative choice and maybe it is an uncommon enough function that
>     >> it
>     >> deserves being a bit hidden...
>     >
>     >
>     > In any case, its the safe bet for NumPy 1.20 at least so I opened
>     a PR:
>     >
>     >¬† ¬† ¬†https://github.com/numpy/numpy/pull/17720
>     <https://github.com/numpy/numpy/pull/17720>
>     >
>     > Name changes, etc. are also possible of course.
>     >
>     > I still think it might be nice to find a better place for this type of
>     > function that `np.lib.stride_tricks` though, but dunno...
>     >
>     > - Sebastian
>     >
>     >
>     >
>     >>
>     >> But I am curious, it sounds like you have both very strong
>     >> reservations, and I would like to understand them better.
>     >>
>     >> The behaviour can be surprising, but that is why the default is a
>     >> read-
>     >> only view.¬† I do not think it is worse than `np.broadcast_to` in this
>     >> regard. (It is nowhere near as dangerous as `as_strided`.)
>     >>
>     >> It is true that it is specific to NumPy (memory model). So that is
>     >> maybe a good enough reason right now.¬† But I am not sure that
>     >> stuffing
>     >> things into a pretty hidden `np.lib.*` namespaces is a great long
>     >> term
>     >> solution either. There is very little useful functionality hidden
>     >> away
>     >> in `np.lib.*` currently.
>     >>
>     >> Cheers,
>     >>
>     >> Sebastian
>     >>
>     >>>>
>     >>>>
>     >>>>> Cheers,
>     >>>>>
>     >>>>> Sebastian
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>     >>>>>> Hello,
>     >>>>>>
>     >>>>>> I would like to draw the attention of this list to PR #17394
>     >>>>>> [1] that
>     >>>>>> adds the implementation of a sliding window view to numpy.
>     >>>>>>
>     >>>>>> Having a sliding window view in numpy is a longstanding open
>     >>>>>> issue
>     >>>>>> (cf
>     >>>>>> #7753 [2] from 2016). A brief summary of the discussions
>     >>>>>> surrounding
>     >>>>>> it
>     >>>>>> can be found in the description of the PR.
>     >>>>>>
>     >>>>>> This PR implements a sliding window view based on stride
>     >>>>>> tricks.
>     >>>>>> Following the discussion in issue #7753, a first
>     >>>>>> implementation
>     >>>>>> was
>     >>>>>> provided by Fanjin Zeng in PR #10771. After some discussion,
>     >>>>>> that PR
>     >>>>>> stalled and I picked up the issue in the present PR #17394.
>     >>>>>> It
>     >>>>>> is
>     >>>>>> based
>     >>>>>> on the first implementation, but follows the changed API as
>     >>>>>> suggested
>     >>>>>> by
>     >>>>>> Eric Wieser.
>     >>>>>>
>     >>>>>> Code reviews have been provided by Bas van Beek, Stephen
>     >>>>>> Hoyer,
>     >>>>>> and
>     >>>>>> Eric
>     >>>>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>     >>>>>>
>     >>>>>>
>     >>>>>> Do you think this is suitable for inclusion in numpy?
>     >>>>>>
>     >>>>>> Do you consider the PR ready?
>     >>>>>>
>     >>>>>> Do you have suggestions or requests?
>     >>>>>>
>     >>>>>>
>     >>>>>> Thanks for your time and consideration!
>     >>>>>> Klaus
>     >>>>>>
>     >>>>>>
>     >>>>>> [1] https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >>>>>> [2] https://github.com/numpy/numpy/issues/7753
>     <https://github.com/numpy/numpy/issues/7753>
>     >>>>>> _______________________________________________
>     >>>>>> NumPy-Discussion mailing list
>     >>>>>> [hidden email] <mailto:[hidden email]>
>     >>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>>>
>     >>>>>
>     >>>>> _______________________________________________
>     >>>>> NumPy-Discussion mailing list
>     >>>>> [hidden email] <mailto:[hidden email]>
>     >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>>
>     >>>> _______________________________________________
>     >>>> NumPy-Discussion mailing list
>     >>>> [hidden email] <mailto:[hidden email]>
>     >>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>
>     >>>
>     >>> _______________________________________________
>     >>> NumPy-Discussion mailing list
>     >>> [hidden email] <mailto:[hidden email]>
>     >>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>
>     >> _______________________________________________
>     >> NumPy-Discussion mailing list
>     >> [hidden email] <mailto:[hidden email]>
>     >> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >
>     >
>     > _______________________________________________
>     > NumPy-Discussion mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >
>     _______________________________________________
>     NumPy-Discussion mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

ralfgommers


On Fri, Nov 6, 2020 at 4:03 PM Zimmermann Klaus <[hidden email]> wrote:
Hi,

On 06/11/2020 15:58, Ralf Gommers wrote:
> On Fri, Nov 6, 2020 at 9:51 AM Zimmermann Klaus
> <[hidden email] <mailto:[hidden email]>> wrote:
>     I have absolutely no problem keeping this out of the main namespace.
>
>     In fact I'd like to point out that it was not my idea. Rather, it was
>     proposed by Bas van Beek in the comments [1,2] and received a little
>     more scrutiny from Eric Wieser in [3].
>
> Thanks, between two PRs with that many comments, I couldn't figure that
> out - just saw the commit that make the change.

Understandable, no worries.

>     On the subject matter, I am also curious about the potential for
>     confusion. What other behavior could one expect from a sliding window
>     view with this shape?
>
>     As I said, I am completely fine with keeping this out of the main
>     namespace, but I agree with Sebastian's comment, that
>     `np.lib.stride_tricks` is perhaps not the best namespace.
>
>
> I agree that that's not a great namespace. There's multiple issues with
> namespaces, we basically have three good ones (fft, linalg, random) and
> a bunch of other ones that range from questionable to terrible. See
> https://github.com/numpy/numpy/blob/master/numpy/tests/test_public_api.py#L127
> <https://github.com/numpy/numpy/blob/master/numpy/tests/test_public_api.py#L127>
> for details.
>
> This would be a good thing to work on - making the `numpy.lib` namespace
> not bleed into `numpy` via `import *` is one thing to do there, and
> there's many others. But given backwards compat constraints it's not easy.

I understand cleaning up all the namespaces is a giant task, so far, far
out of scope here. As said before, I also completely agree to keep it
out of the main namespace (though I will still argue below :P).

I was just wondering if, of the top your head, an existing, better fit
comes to mind?

Not really. Outside of stride_tricks there's nothing that quite fits. This function is more in scope for something like scipy.signal.

Cheers,
Ralf



>     The reason
>     from my point of view is that stride tricks is really a technical (and
>     slightly ominous) name that might throw of more application oriented
>     programmers from finding and using this function. Thinking of my
>     scientist colleagues, I think those are exactly the kind of users that
>     could benefit from such a prototyping tool.
>
>
> That phrasing is one of a number of concerns. NumPy is normally not in
> the business of providing things that are okay as a prototyping tool,
> but are potentially extremely slow (as pointed out in the Notes section
> of the docstring). A function like that would basically not be the right
> tool for almost anything in, e.g., SciPy - it requires an iterative
> algorithm. In NumPy we don't prefer performance at all costs, but in
> general it's pretty decent rather than "Numba or Cython may gain you
> 100x here".

I still think that the performance concern is a bit overblown. Yes,
application with large windows can need more FLOPs by an equally large
factor. But most such applications will use small to moderate windows.
Furthermore, this view focuses only on FLOPs. In my current field of
climate science (and many others), that is almost never the limiting
factor. Memory demands are far more problematic and incidentally, those
are more likely to increase in other methods that require the storage of
ancillary, temporary data.

> Other issues include:
> 2) It is very specific to NumPy's memory model (as pointed out by you
> and Sebastian) - just like the rest of stride_tricks
Not wrong, but on the other hand, that memory model is not exotic. C,
Fortran, and any number of other languages play very nicely with this,
just as important downstream libraries like dask.

> 3) It has "view" in the name, which doesn't quite make sense for the
> main namespace (also connected to point 2 above).
Ok.

> 4) The cost of putting something in the main namespace for other
> array/tensor libraries is large. Maybe other libraries, e.g. CuPy, Dask,
> TensorFlow, PyTorch, JAX, MXNet, aim to reimplement part or all of the
> main NumPy namespace as well as possible. This would trigger discussions
> and likely many person-weeks of work for others.
Agreed. Though I have to say that my whole motivation comes from
corresponding issues in dask that where specifically waiting for (the
older version of) this PR (see [1, 2,...]). But I understand that dask
is effectively much closer to the numpy memory model than, say, CuPy, so
don't take this to mean it should be in the main namespace.

> 5) It's a useful function, but it's very much on the margins of NumPy's
> scope. It could easily have gone into, for example, scipy.signal. At
> this point the bar for functions going into the main namespace should be> (and is) high.
I agree that the bar for the main namespace should be high!

> All this taken together means it's not even a toss-up for me. If it were
> just one or two of these points, maybe. But given all the above, I'm
> pretty confident saying "it does not belong in the main namespace".
Again, I am happy with that.


Thanks for your thoughts and work! I really appreciate it!

Cheers
Klaus

[1] https://github.com/dask/dask/issues/4659
[2] https://github.com/pydata/xarray/issues/3608
[3] https://github.com/pandas-dev/pandas/issues/26959


>
>
>     Cheers
>     Klaus
>
>
>
>     [1] https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
>     <https://github.com/numpy/numpy/pull/17394#issuecomment-700998618>
>     [2] https://github.com/numpy/numpy/pull/17394#discussion_r498215468
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498215468>
>     [3] https://github.com/numpy/numpy/pull/17394#discussion_r498724340
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498724340>
>
>     On 06/11/2020 01:39, Sebastian Berg wrote:
>     > On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:
>     >> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
>     >>> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
>     >>> [hidden email] <mailto:[hidden email]>>
>     >>> wrote:
>     >>>
>     >>>> On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
>     >>>> [hidden email] <mailto:[hidden email]>>
>     >>>> wrote:
>     >>>>
>     >>>>> Hi all,
>     >>>>>
>     >>>>> just a brief note that I merged this proposal:
>     >>>>>
>     >>>>>     https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >>>>>
>     >>>>> adding `np.sliding_window_view` into the 1.20 release of NumPy.
>     >>>>>
>     >>>>> There was only one public API change, and that is that the
>     >>>>> `shape`
>     >>>>> argument is now called `window_shape`.
>     >>>>>
>     >>>>> This is still a good time for feedback in case you have a
>     >>>>> better
>     >>>>> idea
>     >>>>> e.g. for the function or parameter names.
>     >>>>>
>     >>>>
>     >>>> The old PR had this in the lib.stride_tricks namespace. Seeing it
>     >>>> in the
>     >>>> main namespace is unexpected and likely will lead to
>     >>>> issues/questions,
>     >>>> given that such an overlapping view is going to do behave in ways
>     >>>> the
>     >>>> average user will be surprised by. It may also lead to requests
>     >>>> for
>     >>>> other
>     >>>> array/tensor libraries to implement this. I don't see any
>     >>>> discussion on
>     >>>> this in PR 17394, it looks like a decision by the PR author that
>     >>>> no
>     >>>> one
>     >>>> commented on - reconsider that?
>     >>>>
>     >>>> Cheers,
>     >>>> Ralf
>     >>>>
>     >>>
>     >>> +1 let's keep this in the lib.stride_tricks namespace.
>     >>>
>     >>
>     >> I have no reservations against having it in the main namespace and am
>     >> happy either way (it can still be exposed later in any case). It is
>     >> the
>     >> conservative choice and maybe it is an uncommon enough function that
>     >> it
>     >> deserves being a bit hidden...
>     >
>     >
>     > In any case, its the safe bet for NumPy 1.20 at least so I opened
>     a PR:
>     >
>     >     https://github.com/numpy/numpy/pull/17720
>     <https://github.com/numpy/numpy/pull/17720>
>     >
>     > Name changes, etc. are also possible of course.
>     >
>     > I still think it might be nice to find a better place for this type of
>     > function that `np.lib.stride_tricks` though, but dunno...
>     >
>     > - Sebastian
>     >
>     >
>     >
>     >>
>     >> But I am curious, it sounds like you have both very strong
>     >> reservations, and I would like to understand them better.
>     >>
>     >> The behaviour can be surprising, but that is why the default is a
>     >> read-
>     >> only view.  I do not think it is worse than `np.broadcast_to` in this
>     >> regard. (It is nowhere near as dangerous as `as_strided`.)
>     >>
>     >> It is true that it is specific to NumPy (memory model). So that is
>     >> maybe a good enough reason right now.  But I am not sure that
>     >> stuffing
>     >> things into a pretty hidden `np.lib.*` namespaces is a great long
>     >> term
>     >> solution either. There is very little useful functionality hidden
>     >> away
>     >> in `np.lib.*` currently.
>     >>
>     >> Cheers,
>     >>
>     >> Sebastian
>     >>
>     >>>>
>     >>>>
>     >>>>> Cheers,
>     >>>>>
>     >>>>> Sebastian
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>     >>>>>> Hello,
>     >>>>>>
>     >>>>>> I would like to draw the attention of this list to PR #17394
>     >>>>>> [1] that
>     >>>>>> adds the implementation of a sliding window view to numpy.
>     >>>>>>
>     >>>>>> Having a sliding window view in numpy is a longstanding open
>     >>>>>> issue
>     >>>>>> (cf
>     >>>>>> #7753 [2] from 2016). A brief summary of the discussions
>     >>>>>> surrounding
>     >>>>>> it
>     >>>>>> can be found in the description of the PR.
>     >>>>>>
>     >>>>>> This PR implements a sliding window view based on stride
>     >>>>>> tricks.
>     >>>>>> Following the discussion in issue #7753, a first
>     >>>>>> implementation
>     >>>>>> was
>     >>>>>> provided by Fanjin Zeng in PR #10771. After some discussion,
>     >>>>>> that PR
>     >>>>>> stalled and I picked up the issue in the present PR #17394.
>     >>>>>> It
>     >>>>>> is
>     >>>>>> based
>     >>>>>> on the first implementation, but follows the changed API as
>     >>>>>> suggested
>     >>>>>> by
>     >>>>>> Eric Wieser.
>     >>>>>>
>     >>>>>> Code reviews have been provided by Bas van Beek, Stephen
>     >>>>>> Hoyer,
>     >>>>>> and
>     >>>>>> Eric
>     >>>>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>     >>>>>>
>     >>>>>>
>     >>>>>> Do you think this is suitable for inclusion in numpy?
>     >>>>>>
>     >>>>>> Do you consider the PR ready?
>     >>>>>>
>     >>>>>> Do you have suggestions or requests?
>     >>>>>>
>     >>>>>>
>     >>>>>> Thanks for your time and consideration!
>     >>>>>> Klaus
>     >>>>>>
>     >>>>>>
>     >>>>>> [1] https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >>>>>> [2] https://github.com/numpy/numpy/issues/7753
>     <https://github.com/numpy/numpy/issues/7753>
>     >>>>>> _______________________________________________
>     >>>>>> NumPy-Discussion mailing list
>     >>>>>> [hidden email] <mailto:[hidden email]>
>     >>>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>>>
>     >>>>>
>     >>>>> _______________________________________________
>     >>>>> NumPy-Discussion mailing list
>     >>>>> [hidden email] <mailto:[hidden email]>
>     >>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>>
>     >>>> _______________________________________________
>     >>>> NumPy-Discussion mailing list
>     >>>> [hidden email] <mailto:[hidden email]>
>     >>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>>>
>     >>>
>     >>> _______________________________________________
>     >>> NumPy-Discussion mailing list
>     >>> [hidden email] <mailto:[hidden email]>
>     >>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >>
>     >> _______________________________________________
>     >> NumPy-Discussion mailing list
>     >> [hidden email] <mailto:[hidden email]>
>     >> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >
>     >
>     > _______________________________________________
>     > NumPy-Discussion mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >
>     _______________________________________________
>     NumPy-Discussion mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Add sliding_window_view method to numpy

Zimmermann Klaus
Hi Ralf,

On 26/11/2020 15:17, Ralf Gommers wrote:
> On Fri, Nov 6, 2020 at 4:03 PM Zimmermann Klaus
> <[hidden email] <mailto:[hidden email]>> wrote:
>     I was just wondering if, of the top your head, an existing, better fit
>     comes to mind?
>
>
> Not really. Outside of stride_tricks there's nothing that quite fits.
> This function is more in scope for something like scipy.signal.

Alright, let's keep it as is then.

Thanks and cheers
Klaus


>
> Cheers,
> Ralf
>
>
>
>     >¬† ¬† ¬†The reason
>     >¬† ¬† ¬†from my point of view is that stride tricks is really a
>     technical (and
>     >¬† ¬† ¬†slightly ominous) name that might throw of more application
>     oriented
>     >¬† ¬† ¬†programmers from finding and using this function. Thinking of my
>     >¬† ¬† ¬†scientist colleagues, I think those are exactly the kind of
>     users that
>     >¬† ¬† ¬†could benefit from such a prototyping tool.
>     >
>     >
>     > That phrasing is one of a number of concerns. NumPy is normally not in
>     > the business of providing things that are okay as a prototyping tool,
>     > but are potentially extremely slow (as pointed out in the Notes
>     section
>     > of the docstring). A function like that would basically not be the
>     right
>     > tool for almost anything in, e.g., SciPy - it requires an iterative
>     > algorithm. In NumPy we don't prefer performance at all costs, but in
>     > general it's pretty decent rather than "Numba or Cython may gain you
>     > 100x here".
>
>     I still think that the performance concern is a bit overblown. Yes,
>     application with large windows can need more FLOPs by an equally large
>     factor. But most such applications will use small to moderate windows.
>     Furthermore, this view focuses only on FLOPs. In my current field of
>     climate science (and many others), that is almost never the limiting
>     factor. Memory demands are far more problematic and incidentally, those
>     are more likely to increase in other methods that require the storage of
>     ancillary, temporary data.
>
>     > Other issues include:
>     > 2) It is very specific to NumPy's memory model (as pointed out by you
>     > and Sebastian) - just like the rest of stride_tricks
>     Not wrong, but on the other hand, that memory model is not exotic. C,
>     Fortran, and any number of other languages play very nicely with this,
>     just as important downstream libraries like dask.
>
>     > 3) It has "view" in the name, which doesn't quite make sense for the
>     > main namespace (also connected to point 2 above).
>     Ok.
>
>     > 4) The cost of putting something in the main namespace for other
>     > array/tensor libraries is large. Maybe other libraries, e.g. CuPy,
>     Dask,
>     > TensorFlow, PyTorch, JAX, MXNet, aim to reimplement part or all of the
>     > main NumPy namespace as well as possible. This would trigger
>     discussions
>     > and likely many person-weeks of work for others.
>     Agreed. Though I have to say that my whole motivation comes from
>     corresponding issues in dask that where specifically waiting for (the
>     older version of) this PR (see [1, 2,...]). But I understand that dask
>     is effectively much closer to the numpy memory model than, say, CuPy, so
>     don't take this to mean it should be in the main namespace.
>
>     > 5) It's a useful function, but it's very much on the margins of
>     NumPy's
>     > scope. It could easily have gone into, for example, scipy.signal. At
>     > this point the bar for functions going into the main namespace
>     should be> (and is) high.
>     I agree that the bar for the main namespace should be high!
>
>     > All this taken together means it's not even a toss-up for me. If
>     it were
>     > just one or two of these points, maybe. But given all the above, I'm
>     > pretty confident saying "it does not belong in the main namespace".
>     Again, I am happy with that.
>
>
>     Thanks for your thoughts and work! I really appreciate it!
>
>     Cheers
>     Klaus
>
>     [1] https://github.com/dask/dask/issues/4659
>     <https://github.com/dask/dask/issues/4659>
>     [2] https://github.com/pydata/xarray/issues/3608
>     <https://github.com/pydata/xarray/issues/3608>
>     [3] https://github.com/pandas-dev/pandas/issues/26959
>     <https://github.com/pandas-dev/pandas/issues/26959>
>
>
>     >
>     >
>     >¬† ¬† ¬†Cheers
>     >¬† ¬† ¬†Klaus
>     >
>     >
>     >
>     >¬† ¬† ¬†[1]
>     https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
>     <https://github.com/numpy/numpy/pull/17394#issuecomment-700998618>
>     >¬† ¬†
>     ¬†<https://github.com/numpy/numpy/pull/17394#issuecomment-700998618
>     <https://github.com/numpy/numpy/pull/17394#issuecomment-700998618>>
>     >¬† ¬† ¬†[2]
>     https://github.com/numpy/numpy/pull/17394#discussion_r498215468
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498215468>
>     >¬† ¬†
>     ¬†<https://github.com/numpy/numpy/pull/17394#discussion_r498215468
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498215468>>
>     >¬† ¬† ¬†[3]
>     https://github.com/numpy/numpy/pull/17394#discussion_r498724340
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498724340>
>     >¬† ¬†
>     ¬†<https://github.com/numpy/numpy/pull/17394#discussion_r498724340
>     <https://github.com/numpy/numpy/pull/17394#discussion_r498724340>>
>     >
>     >¬† ¬† ¬†On 06/11/2020 01:39, Sebastian Berg wrote:
>     >¬† ¬† ¬†> On Thu, 2020-11-05 at 17:35 -0600, Sebastian Berg wrote:
>     >¬† ¬† ¬†>> On Thu, 2020-11-05 at 12:51 -0800, Stephan Hoyer wrote:
>     >¬† ¬† ¬†>>> On Thu, Nov 5, 2020 at 11:16 AM Ralf Gommers <
>     >¬† ¬† ¬†>>> [hidden email] <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     >¬† ¬† ¬†>>> wrote:
>     >¬† ¬† ¬†>>>
>     >¬† ¬† ¬†>>>> On Thu, Nov 5, 2020 at 4:56 PM Sebastian Berg <
>     >¬† ¬† ¬†>>>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email] <mailto:[hidden email]>>>
>     >¬† ¬† ¬†>>>> wrote:
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>>> Hi all,
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> just a brief note that I merged this proposal:
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>>¬† ¬† ¬†https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >¬† ¬† ¬†<https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>>
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> adding `np.sliding_window_view` into the 1.20 release of
>     NumPy.
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> There was only one public API change, and that is that the
>     >¬† ¬† ¬†>>>>> `shape`
>     >¬† ¬† ¬†>>>>> argument is now called `window_shape`.
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> This is still a good time for feedback in case you have a
>     >¬† ¬† ¬†>>>>> better
>     >¬† ¬† ¬†>>>>> idea
>     >¬† ¬† ¬†>>>>> e.g. for the function or parameter names.
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>> The old PR had this in the lib.stride_tricks namespace.
>     Seeing it
>     >¬† ¬† ¬†>>>> in the
>     >¬† ¬† ¬†>>>> main namespace is unexpected and likely will lead to
>     >¬† ¬† ¬†>>>> issues/questions,
>     >¬† ¬† ¬†>>>> given that such an overlapping view is going to do behave
>     in ways
>     >¬† ¬† ¬†>>>> the
>     >¬† ¬† ¬†>>>> average user will be surprised by. It may also lead to
>     requests
>     >¬† ¬† ¬†>>>> for
>     >¬† ¬† ¬†>>>> other
>     >¬† ¬† ¬†>>>> array/tensor libraries to implement this. I don't see any
>     >¬† ¬† ¬†>>>> discussion on
>     >¬† ¬† ¬†>>>> this in PR 17394, it looks like a decision by the PR
>     author that
>     >¬† ¬† ¬†>>>> no
>     >¬† ¬† ¬†>>>> one
>     >¬† ¬† ¬†>>>> commented on - reconsider that?
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>> Cheers,
>     >¬† ¬† ¬†>>>> Ralf
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>
>     >¬† ¬† ¬†>>> +1 let's keep this in the lib.stride_tricks namespace.
>     >¬† ¬† ¬†>>>
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> I have no reservations against having it in the main
>     namespace and am
>     >¬† ¬† ¬†>> happy either way (it can still be exposed later in any
>     case). It is
>     >¬† ¬† ¬†>> the
>     >¬† ¬† ¬†>> conservative choice and maybe it is an uncommon enough
>     function that
>     >¬† ¬† ¬†>> it
>     >¬† ¬† ¬†>> deserves being a bit hidden...
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†> In any case, its the safe bet for NumPy 1.20 at least so I
>     opened
>     >¬† ¬† ¬†a PR:
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>¬† ¬† ¬†https://github.com/numpy/numpy/pull/17720
>     <https://github.com/numpy/numpy/pull/17720>
>     >¬† ¬† ¬†<https://github.com/numpy/numpy/pull/17720
>     <https://github.com/numpy/numpy/pull/17720>>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†> Name changes, etc. are also possible of course.
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†> I still think it might be nice to find a better place for
>     this type of
>     >¬† ¬† ¬†> function that `np.lib.stride_tricks` though, but dunno...
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†> - Sebastian
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> But I am curious, it sounds like you have both very strong
>     >¬† ¬† ¬†>> reservations, and I would like to understand them better.
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> The behaviour can be surprising, but that is why the
>     default is a
>     >¬† ¬† ¬†>> read-
>     >¬† ¬† ¬†>> only view.¬† I do not think it is worse than
>     `np.broadcast_to` in this
>     >¬† ¬† ¬†>> regard. (It is nowhere near as dangerous as `as_strided`.)
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> It is true that it is specific to NumPy (memory model). So
>     that is
>     >¬† ¬† ¬†>> maybe a good enough reason right now.¬† But I am not sure that
>     >¬† ¬† ¬†>> stuffing
>     >¬† ¬† ¬†>> things into a pretty hidden `np.lib.*` namespaces is a
>     great long
>     >¬† ¬† ¬†>> term
>     >¬† ¬† ¬†>> solution either. There is very little useful functionality
>     hidden
>     >¬† ¬† ¬†>> away
>     >¬† ¬† ¬†>> in `np.lib.*` currently.
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> Cheers,
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> Sebastian
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>>> Cheers,
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> Sebastian
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> On Mon, 2020-10-12 at 08:39 +0000, Zimmermann Klaus wrote:
>     >¬† ¬† ¬†>>>>>> Hello,
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> I would like to draw the attention of this list to PR
>     #17394
>     >¬† ¬† ¬†>>>>>> [1] that
>     >¬† ¬† ¬†>>>>>> adds the implementation of a sliding window view to numpy.
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Having a sliding window view in numpy is a longstanding
>     open
>     >¬† ¬† ¬†>>>>>> issue
>     >¬† ¬† ¬†>>>>>> (cf
>     >¬† ¬† ¬†>>>>>> #7753 [2] from 2016). A brief summary of the discussions
>     >¬† ¬† ¬†>>>>>> surrounding
>     >¬† ¬† ¬†>>>>>> it
>     >¬† ¬† ¬†>>>>>> can be found in the description of the PR.
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> This PR implements a sliding window view based on stride
>     >¬† ¬† ¬†>>>>>> tricks.
>     >¬† ¬† ¬†>>>>>> Following the discussion in issue #7753, a first
>     >¬† ¬† ¬†>>>>>> implementation
>     >¬† ¬† ¬†>>>>>> was
>     >¬† ¬† ¬†>>>>>> provided by Fanjin Zeng in PR #10771. After some
>     discussion,
>     >¬† ¬† ¬†>>>>>> that PR
>     >¬† ¬† ¬†>>>>>> stalled and I picked up the issue in the present PR #17394.
>     >¬† ¬† ¬†>>>>>> It
>     >¬† ¬† ¬†>>>>>> is
>     >¬† ¬† ¬†>>>>>> based
>     >¬† ¬† ¬†>>>>>> on the first implementation, but follows the changed API as
>     >¬† ¬† ¬†>>>>>> suggested
>     >¬† ¬† ¬†>>>>>> by
>     >¬† ¬† ¬†>>>>>> Eric Wieser.
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Code reviews have been provided by Bas van Beek, Stephen
>     >¬† ¬† ¬†>>>>>> Hoyer,
>     >¬† ¬† ¬†>>>>>> and
>     >¬† ¬† ¬†>>>>>> Eric
>     >¬† ¬† ¬†>>>>>> Wieser. Sebastian Berg added the "62 - Python API" label.
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Do you think this is suitable for inclusion in numpy?
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Do you consider the PR ready?
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Do you have suggestions or requests?
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> Thanks for your time and consideration!
>     >¬† ¬† ¬†>>>>>> Klaus
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>> [1] https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>
>     >¬† ¬† ¬†<https://github.com/numpy/numpy/pull/17394
>     <https://github.com/numpy/numpy/pull/17394>>
>     >¬† ¬† ¬†>>>>>> [2] https://github.com/numpy/numpy/issues/7753
>     <https://github.com/numpy/numpy/issues/7753>
>     >¬† ¬† ¬†<https://github.com/numpy/numpy/issues/7753
>     <https://github.com/numpy/numpy/issues/7753>>
>     >¬† ¬† ¬†>>>>>> _______________________________________________
>     >¬† ¬† ¬†>>>>>> NumPy-Discussion mailing list
>     >¬† ¬† ¬†>>>>>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†>>>>>>
>     https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>>>>>>
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>>> _______________________________________________
>     >¬† ¬† ¬†>>>>> NumPy-Discussion mailing list
>     >¬† ¬† ¬†>>>>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†>>>>>
>     https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>>>>>
>     >¬† ¬† ¬†>>>> _______________________________________________
>     >¬† ¬† ¬†>>>> NumPy-Discussion mailing list
>     >¬† ¬† ¬†>>>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†>>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>>>>
>     >¬† ¬† ¬†>>>
>     >¬† ¬† ¬†>>> _______________________________________________
>     >¬† ¬† ¬†>>> NumPy-Discussion mailing list
>     >¬† ¬† ¬†>>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†>>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>>
>     >¬† ¬† ¬†>> _______________________________________________
>     >¬† ¬† ¬†>> NumPy-Discussion mailing list
>     >¬† ¬† ¬†>> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†>> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†> _______________________________________________
>     >¬† ¬† ¬†> NumPy-Discussion mailing list
>     >¬† ¬† ¬†> [hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†> https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >¬† ¬† ¬†>
>     >¬† ¬† ¬†_______________________________________________
>     >¬† ¬† ¬†NumPy-Discussion mailing list
>     >¬† ¬† ¬†[hidden email]
>     <mailto:[hidden email]>
>     <mailto:[hidden email]
>     <mailto:[hidden email]>>
>     >¬† ¬† ¬†https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >¬† ¬† ¬†<https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>>
>     >
>     >
>     > _______________________________________________
>     > NumPy-Discussion mailing list
>     > [hidden email] <mailto:[hidden email]>
>     > https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>     >
>     _______________________________________________
>     NumPy-Discussion mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://mail.python.org/mailman/listinfo/numpy-discussion
>     <https://mail.python.org/mailman/listinfo/numpy-discussion>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion