Forcing new dimensions to appear at front in advanced indexing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Forcing new dimensions to appear at front in advanced indexing

Michael Lamparski
Hi all,

So, in advanced indexing, numpy decides where to put new axes based on whether the "advanced indices" are all next to each other.

>>> np.random.random((3,4,5,6,7,8))[:, [[0,0],[0,0]], 1, :].shape
(3, 2, 2, 6, 7, 8)
>>> np.random.random((3,4,5,6,7,8))[:, [[0,0],[0,0]], :, 1].shape
(2, 2, 3, 5, 7, 8)

In creating a wrapper type around arrays, I'm finding myself needing to suppress this behavior, so that the new axes consistently appear in the front.  I thought of a dumb hat trick:

def index(x, indices):
    return x[(True, None) + indices]

Which certainly gets the new dimensions where I want them, but it introduces a ghost dimension of 1 (and sometimes two such dimensions!) in a place where I'm not sure I can easily find it.

>>> np.random.random((3,4,5,6,7,8))[True, None, 1].shape
(1, 1, 4, 5, 6, 7, 8)
>>> np.random.random((3,4,5,6,7,8))[True, None, :, [[0,0],[0,0]], 1, :].shape
(2, 2, 1, 3, 6, 7, 8)
>>> np.random.random((3,4,5,6,7,8))[True, None, :, [[0,0],[0,0]], :, 1].shape
(2, 2, 1, 3, 5, 7, 8)


any better ideas?

---

Michael

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Forcing new dimensions to appear at front in advanced indexing

Sebastian Berg
On Tue, 2018-06-19 at 19:37 -0400, Michael Lamparski wrote:

> Hi all,
>
> So, in advanced indexing, numpy decides where to put new axes based
> on whether the "advanced indices" are all next to each other.
>
> >>> np.random.random((3,4,5,6,7,8))[:, [[0,0],[0,0]], 1, :].shape
> (3, 2, 2, 6, 7, 8)
> >>> np.random.random((3,4,5,6,7,8))[:, [[0,0],[0,0]], :, 1].shape
> (2, 2, 3, 5, 7, 8)
>
> In creating a wrapper type around arrays, I'm finding myself needing
> to suppress this behavior, so that the new axes consistently appear
> in the front.  I thought of a dumb hat trick:
>
> def index(x, indices):
>     return x[(True, None) + indices]
>
> Which certainly gets the new dimensions where I want them, but it
> introduces a ghost dimension of 1 (and sometimes two such
> dimensions!) in a place where I'm not sure I can easily find it.
>
> >>> np.random.random((3,4,5,6,7,8))[True, None, 1].shape
> (1, 1, 4, 5, 6, 7, 8)
> >>> np.random.random((3,4,5,6,7,8))[True, None, :, [[0,0],[0,0]], 1,
> :].shape
> (2, 2, 1, 3, 6, 7, 8)
> >>> np.random.random((3,4,5,6,7,8))[True, None, :, [[0,0],[0,0]], :,
> 1].shape
> (2, 2, 1, 3, 5, 7, 8)
>
> any better ideas?
>
We have proposed `arr.vindex[...]` to do this and there are is a pure
python implementation of it out there, I think it may be linked here
somewhere:

https://github.com/numpy/numpy/pull/6256

There is a way that will generally work using triple indexing:

arr[..., None, None][orig_indx * (slice(None), np.array(0))][..., 0]

The first and last indexing operation is just a view creation, so it is
basically a no-op. Now doing this gives me the shiver, but it will work
always. If you want to have a no-copy behaviour in case your original
index is ont an advanced indexing operation, you should replace the
np.array(0) with just 0.

- Sebastian





> ---
>
> Michael
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Forcing new dimensions to appear at front in advanced indexing

Michael Lamparski
> There is a way that will generally work using triple indexing:
>
> arr[..., None, None][orig_indx + (slice(None), np.array(0))][..., 0]

Impressive! (note: I fixed the * typo in the quote)

The first and last indexing operation is just a view creation, so it is
> basically a no-op. Now doing this gives me the shiver, but it will work
> always. If you want to have a no-copy behaviour in case your original
> index is ont an advanced indexing operation, you should replace the
> np.array(0) with just 0.

I agree about the shivers, but any workaround is good to have nonetheless.

If the index is not an advanced indexing operation, does it not suffice to simply apply the index tuple as-is?

Michael

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Forcing new dimensions to appear at front in advanced indexing

Sebastian Berg
On Wed, 2018-06-20 at 09:15 -0400, Michael Lamparski wrote:

> > There is a way that will generally work using triple indexing:
> >
> > arr[..., None, None][orig_indx + (slice(None), np.array(0))][...,
> 0]
>
> Impressive! (note: I fixed the * typo in the quote)
>
> > The first and last indexing operation is just a view creation, so
> it is
> > basically a no-op. Now doing this gives me the shiver, but it will
> work
> > always. If you want to have a no-copy behaviour in case your
> original
> > index is ont an advanced indexing operation, you should replace the
> > np.array(0) with just 0.
>
> I agree about the shivers, but any workaround is good to have
> nonetheless.
>
> If the index is not an advanced indexing operation, does it not
> suffice to simply apply the index tuple as-is?
Yes, with the `np.array(0)` however, the result will forced to be a
copy and not a view into the original array, when writing the line
first I thought of "force advanced indexing", which there is likely no
reason for though.
If you replace it with 0, the result will be an identical view when the
index is not advanced (with only a tiny bit of call overhead).

So it might be nice to just use 0 instead, since if your index is
advanced indexing, there is no difference between the two. But then you
do not have to check if there is advanced indexing going on at all.

Btw. if you want to use it for an object, I might suggest to actually
use:

object.vindex[...]

notation for this logic (requires a slightly annoying helper class).
The NEP is basically just a draft/proposal status, but xarray is
already using that indexing method/property IIRC, so that name is
relatively certain by now.

I frankly am not sure right now if the vindex proposal was with a
forced copy or not, probably it was.

- Sebastian


>
> Michael
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment