Re: Using nditer + external_loop to Always Iterate by Column

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Using nditer + external_loop to Always Iterate by Column

William Ayd
I am trying to use the nditer to traverse each column of a 2D array, returning the column as a 1D array. Consulting the docs, I found this example which works perfectly fine:

In [65]: a = np.arange(6).reshape(2,3)                                                                                                                             

In [66]: for x in np.nditer(a, flags=['external_loop'], order='F'): 
    ...:     print(x, end=' '
    ...:                                                                                                                                                           
[0 3] [1 4] [2 5] 

When changing the shape of the input array to (1, 3) however, this doesn’t yield what I am hoping for any more (essentially [0], [1] [2]):

In [68]: for x in np.nditer(a, flags=['external_loop'], order='F'): 
    ...:     print(x, end=' '
    ...:                                                                                                                                                           
[0 1 2] 

I suspect this may have to do with the fact that the (1, 3) array is both C and F contiguous, and it is trying to return as large of a 1D F-contiguous array as it can. However, I didn’t see any way to really force it to go by columns. My best guess was the itershape argument though I couldn’t figure out how to get that to work and didn’t see much in the documentation.

Thanks in advance for the help!

- Will




_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Using nditer + external_loop to Always Iterate by Column

Hameer Abbasi

Hi William,

 

You can simply use a for loop for that task:

 

Python 3.8.2 | packaged by conda-forge | (default, Apr 24 2020, 07:56:27)

Type 'copyright', 'credits' or 'license' for more information

IPython 7.14.0 -- An enhanced Interactive Python. Type '?' for help.

 

In [1]: import numpy as np

 

In [2]: a = np.arange(3).reshape((1, 3))

 

In [3]: for x in a.T:

   ...:     print(x)

   ...:

[0]

[1]

[2]

 

Best regards,

Hameer Abbasi

 

From: NumPy-Discussion <numpy-discussion-bounces+einstein.edison=[hidden email]> on behalf of William Ayd <[hidden email]>
Reply to: Discussion of Numerical Python <[hidden email]>
Date: Tuesday, 19. May 2020 at 01:42
To: "[hidden email]" <[hidden email]>
Subject: Re: [Numpy-discussion] Using nditer + external_loop to Always Iterate by Column

 

I am trying to use the nditer to traverse each column of a 2D array, returning the column as a 1D array. Consulting the docs, I found this example which works perfectly fine:

 

In [65]: a = np.arange(6).reshape(2,3)                                                                                                                             

 

In [66]: for x in np.nditer(a, flags=['external_loop'], order='F'): 

    ...:     print(x, end=' '

    ...:                                                                                                                                                           

[0 3] [1 4] [2 5] 

 

When changing the shape of the input array to (1, 3) however, this doesn’t yield what I am hoping for any more (essentially [0], [1] [2]):

 

In [68]: for x in np.nditer(a, flags=['external_loop'], order='F'): 

    ...:     print(x, end=' '

    ...:                                                                                                                                                           

[0 1 2] 

 

I suspect this may have to do with the fact that the (1, 3) array is both C and F contiguous, and it is trying to return as large of a 1D F-contiguous array as it can. However, I didn’t see any way to really force it to go by columns. My best guess was the itershape argument though I couldn’t figure out how to get that to work and didn’t see much in the documentation.

 

Thanks in advance for the help!

 

- Will

 

 




_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Using nditer + external_loop to Always Iterate by Column

Eric Wieser
In reply to this post by William Ayd
Hi Will,

To force an iteration to run along certain axes, I believe you should be using `op_axes`. Your diagnosis is correct that `external_loop` is trying to help you be more optimal, since it's purpose is exactly that: optimization.

Unfortunately, if you use `op_axes` you'll run into https://github.com/numpy/numpy/issues/9808.

Eric

On Tue, 19 May 2020 at 00:42, William Ayd <[hidden email]> wrote:
I am trying to use the nditer to traverse each column of a 2D array, returning the column as a 1D array. Consulting the docs, I found this example which works perfectly fine:

In [65]: a = np.arange(6).reshape(2,3)                                                                                                                             

In [66]: for x in np.nditer(a, flags=['external_loop'], order='F'): 
    ...:     print(x, end=' '
    ...:                                                                                                                                                           
[0 3] [1 4] [2 5] 

When changing the shape of the input array to (1, 3) however, this doesn’t yield what I am hoping for any more (essentially [0], [1] [2]):

In [68]: for x in np.nditer(a, flags=['external_loop'], order='F'): 
    ...:     print(x, end=' '
    ...:                                                                                                                                                           
[0 1 2] 

I suspect this may have to do with the fact that the (1, 3) array is both C and F contiguous, and it is trying to return as large of a 1D F-contiguous array as it can. However, I didn’t see any way to really force it to go by columns. My best guess was the itershape argument though I couldn’t figure out how to get that to work and didn’t see much in the documentation.

Thanks in advance for the help!

- Will



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Using nditer + external_loop to Always Iterate by Column

Sebastian Berg
On Tue, 2020-05-19 at 17:11 +0100, Eric Wieser wrote:

> Hi Will,
>
> To force an iteration to run along certain axes, I believe you should
> be
> using `op_axes`. Your diagnosis is correct that `external_loop` is
> trying
> to help you be more optimal, since it's purpose is exactly that:
> optimization.
>
> Unfortunately, if you use `op_axes` you'll run into
> https://github.com/numpy/numpy/issues/9808.


Yeah, I do not think what you want is possible with nditer. `op_axes`
allows you to ignore certain axis, but as Eric points out, that
actually ignores that axis entirely. So it acts similar to slicing the
whole axis away.

I previously thought it could be nice to have a new
`arr.iteraxis(axis=None)` command. As a base, it would act like a flat
iter, otherwise iterate the axis listed. Maybe with an additional flag
whether its allowed to optimize iteration order (although from the
python side I expect we do not want to do that by default).

The reason is also that I am not sure I like `arr.flat` and `for subarr
in arr` too much, because the list-of-list like iteration seems only
semi-natural for an N-D array.

- Sebastian


>
> Eric
>
> On Tue, 19 May 2020 at 00:42, William Ayd <[hidden email]>
> wrote:
>
> > I am trying to use the nditer to traverse each column of a 2D
> > array,
> > returning the column as a 1D array. Consulting the docs, I found
> > this
> > example which works perfectly fine:
> >
> > In [*65*]: a = np.arange(6).reshape(2,3)
> >
> >
> >
> > In [*66*]: *for* x *in* np.nditer(a, flags=['external_loop'],
> > order='F'):
> >     ...:     print(x, end=' ')
> >     ...:
> >
> >
> > [0 3] [1 4] [2 5]
> >
> > When changing the shape of the input array to (1, 3) however, this
> > doesn’t
> > yield what I am hoping for any more (essentially [0], [1] [2]):
> >
> > In [*68*]: *for* x *in* np.nditer(a, flags=['external_loop'],
> > order='F'):
> >     ...:     print(x, end=' ')
> >     ...:
> >
> >
> > [0 1 2]
> >
> > I suspect this may have to do with the fact that the (1, 3) array
> > is both
> > C and F contiguous, and it is trying to return as large of a 1D
> > F-contiguous array as it can. However, I didn’t see any way to
> > really force
> > it to go by columns. My best guess was the *itershape* argument
> > though I
> > couldn’t figure out how to get that to work and didn’t see much in
> > the
> > documentation.
> >
> > Thanks in advance for the help!
> >
> > - Will
> >
> >
> >
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Using nditer + external_loop to Always Iterate by Column

William Ayd
In reply to this post by William Ayd
Thanks Hameer for the suggestion. I perhaps should have clarified that this is a small example for something larger, where I am iterating over a sequence of arrays, computing some reduction on the first and writing to the second. I am hoping to keep it all contained within one nditer.

Thanks Eric for the suggestion. Notwithstanding the bug that would seem to be promising. I’ll do a little more research and see what I can find.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion