very simple iteration question.

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

very simple iteration question.

a g-3
Hi.  This is a very basic question, sorry if it's irritating.  If i
didn't find the answer written already somewhere on the site, please
point me to it.  That'd be great.

OK: how do i iterate over an axis other than 0?

I have a 3D array of data[year, week, location].  I want to iterate
over each year at each location and run a series of stats on the
columns (on the 52 weeks in a particular year at a particular location).
 'for years in data:' will get the first one, but then how do i not
iterate over the 1 axis and iterate over the 2 axis instead?

thanks,
alex.
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Nadav Horesh

for i in range(52):
   week_data = data[:,i,:]

OR

for week_data in data.transpose(1,0,2):
   ...

  Nadav


-----הודעה מקורית-----
מאת: [hidden email] בשם a g
נשלח: ד 30-אפריל-08 11:11
אל: [hidden email]
נושא: [Numpy-discussion] very simple iteration question.
 
Hi.  This is a very basic question, sorry if it's irritating.  If i
didn't find the answer written already somewhere on the site, please
point me to it.  That'd be great.

OK: how do i iterate over an axis other than 0?

I have a 3D array of data[year, week, location].  I want to iterate
over each year at each location and run a series of stats on the
columns (on the 52 weeks in a particular year at a particular location).
 'for years in data:' will get the first one, but then how do i not
iterate over the 1 axis and iterate over the 2 axis instead?

thanks,
alex.
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Anne Archibald
In reply to this post by a g-3
2008/4/30 a g <[hidden email]>:

> Hi.  This is a very basic question, sorry if it's irritating.  If i
>  didn't find the answer written already somewhere on the site, please
>  point me to it.  That'd be great.
>
>  OK: how do i iterate over an axis other than 0?
>
>  I have a 3D array of data[year, week, location].  I want to iterate
>  over each year at each location and run a series of stats on the
>  columns (on the 52 weeks in a particular year at a particular location).
>   'for years in data:' will get the first one, but then how do i not
>  iterate over the 1 axis and iterate over the 2 axis instead?

Well, there's always

for i in xrange(A.shape[1]):
    do_something(A[:,i,...])

But that's kind of ugly in this day and age. I would be tempted by

for a in np.rollaxis(A,1):
    do_something(a)

It's worth mentioning that traversing an array along an axis not the
first usually results in subarrays that are not contiguous in memory.
While numpy makes working with these convenient, they may not be very
efficient for cache reasons (for example, modern processors load from
memory - an extremely slow operation - in blocks that may be as large
as 64 bytes; if you only use eight of these before moving on, your
code will use much more memory bandwidth than it needs to). If this is
a concern, you can usually transparently rearrange your array's memory
layout to avoid it.

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Damian Eads-2
In reply to this post by a g-3
Hi Alex,

a g wrote:
> Hi.  This is a very basic question, sorry if it's irritating.  If i
> didn't find the answer written already somewhere on the site, please
> point me to it.  That'd be great.

You should look at any of the documents below and read up on array
slicing. It is perhaps the most important and pervasive concept of Numpy
and should be understood by all users.

     Numpy Tutorial: http://www.scipy.org/Tentative_NumPy_Tutorial
     Numpy for MATLAB users: http://www.scipy.org/NumPy_for_Matlab_Users
     Guide to Numpy

> OK: how do i iterate over an axis other than 0?
>
> I have a 3D array of data[year, week, location].  I want to iterate
> over each year at each location and run a series of stats on the
> columns (on the 52 weeks in a particular year at a particular location).
>  'for years in data:' will get the first one, but then how do i not
> iterate over the 1 axis and iterate over the 2 axis instead?

It is not clear to me whether you want to slice or iterate over an
array. Assuming you are fixing the year and location, the following code
iterates over data for fixed year and location.

for week in xrange(0, 52):
     <do something with> data[year, week, loc]

Slicing is more efficient and you should use it if you can. Fixing the
year and location, the following computes the mean and standard
deviation across all weeks. All of the statements below yield scalars.

     data[year, :, loc].mean() -- takes the mean of the data across weeks
     data[year, :, loc].std() -- takes the standard deviation of the
data across weeks

You should download IPython and type help(numpy.array) to see one set of
functions you can call on the result of a slice (sum, min, etc.).

Although I don't know what statistics you are computing for sure, the
following code might be useful since it computes a statistic across all
weeks for each year and location value.

     data.mean(axis=1)

It yields a num_years by num_locations array mu where mu[y, l] is the
average data value across all weeks for year y and loc l.

I hope this helps.

Damian
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Chris Barker - NOAA Federal
In reply to this post by a g-3
a g wrote:
> OK: how do i iterate over an axis other than 0?

This ties in nicely with some of the discussion about interating over
matrices. It ahs been suggested that it would be nice to have iterators
for matrices, so you could do:

for row in M.rows:
   ...

and

for column in M.cols:
   ...


If so, then wouldn't it make sense to have built in iterators for
nd-arrays as well? something like:

for subarray in A.iter(axis=i):
    ...

where axis would default to 0.

This is certainly cleaner than:

for j in range(A.shape[i]):
     subarray = A[:,:,j,:]

Wait! I have no idea how to spell that generically -- i.e. the ith
index, where i is a variable. There must be a way to build a slice
object dynamically, but I don't know it.

How often would i be a variable, rather than hard coded? I have no idea.

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Anne Archibald
2008/4/30 Christopher Barker <[hidden email]>:

> a g wrote:
>  > OK: how do i iterate over an axis other than 0?
>
>  This ties in nicely with some of the discussion about interating over
>  matrices. It ahs been suggested that it would be nice to have iterators
>  for matrices, so you could do:
>
>  for row in M.rows:
>    ...
>
>  and
>
>  for column in M.cols:
>    ...
>
>
>  If so, then wouldn't it make sense to have built in iterators for
>  nd-arrays as well? something like:
>
>  for subarray in A.iter(axis=i):
>     ...
>
>  where axis would default to 0.
>
>  This is certainly cleaner than:
>
>  for j in range(A.shape[i]):
>      subarray = A[:,:,j,:]
>
>  Wait! I have no idea how to spell that generically -- i.e. the ith
>  index, where i is a variable. There must be a way to build a slice
>  object dynamically, but I don't know it.

Slices can be built without too much trouble using slice(), but it's
much easier to just write

for subarray in np.rollaxis(A,i):
    ...

rollaxis() just pulls the specified axis to the front. (It doesn't do
what I thought it would, which is a cyclic permutation of the axes).

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Chris Barker - NOAA Federal
Anne Archibald wrote:
>> it's much easier to just write
>
> for subarray in np.rollaxis(A,i):
>     ...

cool, thanks! So the answer to the OPs question:

> OK: how do i iterate over an axis other than 0?
>
> I have a 3D array of data[year, week, location].  I want to iterate
> over each year at each location  ...

for loc in np.rollaxis(data, 2):
     for year in np.rollaxis(data, 0): # rollaxis not required here, but
                                       # for symmetry's sake...

     ....


I think I still like the idea of an iterator (or maybe making rollaxis a
method?), but this works pretty well.

-Chris



--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Gael Varoquaux
On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
> I think I still like the idea of an iterator (or maybe making rollaxis a
> method?), but this works pretty well.

Generally, in object oriented programming, you expect a method like
rollaxis to modify an object inplace. At least that would be my
expectation.

Gaël
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Alexander Michael
On Wed, Apr 30, 2008 at 3:09 PM, Gael Varoquaux
<[hidden email]> wrote:
> On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
>  > I think I still like the idea of an iterator (or maybe making rollaxis a
>  > method?), but this works pretty well.
>
>  Generally, in object oriented programming, you expect a method like
>  rollaxis to modify an object inplace. At least that would be my
>  expectation.

Gaël,

I'm not sure where you learned this expectation, as I don't think (but
I could be wrong -- so educate me!) it is universal for OOP nor
encouraged for Python in particular. OOP promotes encapsulation by
providing mechanisms for objects to publish an interface and thereby
hide the details of their inner workings, but I don't think there is a
common OOP philosophy that disallows methods returning views and
requires all transformations be performed in-place. A design
philosophy like you've espoused is tractable in languages that support
overloading, like C++, but it is not tractable in Python (at least not
cleanly within the design of the base language). How do you create a
function that returns a "flat" iterator of a container generically? In
C++, each container would overload the flat function. In Python, your
only hope is to ensure that the flat function only requires a
standardized interface to a container object, but that would likely
depend on the standardized interface including some low level
iteration method from which to build more complex iterators. Even in
C++ where such a design philosophy is possible at some level, objects
often expose const iterators of various sorts as methods. Finally,
Python has plenty of counter-examples to the maxim: all the string
methods because strings are designed to be immutable, set object
methods (even for mutable sets), various iterators for dicts, etc.

On this topic. I would love to see numpy evolve a moderately generic
array interface so that we could write stand-alone functions that work
generically with ndarrays, as well as "masked arrays" and "sparse
arrays" so that there was "one dot-product to rule them all", so to
speak. Right now, you can't use functions like numpy.dot on a
numpy.ma.MaskedArray, for example.

Regards,
Alex
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: very simple iteration question.

Bryan Cole-2
In reply to this post by Gael Varoquaux

On Wed, 2008-04-30 at 21:09 +0200, Gael Varoquaux wrote:
> On Wed, Apr 30, 2008 at 11:57:44AM -0700, Christopher Barker wrote:
> > I think I still like the idea of an iterator (or maybe making rollaxis a
> > method?), but this works pretty well.
>
> Generally, in object oriented programming, you expect a method like
> rollaxis to modify an object inplace. At least that would be my
> expectation.

BTW. rollaxis isn't a method.

I was completely unaware of this function. Learned something new
today...

BC

>
> Gaël


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion