array comprehension

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

array comprehension

Neal Becker
I find I often write:
np.array ([some list comprehension])

mainly because list comprehensions are just so sweet.

But I imagine this isn't particularly efficient.

I wonder if numpy has a "better" way, and if not, maybe it would be a nice
addition?

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Francesc Alted-3


2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
I find I often write:
np.array ([some list comprehension])

mainly because list comprehensions are just so sweet.

But I imagine this isn't particularly efficient.

Right.  Using a generator and np.fromiter() will avoid the creation of the intermediate list.  Something like:

np.fromiter((i for i in range(x)))  # use xrange for Python 2
 

I wonder if numpy has a "better" way, and if not, maybe it would be a nice
addition?

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



--
Francesc Alted

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Neal Becker
Francesc Alted wrote:

> 2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
>
>> I find I often write:
>> np.array ([some list comprehension])
>>
>> mainly because list comprehensions are just so sweet.
>>
>> But I imagine this isn't particularly efficient.
>>
>
> Right.  Using a generator and np.fromiter() will avoid the creation of the
> intermediate list.  Something like:
>
> np.fromiter((i for i in range(x)))  # use xrange for Python 2
>
>
Does this generalize to >1 dimensions?

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Francesc Alted-3


2016-11-04 14:36 GMT+01:00 Neal Becker <[hidden email]>:
Francesc Alted wrote:

> 2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
>
>> I find I often write:
>> np.array ([some list comprehension])
>>
>> mainly because list comprehensions are just so sweet.
>>
>> But I imagine this isn't particularly efficient.
>>
>
> Right.  Using a generator and np.fromiter() will avoid the creation of the
> intermediate list.  Something like:
>
> np.fromiter((i for i in range(x)))  # use xrange for Python 2
>
>
Does this generalize to >1 dimensions?

A reshape() is not enough?  What do you want to do exactly?
 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



--
Francesc Alted

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Stephan Hoyer-2
On Fri, Nov 4, 2016 at 7:12 AM, Francesc Alted <[hidden email]> wrote:
Does this generalize to >1 dimensions?

A reshape() is not enough?  What do you want to do exactly?

np.fromiter takes scalar input and only builds a 1D array. So it actually can't combine multiple values at once unless they are flattened out in Python. It could be nice to add support for non-scalar inputs, stacking them similarly to np.array. Likewise, it could be nice to add an axis argument, so it can work similarly to np.stack.

More generally, you might want to iterate and rebuild over arbitrary dimension(s) of an array. Something like
np.stack([x for x in np.unstack(y, axis)], axis)

But, we also don't have an unstack function. This would mostly be syntactic sugar, but I think it would be a nice addition. Such a function actually exists in TensorFlow:

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Daπid
On 4 November 2016 at 16:04, Stephan Hoyer <[hidden email]> wrote:
>
> But, we also don't have an unstack function. This would mostly be syntactic
> sugar, but I think it would be a nice addition. Such a function actually
> exists in TensorFlow:
> https://g3doc.corp.google.com/third_party/tensorflow/g3doc/api_docs/python/array_ops.md?cl=head#unstack

That link is behind a login wall. This is the public version:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/g3doc/api_docs/python/array_ops.md
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Ryan May-3
In reply to this post by Stephan Hoyer-2
On Fri, Nov 4, 2016 at 9:04 AM, Stephan Hoyer <[hidden email]> wrote:
On Fri, Nov 4, 2016 at 7:12 AM, Francesc Alted <[hidden email]> wrote:
Does this generalize to >1 dimensions?

A reshape() is not enough?  What do you want to do exactly?

np.fromiter takes scalar input and only builds a 1D array. So it actually can't combine multiple values at once unless they are flattened out in Python. It could be nice to add support for non-scalar inputs, stacking them similarly to np.array. Likewise, it could be nice to add an axis argument, so it can work similarly to np.stack.

 itertools.product, itertools.permutation, etc. with np.fromiter (and reshape) is probably also useful here, though it doesn't solve the non-scalar problem.

Ryan

--
Ryan May


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Neal Becker
In reply to this post by Francesc Alted-3
Francesc Alted wrote:

> 2016-11-04 14:36 GMT+01:00 Neal Becker <[hidden email]>:
>
>> Francesc Alted wrote:
>>
>> > 2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
>> >
>> >> I find I often write:
>> >> np.array ([some list comprehension])
>> >>
>> >> mainly because list comprehensions are just so sweet.
>> >>
>> >> But I imagine this isn't particularly efficient.
>> >>
>> >
>> > Right.  Using a generator and np.fromiter() will avoid the creation of
>> the
>> > intermediate list.  Something like:
>> >
>> > np.fromiter((i for i in range(x)))  # use xrange for Python 2
>> >
>> >
>> Does this generalize to >1 dimensions?
>>
>
> A reshape() is not enough?  What do you want to do exactly?
>

I was thinking about:
x = np.array ([[L1] L2]) where L1,L2 take the form of a list comprehension,
as a means to create a 2-D array (in this example)

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Robert Kern-2
In reply to this post by Neal Becker
On Fri, Nov 4, 2016 at 6:36 AM, Neal Becker <[hidden email]> wrote:

>
> Francesc Alted wrote:
>
> > 2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
> >
> >> I find I often write:
> >> np.array ([some list comprehension])
> >>
> >> mainly because list comprehensions are just so sweet.
> >>
> >> But I imagine this isn't particularly efficient.
> >>
> >
> > Right.  Using a generator and np.fromiter() will avoid the creation of the
> > intermediate list.  Something like:
> >
> > np.fromiter((i for i in range(x)))  # use xrange for Python 2
> >
> >
> Does this generalize to >1 dimensions?

No.

--
Robert Kern

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Nathaniel Smith
In reply to this post by Francesc Alted-3

Are you sure fromiter doesn't make an intermediate list or equivalent? It has to collect all the values before it can know the shape or dtype of the array to put them in.


On Nov 4, 2016 5:26 AM, "Francesc Alted" <[hidden email]> wrote:


2016-11-04 13:06 GMT+01:00 Neal Becker <[hidden email]>:
I find I often write:
np.array ([some list comprehension])

mainly because list comprehensions are just so sweet.

But I imagine this isn't particularly efficient.

Right.  Using a generator and np.fromiter() will avoid the creation of the intermediate list.  Something like:

np.fromiter((i for i in range(x)))  # use xrange for Python 2
 

I wonder if numpy has a "better" way, and if not, maybe it would be a nice
addition?

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



--
Francesc Alted

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Stephan Hoyer-2
On Fri, Nov 4, 2016 at 10:24 AM, Nathaniel Smith <[hidden email]> wrote:

Are you sure fromiter doesn't make an intermediate list or equivalent? It has to collect all the values before it can know the shape or dtype of the array to put them in.

fromiter dynamically resizes a NumPy array, like a Python list, except with a growth factor of 1.5 (rather than 1.25):

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Nathaniel Smith

On Nov 4, 2016 10:32 AM, "Stephan Hoyer" <[hidden email]> wrote:
>
> On Fri, Nov 4, 2016 at 10:24 AM, Nathaniel Smith <[hidden email]> wrote:
>>
>> Are you sure fromiter doesn't make an intermediate list or equivalent? It has to collect all the values before it can know the shape or dtype of the array to put them in.
>
> fromiter dynamically resizes a NumPy array, like a Python list, except with a growth factor of 1.5 (rather than 1.25):
> https://github.com/numpy/numpy/blob/bb59409abf5237c155a1dc4c4d5b31e4acf32fbe/numpy/core/src/multiarray/ctors.c#L3721 

Oh, right, and the dtype argument is mandatory, which is what makes this possible.

-n


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: array comprehension

Chris Barker - NOAA Federal
On Fri, Nov 4, 2016 at 10:36 AM, Nathaniel Smith <[hidden email]> wrote:

On Nov 4, 2016 10:32 AM, "Stephan Hoyer" <[hidden email]> wrote:
> fromiter dynamically resizes a NumPy array, like a Python list, except with a growth factor of 1.5

 

Oh, right, and the dtype argument is mandatory, which is what makes this possible.

Couldn't it determine the dtype from the first element, and then barf later if an incompatible one shows up?

And then we could adapt this code to np.array() and get nice performance with no extra functions to think about calling...

And off the top of my head, I can't think of why it couldn't be generalized to the nd case as well.

-CHB


--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Loading...