recarray fun

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

recarray fun

Chris Barker - NOAA Federal
Hi folks,

Someone on the wxPython list posted a nifty recarray example that I
don't quite understand. The idea is to have an array for an RGBA image:

rgbarec = numpy.dtype({'r':(numpy.uint8,0),
                        'g':(numpy.uint8,1),
                        'b':(numpy.uint8,2),
                        'a':(numpy.uint8,3)})

A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) )

what I don't understand is having BOTH numpy.uint32 and rgbrec as the
dtype. How does that work?

Actually, with a bit of testing, it's pretty cool -- if you index like:

A[i,j] ==> uint32

A['r'][i,j] ==> uint8

pretty cool really.

it seems that numpy is treating it as both a recarray and a regular
uint32 array, which makes some sense, but I'm still confused by the
semantics.

also:
 >>> A
array([[4278190080,
...
        [4278190080, 4278190080, 4278190080, 4278190080, 4278190080]],
dtype=uint32)

but:
 >>> A.dtype
dtype(('>u4', [('r', '|u1'), ('g', '|u1'), ('b', '|u1'), ('a', '|u1')]))

so what is the dtype?

Also, I see advantages and disadvantages to either way. If you do:

A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) )

then you can do:

A[i,j]['r'] to get the red value of a pixel
A[i,j] = (red, green, blue, alpha) to set a pixel
A[:,:] = (red, green, blue, alpha) to make the whole image one color

With the "dual dtype" approach:
A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) )

You need to set the pixels separately:
A['r'][i,j] = red
B['r'][i,j] = green
...
or construct uint32 values:

C = numpy.uint32(red)
C += numpy.uint32(green) << 8
C += numpy.uint32(blue) << 16

Which is really ugly! (and I may not even have it right!). Is there a
better way?

One more idea -- is there a way to have two arrays, each with a
different dtype, that point to the same data? maybe:

RGBImage = numpy.zeros(shape, dtype=rgbarec )

IntImage = RGBImage.view()
IntImage.dtype = numpy.uint32

That way you could work with it either way, whichever was easier in the
context.

So -- what are folks' thoughts about how best to deal with images as
numpy arrays? (I'll put it in a Wiki page if I get some good comments)

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Travis Oliphant-5
Christopher Barker wrote:

> Hi folks,
>
> Someone on the wxPython list posted a nifty recarray example that I
> don't quite understand. The idea is to have an array for an RGBA image:
>
> rgbarec = numpy.dtype({'r':(numpy.uint8,0),
>                         'g':(numpy.uint8,1),
>                         'b':(numpy.uint8,2),
>                         'a':(numpy.uint8,3)})
>
> A = numpy.zeros(shape, dtype=(numpy.uint32, rgbarec) )
>
> what I don't understand is having BOTH numpy.uint32 and rgbrec as the
> dtype. How does that work?
>  

Basically, the rgbarec defines the fields, but the "base-type" for the
numpy array is numpy.uint32 (rather than VOID which is the default
data-type for arrays with fields defined).   This is why it prints the
way it does.

I'm not sure what the real value is doing it that way as opposed to just
having two views on the data:  one as a uint32 and another as a "normal"
recarray with a VOID data-type underlying it) like you suggest at the end.

I think it's really just the "architecture" showing through to the user
layer.  

-Travis



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Stéfan van der Walt
In reply to this post by Chris Barker - NOAA Federal
Hi Chris

2008/4/30 Christopher Barker <[hidden email]>:
>  Someone on the wxPython list posted a nifty recarray example that I
>  don't quite understand. The idea is to have an array for an RGBA image:
>
>  rgbarec = numpy.dtype({'r':(numpy.uint8,0),
>                         'g':(numpy.uint8,1),
>                         'b':(numpy.uint8,2),
>                         'a':(numpy.uint8,3)})
>

I find manipulating data with records arrays highly intuitive; I had
the same approach in mind when I wrote

http://www.scipy.org/RecordArrays

>  One more idea -- is there a way to have two arrays, each with a
>  different dtype, that point to the same data? maybe:
>
>  RGBImage = numpy.zeros(shape, dtype=rgbarec )
>
>  IntImage = RGBImage.view()
>  IntImage.dtype = numpy.uint32
>
>  That way you could work with it either way, whichever was easier in the
>  context.

That's the way, or just rgba_image.view(numpy.int32).

Cheers
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Chris Barker - NOAA Federal
Stéfan van der Walt wrote:
> That's the way, or just rgba_image.view(numpy.int32).

ah -- interestingly, I tried:

rgba_image.view(dtype=numpy.int32)

and got:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: view() takes no keyword arguments

Since it is optional, shouldn't it be keyword argument?

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Stéfan van der Walt
2008/4/30 Christopher Barker <[hidden email]>:

> Stéfan van der Walt wrote:
>  > That's the way, or just rgba_image.view(numpy.int32).
>
>  ah -- interestingly, I tried:
>
>  rgba_image.view(dtype=numpy.int32)
>
>  and got:
>
>  Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
>  TypeError: view() takes no keyword arguments
>
>  Since it is optional, shouldn't it be keyword argument?

Thanks, fixed in r5115.

Cheers
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Travis Oliphant-5
Stéfan van der Walt wrote:

> 2008/4/30 Christopher Barker <[hidden email]>:
>  
>> Stéfan van der Walt wrote:
>>  > That's the way, or just rgba_image.view(numpy.int32).
>>
>>  ah -- interestingly, I tried:
>>
>>  rgba_image.view(dtype=numpy.int32)
>>
>>  and got:
>>
>>  Traceback (most recent call last):
>>    File "<stdin>", line 1, in <module>
>>  TypeError: view() takes no keyword arguments
>>
>>  Since it is optional, shouldn't it be keyword argument?
>>    
>
> Thanks, fixed in r5115.
>
>  
This was too hasty.   I had considered this before.

The problem with this is that the object can be either a type object or
a data-type object.  You can use view to both re-cast a numpy array as
another subtype or as another data-type.

So, please revert the change until a better solution is posted.


-Travis


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Chris Barker - NOAA Federal
Travis E. Oliphant wrote:

> Stéfan van der Walt wrote:
>> 2008/4/30 Christopher Barker <[hidden email]>:
>>>  Since it is optional, shouldn't it be keyword argument?
>>>    
>> Thanks, fixed in r5115.
>>
>>  
> This was too hasty.   I had considered this before.
>
> The problem with this is that the object can be either a type object or
> a data-type object.  You can use view to both re-cast a numpy array as
> another subtype or as another data-type.

Is the issue here that this is a slightly different meaning than the
"dtype" argument everywhere else? Frankly, that doesn't bother me -- it
is a superset of the functionality, is it not?

Though I guess I don't really understand quite what the difference is
between a subtype and a data-type. Or a type object vs. a datatype object.

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Anne Archibald
In reply to this post by Travis Oliphant-5
2008/5/1 Travis E. Oliphant <[hidden email]>:

> Stéfan van der Walt wrote:
>  > 2008/4/30 Christopher Barker <[hidden email]>:
>  >
>  >> Stéfan van der Walt wrote:
>  >>  > That's the way, or just rgba_image.view(numpy.int32).
>  >>
>  >>  ah -- interestingly, I tried:
>  >>
>  >>  rgba_image.view(dtype=numpy.int32)
>  >>
>  >>  and got:
>  >>
>  >>  Traceback (most recent call last):
>  >>    File "<stdin>", line 1, in <module>
>  >>  TypeError: view() takes no keyword arguments
>  >>
>  >>  Since it is optional, shouldn't it be keyword argument?
>  >>
>  >
>  > Thanks, fixed in r5115.
>  >
>  >
>  This was too hasty.   I had considered this before.
>
>  The problem with this is that the object can be either a type object or
>  a data-type object.  You can use view to both re-cast a numpy array as
>  another subtype or as another data-type.
>
>  So, please revert the change until a better solution is posted.

If we're going to support keyword arguments, shouldn't those two
options be *different* keywords (dtype and ndarray_subclass, say)?
Then a single non-keyword argument tells numpy to guess which one you
wanted...

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Stéfan van der Walt
In reply to this post by Travis Oliphant-5
2008/5/1 Travis E. Oliphant <[hidden email]>:

> Stéfan van der Walt wrote:
>  > 2008/4/30 Christopher Barker <[hidden email]>:
>  >
>  >> Stéfan van der Walt wrote:
>  >>  > That's the way, or just rgba_image.view(numpy.int32).
>  >>
>  >>  ah -- interestingly, I tried:
>  >>
>  >>  rgba_image.view(dtype=numpy.int32)
>  >>
>  >>  and got:
>  >>
>  >>  Traceback (most recent call last):
>  >>    File "<stdin>", line 1, in <module>
>  >>  TypeError: view() takes no keyword arguments
>  >>
>  >>  Since it is optional, shouldn't it be keyword argument?
>  >>
>  >
>  > Thanks, fixed in r5115.
>  >
>  >
>  This was too hasty.   I had considered this before.
>
>  The problem with this is that the object can be either a type object or
>  a data-type object.  You can use view to both re-cast a numpy array as
>  another subtype or as another data-type.
>
>  So, please revert the change until a better solution is posted.

OK, I see your point.  I'm working on a patch that does the following:

def view(type_or_dtype=None, dtype=None, type=None):
    if type_or_dtype:
         if dtype:
             raise ValueError("Cannot specify dtype twice")
         if type:
             raise ValueError("Cannot specify type twice")

        if isinstance(type_or_dtype,py_type):
            type = type_or_dtype

        if isinstance(type_or_dtype,numpy_dtype):
            dtype = type_or_dtype

    return x.view(type=type).view(dtype=dtype)

Would that be a satisfying solution?  I'll be back around 21:00 SAST
to attend to the matter.

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Travis Oliphant-5
Stéfan van der Walt wrote:

> 2008/5/1 Travis E. Oliphant <[hidden email]>:
>  
>> Stéfan van der Walt wrote:
>>  > 2008/4/30 Christopher Barker <[hidden email]>:
>>  >
>>  >> Stéfan van der Walt wrote:
>>  >>  > That's the way, or just rgba_image.view(numpy.int32).
>>  >>
>>  >>  ah -- interestingly, I tried:
>>  >>
>>  >>  rgba_image.view(dtype=numpy.int32)
>>  >>
>>  >>  and got:
>>  >>
>>  >>  Traceback (most recent call last):
>>  >>    File "<stdin>", line 1, in <module>
>>  >>  TypeError: view() takes no keyword arguments
>>  >>
>>  >>  Since it is optional, shouldn't it be keyword argument?
>>  >>
>>  >
>>  > Thanks, fixed in r5115.
>>  >
>>  >
>>  This was too hasty.   I had considered this before.
>>
>>  The problem with this is that the object can be either a type object or
>>  a data-type object.  You can use view to both re-cast a numpy array as
>>  another subtype or as another data-type.
>>
>>  So, please revert the change until a better solution is posted.
>>    
>
> OK, I see your point.  I'm working on a patch that does the following:
>
> def view(type_or_dtype=None, dtype=None, type=None):
>     if type_or_dtype:
>          if dtype:
>              raise ValueError("Cannot specify dtype twice")
>          if type:
>              raise ValueError("Cannot specify type twice")
>
>         if isinstance(type_or_dtype,py_type):
>             type = type_or_dtype
>
>         if isinstance(type_or_dtype,numpy_dtype):
>             dtype = type_or_dtype
>
>     return x.view(type=type).view(dtype=dtype)
>
> Would that be a satisfying solution?  I'll be back around 21:00 SAST
> to attend to the matter.
>  

Yes, I think that would work.   You need to do some checking for
type=None and dtype=None as well, though.

That way, the first argument would continue to work as now but be
labeled correctly, but it would also support dtype= and type= keywords.  

-Travis

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Chris Barker - NOAA Federal
Travis E. Oliphant wrote:
>> def view(type_or_dtype=None, dtype=None, type=None):

> Yes, I think that would work.

Is there a way to deprecate this for future API-incompatible versions?
It's better than non keywords, but a bit ugly.

Maybe we should have a Wiki page for "stuff we'd like to change, but
won't until major API breakage is otherwise occurring"

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Stéfan van der Walt
In reply to this post by Travis Oliphant-5
2008/5/1 Travis E. Oliphant <[hidden email]>:

>  > OK, I see your point.  I'm working on a patch that does the following:
>  >
>  > def view(type_or_dtype=None, dtype=None, type=None):
>  >     if type_or_dtype:
>  >          if dtype:
>  >              raise ValueError("Cannot specify dtype twice")
>  >          if type:
>  >              raise ValueError("Cannot specify type twice")
>  >
>  >         if isinstance(type_or_dtype,py_type):
>  >             type = type_or_dtype
>  >
>  >         if isinstance(type_or_dtype,numpy_dtype):
>  >             dtype = type_or_dtype
>  >
>  >     return x.view(type=type).view(dtype=dtype)
>  >
>  > Would that be a satisfying solution?  I'll be back around 21:00 SAST
>  > to attend to the matter.
>  >
>
>  Yes, I think that would work.   You need to do some checking for
>  type=None and dtype=None as well, though.
>
>  That way, the first argument would continue to work as now but be
>  labeled correctly, but it would also support dtype= and type= keywords.

Please review http://projects.scipy.org/scipy/numpy/changeset/5117.

Thanks
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Travis Oliphant-5

> Please review http://projects.scipy.org/scipy/numpy/changeset/5117.
>  
Stefan,

I don't think we really need the dtype_or_type keyword.    It seems that
we could just check the first argument (dtype) to see if it is a subtype
of the ndarray and assume that it is type= in that case.  

-Travis

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Alan G Isaac
In reply to this post by Chris Barker - NOAA Federal
On Thu, 01 May 2008, Christopher Barker apparently wrote:
> Maybe we should have a Wiki page for "stuff we'd like to change, but
> won't until major API breakage is otherwise occurring"


Perhaps <URL:http://www.scipy.org/ProposedEnhancements>
would suffice?

Cheers,
Alan




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Travis Oliphant-5
In reply to this post by Stéfan van der Walt
Stéfan van der Walt wrote:

> 2008/5/1 Travis E. Oliphant <[hidden email]>:
>  
>>  > OK, I see your point.  I'm working on a patch that does the following:
>>  >
>>  > def view(type_or_dtype=None, dtype=None, type=None):
>>  >     if type_or_dtype:
>>  >          if dtype:
>>  >              raise ValueError("Cannot specify dtype twice")
>>  >          if type:
>>  >              raise ValueError("Cannot specify type twice")
>>  >
>>  >         if isinstance(type_or_dtype,py_type):
>>  >             type = type_or_dtype
>>  >
>>  >         if isinstance(type_or_dtype,numpy_dtype):
>>  >             dtype = type_or_dtype
>>  >
>>  >     return x.view(type=type).view(dtype=dtype)
>>  >
>>  > Would that be a satisfying solution?  I'll be back around 21:00 SAST
>>  > to attend to the matter.
>>  >
>>
>>  Yes, I think that would work.   You need to do some checking for
>>  type=None and dtype=None as well, though.
>>
>>  That way, the first argument would continue to work as now but be
>>  labeled correctly, but it would also support dtype= and type= keywords.
>>    
>
> Please review http://projects.scipy.org/scipy/numpy/changeset/5117.
>
>  
Check out

http://projects.scipy.org/scipy/numpy/changeset/5119

-Travis




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: recarray fun

Stéfan van der Walt
2008/5/1 Travis E. Oliphant <[hidden email]>:

> Stéfan van der Walt wrote:
>
>
> > 2008/5/1 Travis E. Oliphant <[hidden email]>:
>  >
>  >>  > OK, I see your point.  I'm working on a patch that does the following:
>  >>  >
>  >>  > def view(type_or_dtype=None, dtype=None, type=None):
>  >>  >     if type_or_dtype:
>  >>  >          if dtype:
>  >>  >              raise ValueError("Cannot specify dtype twice")
>  >>  >          if type:
>  >>  >              raise ValueError("Cannot specify type twice")
>  >>  >
>  >>  >         if isinstance(type_or_dtype,py_type):
>  >>  >             type = type_or_dtype
>  >>  >
>  >>  >         if isinstance(type_or_dtype,numpy_dtype):
>  >>  >             dtype = type_or_dtype
>  >>  >
>  >>  >     return x.view(type=type).view(dtype=dtype)
>  >>  >
>  >>  > Would that be a satisfying solution?  I'll be back around 21:00 SAST
>  >>  > to attend to the matter.
>  >>  >
>  >>
>  >>  Yes, I think that would work.   You need to do some checking for
>  >>  type=None and dtype=None as well, though.
>  >>
>  >>  That way, the first argument would continue to work as now but be
>  >>  labeled correctly, but it would also support dtype= and type= keywords.
>  >>
>  >
>  > Please review http://projects.scipy.org/scipy/numpy/changeset/5117.
>  >
>  >
>  Check out
>
>  http://projects.scipy.org/scipy/numpy/changeset/5119

I think that's fine.  It doesn't support weird combinations like

x.view(np.matrix,dtype=np.int32)

but people probably shouldn't do that anyway.

Cheers
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion