1.1.0rc1 RuntimeErrors

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

1.1.0rc1 RuntimeErrors

Pierre GM-2
All,
Most of the errors that are reported in 1.1.0rc1 are related to the .tolist()
method in numpy.ma, such as :
############################
ERROR: Tests fields retrieval
----------------------------------------------------------------------
Traceback (most recent call last):
   File
"C:\Apps\Python\Lib\site-packages\numpy\ma\tests\test_mrecords.py", line
62, in test_get
     assert_equal(getattr(mbase,field), mbase[field])
   File "C:\Apps\Python\Lib\site-packages\numpy\ma\testutils.py", line
104, in assert_equal
     desired.tolist(),
   File "C:\Apps\Python\Lib\site-packages\numpy\ma\core.py", line 2552,
in tolist
     result = self.filled().tolist()
RuntimeError: array_item not returning smaller-dimensional array
#####################################

Note that the method seems to work, still: for example, the following command
gives the proper output, without RuntimeError

python -c "import numpy as np, numpy.ma as ma; x=ma.array(np.random.rand(5),
mask=[1,0,0,0,0]); print x.tolist()"

The problem looks quite recent, and not related to numpy.ma itself: what
changed recently in the .tolist() method of ndarrays ? Why do we get these
RuntimeErrors ?

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Charles R Harris


On Wed, May 21, 2008 at 9:28 AM, Pierre GM <[hidden email]> wrote:
All,
Most of the errors that are reported in 1.1.0rc1 are related to the .tolist()
method in numpy.ma, such as :
############################
ERROR: Tests fields retrieval
----------------------------------------------------------------------
Traceback (most recent call last):
   File
"C:\Apps\Python\Lib\site-packages\numpy\ma\tests\test_mrecords.py", line
62, in test_get
     assert_equal(getattr(mbase,field), mbase[field])
   File "C:\Apps\Python\Lib\site-packages\numpy\ma\testutils.py", line
104, in assert_equal
     desired.tolist(),
   File "C:\Apps\Python\Lib\site-packages\numpy\ma\core.py", line 2552,
in tolist
     result = self.filled().tolist()
RuntimeError: array_item not returning smaller-dimensional array
#####################################

Note that the method seems to work, still: for example, the following command
gives the proper output, without RuntimeError

python -c "import numpy as np, numpy.ma as ma; x=ma.array(np.random.rand(5),
mask=[1,0,0,0,0]); print x.tolist()"

The problem looks quite recent, and not related to numpy.ma itself: what
changed recently in the .tolist() method of ndarrays ? Why do we get these
RuntimeErrors ?


I expect it comes from the matrix churn.  The tolist method was one of those with a  workaround for the failure to reduce dimensions. I'll look at it later if someone doesn't beat me to it.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Alan McIntyre-2
On Wed, May 21, 2008 at 11:35 AM, Charles R Harris
<[hidden email]> wrote:
> On Wed, May 21, 2008 at 9:28 AM, Pierre GM <[hidden email]> wrote:
>> The problem looks quite recent, and not related to numpy.ma itself: what
>> changed recently in the .tolist() method of ndarrays ? Why do we get these
>> RuntimeErrors ?
>
> I expect it comes from the matrix churn.  The tolist method was one of those
> with a  workaround for the failure to reduce dimensions. I'll look at it
> later if someone doesn't beat me to it.

There's some commentary and a patch on NumPy ticket 793 on this issue:

http://scipy.org/scipy/numpy/ticket/793

Hope it's helpful.
Alan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Pierre GM-2
On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
> There's some commentary and a patch on NumPy ticket 793 on this issue:
>
> http://scipy.org/scipy/numpy/ticket/793

OK, thanks a lot ! That's a C problem, then...
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Alan McIntyre-2
On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]> wrote:
> On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
>> There's some commentary and a patch on NumPy ticket 793 on this issue:
>>
>> http://scipy.org/scipy/numpy/ticket/793
>
> OK, thanks a lot ! That's a C problem, then...

It's probably worth mentioning that I'm not that familiar with all the
innards of NumPy yet, so take my comments and patch on that issue with
a (fairly large) grain of salt. ;)
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Charles R Harris


On Wed, May 21, 2008 at 10:07 AM, Alan McIntyre <[hidden email]> wrote:
On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]> wrote:
> On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
>> There's some commentary and a patch on NumPy ticket 793 on this issue:
>>
>> http://scipy.org/scipy/numpy/ticket/793
>
> OK, thanks a lot ! That's a C problem, then...

It's probably worth mentioning that I'm not that familiar with all the
innards of NumPy yet, so take my comments and patch on that issue with
a (fairly large) grain of salt. ;)

This was introduced by Travis in r5138 as part of the matrix changes.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Charles R Harris


On Wed, May 21, 2008 at 2:39 PM, Charles R Harris <[hidden email]> wrote:


On Wed, May 21, 2008 at 10:07 AM, Alan McIntyre <[hidden email]> wrote:
On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]> wrote:
> On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
>> There's some commentary and a patch on NumPy ticket 793 on this issue:
>>
>> http://scipy.org/scipy/numpy/ticket/793
>
> OK, thanks a lot ! That's a C problem, then...

It's probably worth mentioning that I'm not that familiar with all the
innards of NumPy yet, so take my comments and patch on that issue with
a (fairly large) grain of salt. ;)

This was introduced by Travis in r5138 as part of the matrix changes.

Alan's change looks a bit iffy to me because the looser check would pass things like matrices and there would be no check for decreasing dimensions to throw an error. So I think the safest thing for 1.1 is to back out Travis' change and rethink this for 1.2.

Chuck

Chuck




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Charles R Harris


On Wed, May 21, 2008 at 3:10 PM, Charles R Harris <[hidden email]> wrote:


On Wed, May 21, 2008 at 2:39 PM, Charles R Harris <[hidden email]> wrote:


On Wed, May 21, 2008 at 10:07 AM, Alan McIntyre <[hidden email]> wrote:
On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]> wrote:
> On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
>> There's some commentary and a patch on NumPy ticket 793 on this issue:
>>
>> http://scipy.org/scipy/numpy/ticket/793
>
> OK, thanks a lot ! That's a C problem, then...

It's probably worth mentioning that I'm not that familiar with all the
innards of NumPy yet, so take my comments and patch on that issue with
a (fairly large) grain of salt. ;)

This was introduced by Travis in r5138 as part of the matrix changes.

Alan's change looks a bit iffy to me because the looser check would pass things like matrices and there would be no check for decreasing dimensions to throw an error. So I think the safest thing for 1.1 is to back out Travis' change and rethink this for 1.2.

Chuck

I backed out r5138 and, with a few test fixes, everything passes. However, there is now a warning exposed in the masked array tests:

/usr/lib/python2.5/site-packages/numpy/ma/core.py:1357: UserWarning: MaskedArray.__setitem__ on fields: The mask is NOT affected!
  warnings.warn("MaskedArray.__setitem__ on fields: "

Pierre?

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Pierre GM-2
On Wednesday 21 May 2008 17:57:30 Charles R Harris wrote:

> On Wed, May 21, 2008 at 3:10 PM, Charles R Harris
> <[hidden email]>
>
> wrote:
> > On Wed, May 21, 2008 at 2:39 PM, Charles R Harris <
> >
> > [hidden email]> wrote:
> >> On Wed, May 21, 2008 at 10:07 AM, Alan McIntyre
> >> <[hidden email]>
> >>
> >> wrote:
> >>> On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]>
> >>>
> >>> wrote:
> >>> > On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
> >>> >> There's some commentary and a patch on NumPy ticket 793 on this
> >>> >> issue:
> >>> >>
> >>> >> http://scipy.org/scipy/numpy/ticket/793
> >>> >
> >>> > OK, thanks a lot ! That's a C problem, then...
> >>>
> >>> It's probably worth mentioning that I'm not that familiar with all the
> >>> innards of NumPy yet, so take my comments and patch on that issue with
> >>> a (fairly large) grain of salt. ;)
> >>
> >> This was introduced by Travis in r5138 as part of the matrix changes.
> >
> > Alan's change looks a bit iffy to me because the looser check would pass
> > things like matrices and there would be no check for decreasing
> > dimensions to throw an error. So I think the safest thing for 1.1 is to
> > back out Travis' change and rethink this for 1.2.
> >
> > Chuck
>
> I backed out r5138 and, with a few test fixes, everything passes. However,
> there is now a warning exposed in the masked array tests:
>
> /usr/lib/python2.5/site-packages/numpy/ma/core.py:1357: UserWarning:
> MaskedArray.__setitem__ on fields: The mask is NOT affected!
>   warnings.warn("MaskedArray.__setitem__ on fields: "
>
> Pierre?

Well, that's a warning all right, not an error.

With "exotic" dtypes (understand, not int,bool,float or complex but named
fields), setting a field doesn't affect the mask, hence the warning.

The reason behind this behavior is that the mask of a MaskedArray is only a
boolean-array: you have (at most) one boolean per element/record. Therefore,  
you can mask/unmask a full record, but not a specific field.  Masking
particular fields is possible with MaskedRecords, however, but with an
overhead that wasn't worth putting in MaskedArray.
Because I've been bitten a couple of times by this mechanism, I figured that a
warning would be the easiest way to remember that MaskedArrays don't handle
records very well.

So, nothing to worry about.

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Charles R Harris


On Wed, May 21, 2008 at 5:56 PM, Pierre GM <[hidden email]> wrote:
On Wednesday 21 May 2008 17:57:30 Charles R Harris wrote:
> On Wed, May 21, 2008 at 3:10 PM, Charles R Harris
> <[hidden email]>
>
> wrote:
> > On Wed, May 21, 2008 at 2:39 PM, Charles R Harris <
> >
> > [hidden email]> wrote:
> >> On Wed, May 21, 2008 at 10:07 AM, Alan McIntyre
> >> <[hidden email]>
> >>
> >> wrote:
> >>> On Wed, May 21, 2008 at 11:56 AM, Pierre GM <[hidden email]>
> >>>
> >>> wrote:
> >>> > On Wednesday 21 May 2008 11:39:32 Alan McIntyre wrote:
> >>> >> There's some commentary and a patch on NumPy ticket 793 on this
> >>> >> issue:
> >>> >>
> >>> >> http://scipy.org/scipy/numpy/ticket/793
> >>> >
> >>> > OK, thanks a lot ! That's a C problem, then...
> >>>
> >>> It's probably worth mentioning that I'm not that familiar with all the
> >>> innards of NumPy yet, so take my comments and patch on that issue with
> >>> a (fairly large) grain of salt. ;)
> >>
> >> This was introduced by Travis in r5138 as part of the matrix changes.
> >
> > Alan's change looks a bit iffy to me because the looser check would pass
> > things like matrices and there would be no check for decreasing
> > dimensions to throw an error. So I think the safest thing for 1.1 is to
> > back out Travis' change and rethink this for 1.2.
> >
> > Chuck
>
> I backed out r5138 and, with a few test fixes, everything passes. However,
> there is now a warning exposed in the masked array tests:
>
> /usr/lib/python2.5/site-packages/numpy/ma/core.py:1357: UserWarning:
> MaskedArray.__setitem__ on fields: The mask is NOT affected!
>   warnings.warn("MaskedArray.__setitem__ on fields: "
>
> Pierre?

Well, that's a warning all right, not an error.

With "exotic" dtypes (understand, not int,bool,float or complex but named
fields), setting a field doesn't affect the mask, hence the warning.

The reason behind this behavior is that the mask of a MaskedArray is only a
boolean-array: you have (at most) one boolean per element/record. Therefore,
you can mask/unmask a full record, but not a specific field.  Masking
particular fields is possible with MaskedRecords, however, but with an
overhead that wasn't worth putting in MaskedArray.
Because I've been bitten a couple of times by this mechanism, I figured that a
warning would be the easiest way to remember that MaskedArrays don't handle
records very well.

So, nothing to worry about.

Should we disable the warning for the tests? It's a bit unnerving and likely to generate mail calling attention to it.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Pierre GM-2

> Should we disable the warning for the tests? It's a bit unnerving and
> likely to generate mail calling attention to it.

I don't have a problem with that. Is there some kind of trapping mechanism we
can use ? A la "fail_unless_raise" ?
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Robert Kern-2
In reply to this post by Charles R Harris
On Wed, May 21, 2008 at 7:12 PM, Charles R Harris
<[hidden email]> wrote:
> Should we disable the warning for the tests? It's a bit unnerving and likely
> to generate mail calling attention to it.

Yes. Known warnings should be explicitly silenced in tests.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Jarrod Millman
In reply to this post by Charles R Harris
On Wed, May 21, 2008 at 5:12 PM, Charles R Harris
<[hidden email]> wrote:
> Should we disable the warning for the tests? It's a bit unnerving and likely
> to generate mail calling attention to it.

+1
I tend to agree that it is disconcerting to have warnings pop up in the tests.
--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: 1.1.0rc1 RuntimeErrors

Robert Kern-2
In reply to this post by Pierre GM-2
On Wed, May 21, 2008 at 7:23 PM, Pierre GM <[hidden email]> wrote:
>
>> Should we disable the warning for the tests? It's a bit unnerving and
>> likely to generate mail calling attention to it.
>
> I don't have a problem with that. Is there some kind of trapping mechanism we
> can use ?

You can filter warnings using warning.filterwarnings() or
warning.simplefilter().

  http://docs.python.org/dev/library/warnings

> A la "fail_unless_raise" ?

I'm not sure I see the connection. Do you need to test that the
warning is issued? If so, then make a filter with the action 'error'
and use self.assertRaises().

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion