Problem with absolute value

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with absolute value

Nicolas P. Rougier

Hi,

I’m a bit surprised with the following code:

>>> import numpy as np
>>> np.seterr(all='warn')
>>> Z = np.array([-128], dtype=np.byte)
>>> print(np.abs(Z))
[-128]

Obviously, it does not return the absolute value and I get no warning.
Is it something expected ? (numpy 1.16.4, python 3.7.3)


Nicolas
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with absolute value

Hameer Abbasi
Hi,

It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well.

TL;DR: It's an error with how the integer is stored in memory and how you're running out of space.

Regards,
Hameer Abbasi

On 03.07.19, 17:03, "NumPy-Discussion on behalf of Nicolas Rougier" <numpy-discussion-bounces+einstein.edison=[hidden email] on behalf of [hidden email]> wrote:

   
    Hi,
   
    I’m a bit surprised with the following code:
   
    >>> import numpy as np
    >>> np.seterr(all='warn')
    >>> Z = np.array([-128], dtype=np.byte)
    >>> print(np.abs(Z))
    [-128]
   
    Obviously, it does not return the absolute value and I get no warning.
    Is it something expected ? (numpy 1.16.4, python 3.7.3)
   
   
    Nicolas
    _______________________________________________
    NumPy-Discussion mailing list
    [hidden email]
    https://mail.python.org/mailman/listinfo/numpy-discussion
   
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with absolute value

Charles R Harris


On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi <[hidden email]> wrote:
Hi,

It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well.

TL;DR: It's an error with how the integer is stored in memory and how you're running out of space.

Regards,
Hameer Abbasi

More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype.

<snip>

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with absolute value

Matthew Brett
Hi,

On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris
<[hidden email]> wrote:

>
>
>
> On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi <[hidden email]> wrote:
>>
>> Hi,
>>
>> It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well.
>>
>> TL;DR: It's an error with how the integer is stored in memory and how you're running out of space.
>>
>> Regards,
>> Hameer Abbasi
>
>
> More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype.

I still think this is a major wart in numpy's abs.

I'd really like to add a new function, `uabs` which would return an
unsigned int for integer inputs.  I'm happy to do a pull request if
that also seems sensible to y'all,

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with absolute value

Charles R Harris


On Wed, Jul 3, 2019 at 11:39 AM Matthew Brett <[hidden email]> wrote:
Hi,

On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris
<[hidden email]> wrote:
>
>
>
> On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi <[hidden email]> wrote:
>>
>> Hi,
>>
>> It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well.
>>
>> TL;DR: It's an error with how the integer is stored in memory and how you're running out of space.
>>
>> Regards,
>> Hameer Abbasi
>
>
> More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype.

I still think this is a major wart in numpy's abs.

I'd really like to add a new function, `uabs` which would return an
unsigned int for integer inputs.  I'm happy to do a pull request if
that also seems sensible to y'all,


Seems reasonable, it would provide something for discussion. Note that the current abs is in the slot provided by Python numeric types. The main problem is that as soon as the unsigned result is combined with a signed type it will get promoted. Which would not be much of a problem except int64 -> double.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with absolute value

Nicolas P. Rougier

Thanks for the quick explanation & answers. I was mostly puzzled by the absence of warning.
In C you would get an overflow warning in such a case, no ?


Nicolas

> On 3 Jul 2019, at 20:01, Charles R Harris <[hidden email]> wrote:
>
>
>
> On Wed, Jul 3, 2019 at 11:39 AM Matthew Brett <[hidden email]> wrote:
> Hi,
>
> On Wed, Jul 3, 2019 at 9:08 AM Charles R Harris
> <[hidden email]> wrote:
> >
> >
> >
> > On Wed, Jul 3, 2019 at 9:08 AM Hameer Abbasi <[hidden email]> wrote:
> >>
> >> Hi,
> >>
> >> It turns out you're running into a bit-error. In general, the two's complement of -2 ** (n-1) with the bit-length being limited to n bits is itself... No way around that. And integers don't set hardware exceptions so checking for errors like these is hard as well.
> >>
> >> TL;DR: It's an error with how the integer is stored in memory and how you're running out of space.
> >>
> >> Regards,
> >> Hameer Abbasi
> >
> >
> > More like the eight bit twos complement of -128 is -128, bytes cannot represent 128. Matlab used to (still does?) solve this problem by returning 127 instead :) Basically, the data needs more precision. Returning an unsigned type would lead to it's own problems with unexpected promotions when the result was used. We could, I suppose, raise a warning, although that might be considered noisy. If you use `absolute` instead, you can specify the dtype.
>
> I still think this is a major wart in numpy's abs.
>
> I'd really like to add a new function, `uabs` which would return an
> unsigned int for integer inputs.  I'm happy to do a pull request if
> that also seems sensible to y'all,
>
>
> Seems reasonable, it would provide something for discussion. Note that the current abs is in the slot provided by Python numeric types. The main problem is that as soon as the unsigned result is combined with a signed type it will get promoted. Which would not be much of a problem except int64 -> double.
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion