Adding "maximum difference" to np.testing.assert_array_equal errors

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Adding "maximum difference" to np.testing.assert_array_equal errors

Stephan Hoyer-2
I have a PR up for review that adds "maximum difference" to the error messages produced by NumPy's testing functions for comparing arrays:
https://github.com/numpy/numpy/pull/12591

Because this changes NumPy's public interface, I'm running it by the mailing list to see if there are any objections or suggestions for improvement.

Example behavior:

>>> x = np.array([1, 2, 3])
>>> y = np.array([1, 2, 3.0001]) 
>>> np.testing.assert_allclose(x, y) 
AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103)
 x: array([1, 2, 3])
 y: array([1. , 2. , 3.0001])

Motivation: when writing numerical algorithms, I frequently find myself experimenting to pick the right value of atol and rtol for np.testing.assert_allclose(). If I make the tolerance too generous, I risk missing regressions in accuracy, so I usually try to pick the smallest values for which tests pass. This change immediately reveals appropriate values to use for these parameters, so I don't need to guess and check.

One alternative would be print both "atol" and "rtol" numbers directly that could be immediately used in assert_allclose(). Here we effectively only have "atol". The main reason why I didn't do this is that I felt it would add more clutter (rtol is slightly redundant with the displayed array values). But we could probably do this if we're willing to split the error message onto a few more lines, e.g.,

AssertionError:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 33.333333333333336%,
 maximum absolute difference 0.00010000000000021103,
 maximum relative difference 3.333333333340368e-05)
 x: array([1, 2, 3])
 y: array([1. , 2. , 3.0001]) 

Best,
Stephan

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Adding "maximum difference" to np.testing.assert_array_equal errors

Stefan van der Walt
On Wed, 19 Dec 2018 09:47:01 -0800, Stephan Hoyer wrote:

> Example behavior:
>
> >>> x = np.array([1, 2, 3])
> >>> y = np.array([1, 2, 3.0001])
> >>> np.testing.assert_allclose(x, y)
> AssertionError:
> Not equal to tolerance rtol=1e-07, atol=0
>
> (mismatch 33.333333333333336%, maximum difference
> 0.00010000000000021103)

This is a helpful addition; thank you!  I don't have a strong preference
around whether to also include rtol.

Stéfan
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Adding "maximum difference" to np.testing.assert_array_equal errors

Christoph Deil


> On 19. Dec 2018, at 19:50, Stefan van der Walt <[hidden email]> wrote:
>
> On Wed, 19 Dec 2018 09:47:01 -0800, Stephan Hoyer wrote:
>> Example behavior:
>>
>>>>> x = np.array([1, 2, 3])
>>>>> y = np.array([1, 2, 3.0001])
>>>>> np.testing.assert_allclose(x, y)
>> AssertionError:
>> Not equal to tolerance rtol=1e-07, atol=0
>>
>> (mismatch 33.333333333333336%, maximum difference
>> 0.00010000000000021103)
>
> This is a helpful addition; thank you!  I don't have a strong preference
> around whether to also include rtol.
>
> Stéfan
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

+1 to this addition.

And +1 to also print rtol.
I frequently use rtol with assert_allclose, and having the value printed helps quickly choose an appropriate value.

Christoph
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Adding "maximum difference" to np.testing.assert_array_equal errors

Juan Nunez-Iglesias
Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan!

On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:

>
>
> > On 19. Dec 2018, at 19:50, Stefan van der Walt <[hidden email]> wrote:
> >
> > On Wed, 19 Dec 2018 09:47:01 -0800, Stephan Hoyer wrote:
> >> Example behavior:
> >>
> >>>>> x = np.array([1, 2, 3])
> >>>>> y = np.array([1, 2, 3.0001])
> >>>>> np.testing.assert_allclose(x, y)
> >> AssertionError:
> >> Not equal to tolerance rtol=1e-07, atol=0
> >>
> >> (mismatch 33.333333333333336%, maximum difference
> >> 0.00010000000000021103)
> >
> > This is a helpful addition; thank you!  I don't have a strong preference
> > around whether to also include rtol.
> >
> > Stéfan
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> +1 to this addition.
>
> And +1 to also print rtol.
> I frequently use rtol with assert_allclose, and having the value
> printed helps quickly choose an appropriate value.
>
> Christoph
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Adding "maximum difference" to np.testing.assert_array_equal errors

Stephan Hoyer-2
OK, it sounds like a popular change :).

I'll add relative error to the message as well. Please comment on the pull request if you have strong feelings about exactly what it should look like.

On Wed, Dec 19, 2018 at 4:41 PM Juan Nunez-Iglesias <[hidden email]> wrote:
Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan!

On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:
>
>
> > On 19. Dec 2018, at 19:50, Stefan van der Walt <[hidden email]> wrote:
> >
> > On Wed, 19 Dec 2018 09:47:01 -0800, Stephan Hoyer wrote:
> >> Example behavior:
> >>
> >>>>> x = np.array([1, 2, 3])
> >>>>> y = np.array([1, 2, 3.0001])
> >>>>> np.testing.assert_allclose(x, y)
> >> AssertionError:
> >> Not equal to tolerance rtol=1e-07, atol=0
> >>
> >> (mismatch 33.333333333333336%, maximum difference
> >> 0.00010000000000021103)
> >
> > This is a helpful addition; thank you!  I don't have a strong preference
> > around whether to also include rtol.
> >
> > Stéfan
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
>
> +1 to this addition.
>
> And +1 to also print rtol.
> I frequently use rtol with assert_allclose, and having the value
> printed helps quickly choose an appropriate value.
>
> Christoph
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion