

I have a PR up for review that adds "maximum difference" to the error messages produced by NumPy's testing functions for comparing arrays: https://github.com/numpy/numpy/pull/12591
Because this changes NumPy's public interface, I'm running it by the mailing list to see if there are any objections or suggestions for improvement. Example behavior:
>>> x = np.array([1, 2, 3]) >>> y = np.array([1, 2, 3.0001])
>>> np.testing.assert_allclose(x, y) AssertionError: Not equal to tolerance rtol=1e07, atol=0
(mismatch 33.333333333333336%, maximum difference 0.00010000000000021103) x: array([1, 2, 3]) y: array([1. , 2. , 3.0001])
Motivation: when writing numerical algorithms, I frequently find myself experimenting to pick the right value of atol and rtol for np.testing.assert_allclose(). If I make the tolerance too generous, I risk missing regressions in accuracy, so I usually try to pick the smallest values for which tests pass. This change immediately reveals appropriate values to use for these parameters, so I don't need to guess and check.
One alternative would be print both "atol" and "rtol" numbers directly that could be immediately used in assert_allclose(). Here we effectively only have "atol". The main reason why I didn't do this is that I felt it would add more clutter (rtol is slightly redundant with the displayed array values). But we could probably do this if we're willing to split the error message onto a few more lines, e.g.,
AssertionError: Not equal to tolerance rtol=1e07, atol=0 (mismatch 33.333333333333336%, maximum absolute difference 0.00010000000000021103, maximum relative difference 3.333333333340368e05) x: array([1, 2, 3]) y: array([1. , 2. , 3.0001])
Best, Stephan
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion


On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
> Example behavior:
>
> >>> x = np.array([1, 2, 3])
> >>> y = np.array([1, 2, 3.0001])
> >>> np.testing.assert_allclose(x, y)
> AssertionError:
> Not equal to tolerance rtol=1e07, atol=0
>
> (mismatch 33.333333333333336%, maximum difference
> 0.00010000000000021103)
This is a helpful addition; thank you! I don't have a strong preference
around whether to also include rtol.
Stéfan
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion


> On 19. Dec 2018, at 19:50, Stefan van der Walt < [hidden email]> wrote:
>
> On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
>> Example behavior:
>>
>>>>> x = np.array([1, 2, 3])
>>>>> y = np.array([1, 2, 3.0001])
>>>>> np.testing.assert_allclose(x, y)
>> AssertionError:
>> Not equal to tolerance rtol=1e07, atol=0
>>
>> (mismatch 33.333333333333336%, maximum difference
>> 0.00010000000000021103)
>
> This is a helpful addition; thank you! I don't have a strong preference
> around whether to also include rtol.
>
> Stéfan
> _______________________________________________
> NumPyDiscussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpydiscussion+1 to this addition.
And +1 to also print rtol.
I frequently use rtol with assert_allclose, and having the value printed helps quickly choose an appropriate value.
Christoph
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion


Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan!
On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:
>
>
> > On 19. Dec 2018, at 19:50, Stefan van der Walt < [hidden email]> wrote:
> >
> > On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
> >> Example behavior:
> >>
> >>>>> x = np.array([1, 2, 3])
> >>>>> y = np.array([1, 2, 3.0001])
> >>>>> np.testing.assert_allclose(x, y)
> >> AssertionError:
> >> Not equal to tolerance rtol=1e07, atol=0
> >>
> >> (mismatch 33.333333333333336%, maximum difference
> >> 0.00010000000000021103)
> >
> > This is a helpful addition; thank you! I don't have a strong preference
> > around whether to also include rtol.
> >
> > Stéfan
> > _______________________________________________
> > NumPyDiscussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpydiscussion>
> +1 to this addition.
>
> And +1 to also print rtol.
> I frequently use rtol with assert_allclose, and having the value
> printed helps quickly choose an appropriate value.
>
> Christoph
> _______________________________________________
> NumPyDiscussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpydiscussion>
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion


OK, it sounds like a popular change :).
I'll add relative error to the message as well. Please comment on the pull request if you have strong feelings about exactly what it should look like. On Wed, Dec 19, 2018 at 4:41 PM Juan NunezIglesias < [hidden email]> wrote: Another +1 on printing rtol, and +100 (can I do that?) on the overall idea! Thanks Stephan!
On Thu, Dec 20, 2018, at 7:27 AM, Christoph Deil wrote:
>
>
> > On 19. Dec 2018, at 19:50, Stefan van der Walt <[hidden email]> wrote:
> >
> > On Wed, 19 Dec 2018 09:47:01 0800, Stephan Hoyer wrote:
> >> Example behavior:
> >>
> >>>>> x = np.array([1, 2, 3])
> >>>>> y = np.array([1, 2, 3.0001])
> >>>>> np.testing.assert_allclose(x, y)
> >> AssertionError:
> >> Not equal to tolerance rtol=1e07, atol=0
> >>
> >> (mismatch 33.333333333333336%, maximum difference
> >> 0.00010000000000021103)
> >
> > This is a helpful addition; thank you! I don't have a strong preference
> > around whether to also include rtol.
> >
> > Stéfan
> > _______________________________________________
> > NumPyDiscussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpydiscussion
>
> +1 to this addition.
>
> And +1 to also print rtol.
> I frequently use rtol with assert_allclose, and having the value
> printed helps quickly choose an appropriate value.
>
> Christoph
> _______________________________________________
> NumPyDiscussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpydiscussion
>
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion
_______________________________________________
NumPyDiscussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpydiscussion

