Release vs development testing.

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Release vs development testing.

Charles R Harris
Hi All,

NumPy currently distinguishes between release and development versions when running tests. Is there a good reason to continue this practice? I ask, because with the last pytest release it would be convenient to always include `pytest.ini ` so that we can register markers. The presence of `pytest.ini` is how we distinguish betweendevelopment from release for testing purposes.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Julian Taylor-3
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Charles R Harris


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.
If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

ralfgommers


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,

Ralf


Chuck
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Charles R Harris


On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,


It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.

Chuck 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

ralfgommers


On Sun, May 12, 2019 at 4:41 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,


It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.

Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?

It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.

Cheers,
Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Charles R Harris


On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 4:41 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,


It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.

Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?

It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.


I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest.  However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. Note that we currently deal with the missing `pytest.ini` in wheels by duplicating the warnings filters in `_pytesttester.py` using the pytest command line.

Adding the `error` filter in the command line worries me, as I'm not sure how the priorities work out, the command line is now appended to the contents of `pytest.ini`.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Charles R Harris


On Mon, May 13, 2019 at 11:25 AM Charles R Harris <[hidden email]> wrote:


On Mon, May 13, 2019 at 3:02 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 4:41 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 7:27 AM Ralf Gommers <[hidden email]> wrote:


On Sun, May 12, 2019 at 3:56 PM Charles R Harris <[hidden email]> wrote:


On Sun, May 12, 2019 at 6:33 AM Julian Taylor <[hidden email]> wrote:
On 12.05.19 14:58, Charles R Harris wrote:
> Hi All,
>
> NumPy currently distinguishes between release and development versions
> when running tests. Is there a good reason to continue this practice? I
> ask, because with the last pytest release it would be convenient to
> always include `pytest.ini ` so that we can register markers. The
> presence of `pytest.ini` is how we distinguish betweendevelopment from
> release for testing purposes.
>

One difference between development and release builds was that in
development releases numpy.testing throws errors on floating point
exceptions while the release version it did not.

I'd prefer to keep this behavior. It's not clear to me if the proposal is to change the behavior or not.

If that is still the case removing the distinction could require a lot
of changes in upstream test suites that are not regularly run against
development builds.

The motivation is not quite clear to me, can you please elaborate on
what you want to do.

NumPy pytest testing is NumPy specific and not used downstream like our nose testing framework was, so I don't see why that should affect other projects. What motivates this question is that the new version of pytest released yesterday raises warnings for non-registered markers, `pytest.mark.slow` in particular, and that was causing CI failures. The easiest way to register a mark is using `pytest.ini`, but we currently don't include that in released wheels, only in source releases.

Adding a pytest.ini file in wheels should be perfectly fine I think,


It is the absence of pytest.ini that makes it a release, for that is the file that turns warnings into errors.

Why don't we just always keep pytest.ini, and move the settings for warnings-to-errors to runtests.py or tools/travis_test.sh?

It's not important how we check, all we need is some mechanism to prevent new warnings creeping in. Same as we set -Wall for building in CI.


I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest.  However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. Note that we currently deal with the missing `pytest.ini` in wheels by duplicating the warnings filters in `_pytesttester.py` using the pytest command line.

Adding the `error` filter in the command line worries me, as I'm not sure how the priorities work out, the command line is now appended to the contents of `pytest.ini`.


Note that this is also a problem when numpy is installed with `setup.py install`. Things work now for runtests because `pytest.ini` is in the directory from which the tests are run.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Nathaniel Smith
In reply to this post by Charles R Harris
On Mon, May 13, 2019, 10:26 Charles R Harris <[hidden email]> wrote:
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest.  However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. 

I bet if you open an issue on pytest explaining that numpy needs to either use unregistered marks or else have some programmatic non-pytest.ini-based way to register marks, then they'll figure something out. I think they added a warning because they're hoping to flush out these kinds of problems so they can fix them before they do the cleanup.

-n

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Release vs development testing.

Charles R Harris


On Mon, May 13, 2019 at 11:50 AM Nathaniel Smith <[hidden email]> wrote:
On Mon, May 13, 2019, 10:26 Charles R Harris <[hidden email]> wrote:
I just checked that current wheels show the warnings when `numpy.test()` is run with latest pytest.  However, moving the `pytest.ini` file into the `numpy` directory is tricky, as we need to tell pytest where to find the installed file (-c option). The simplest short time solution is to ignore the warning, but long term I'm worried that the warning will become an error as pytest is doing this because they want to clean up the their implementation. Ideally there would be a better way to register the marks. 

I bet if you open an issue on pytest explaining that numpy needs to either use unregistered marks or else have some programmatic non-pytest.ini-based way to register marks, then they'll figure something out. I think they added a warning because they're hoping to flush out these kinds of problems so they can fix them before they do the cleanup.


Turns out that markers can be registered in conftest.py, so that fixes the problem for NumPy. SciPy will need this fix also.

Chuck 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion