pytest, fixture and parametrize

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

pytest, fixture and parametrize

Charles R Harris
Hi All,

I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality.

Thoughts?

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

einstein.edison
> On 8. Aug 2018, at 17:23, Charles R Harris <[hidden email]> wrote:
>
> Hi All,
>
> I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality.
>
> Thoughts?
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

Hi all!

I think it mainly depends on how likely we are to move away from Pytest. And I’m assuming not very likely. Anything likely to replace it (which won’t happen short-term in any case) will already have something similar, IMO.

So my vote is we go ahead and use them.

Hameer Abbasi
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

ralfgommers


On Wed, Aug 8, 2018 at 8:27 AM, Hameer Abbasi <[hidden email]> wrote:
> On 8. Aug 2018, at 17:23, Charles R Harris <[hidden email]> wrote:
>
> Hi All,
>
> I'm writing some tests and would like to come to a common understanding as to whether we are should use pytest fixtures and parametrized classes and functions. The immediate problem is testing over all types that should be supported, which something we should to much more of and is most easily achieved using one of those pytest tools. The advantage is code clarity and more informative error messages, the disadvantage is being dependent on pytest as unittest does not support that functionality.
>
> Thoughts?
>
> Chuck
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

Hi all!

I think it mainly depends on how likely we are to move away from Pytest. And I’m assuming not very likely. Anything likely to replace it (which won’t happen short-term in any case) will already have something similar, IMO.

So my vote is we go ahead and use them.

I agree. There's no other viable test framework at the moment, so spending effort reinventing pytest features is a waste of time right now.

Cheers,
Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Chris Barker - NOAA Federal
In reply to this post by Charles R Harris
> as to whether we are should use pytest fixtures and parametrized classes and functions.

Absolutely!

 > the disadvantage is being dependent on pytest as unittest does not
support that functionality.

Which is the whole point of using pytest, yes?

I’m very opinionated about this, but I really dislike unittest — it’s
simply way too Java-y — makes the easy things harder than they should
be, and is missing critical should-be-easy features.

I moved to pure pytest a few years ago, and have been very happy about
it.  In fact, I recently converted some d unittest code to pure
pytest, and it was literally about 1/4 as much code.

The only reason I can see to avoid pytest features is that it’s not in
the standard lib — but it’s not a run-time dependency, and it’s easy
to install and well supported — so all good.

I suppose we may want to avoid some of pytest’s more magical esoteric
features, but certainly not core functionality like fixtures and
parameterized tests.

-CHB
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Chris Barker - NOAA Federal
In reply to this post by ralfgommers
BTW:

with pytest.raises(AnException):
    ....

Is another nice feature.

-CHB
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Tyler Reddy
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on

+0.5 for the fancy fixture stuff in pytest (have to concede that this stuff can be really hard to reason about sometimes, 
especially with custom scoping decorators, but if it helps to mock something like recently-discovered network dependency
for some unit tests it may be better than adding a new dependency or reimplementing effectively the same thing)

On Wed, 8 Aug 2018 at 09:09, Chris Barker - NOAA Federal <[hidden email]> wrote:
BTW:

with pytest.raises(AnException):
    ....

Is another nice feature.

-CHB
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Eric Wieser
In reply to this post by Chris Barker - NOAA Federal

Is another nice feature

You forget that we already have that feature in our testing layer,

with np.testing.assert_raises(AnException):
    pass

On Wed, 8 Aug 2018 at 09:08 Chris Barker - NOAA Federal <[hidden email]> wrote:
BTW:

with pytest.raises(AnException):
    ....

Is another nice feature.

-CHB
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Evgeni Burovski
This is actually available from both nose (old-style np.testing framework), and pytest. Stdlib unittest supports self.assertRaises context manager from python 3.1

On Wed, Aug 8, 2018, 7:30 PM Eric Wieser <[hidden email]> wrote:

Is another nice feature

You forget that we already have that feature in our testing layer,

with np.testing.assert_raises(AnException):
    pass

On Wed, 8 Aug 2018 at 09:08 Chris Barker - NOAA Federal <[hidden email]> wrote:
BTW:

with pytest.raises(AnException):
    ....

Is another nice feature.

-CHB
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Chris Barker - NOAA Federal
On Wed, Aug 8, 2018 at 9:38 AM, Evgeni Burovski <[hidden email]> wrote:
Stdlib unittest supports self.assertRaises context manager from python 3.1

but that requires using unittest :-)

On Wed, Aug 8, 2018, 7:30 PM Eric Wieser <[hidden email]> wrote:

You forget that we already have that feature in our testing layer,

with np.testing.assert_raises(AnException):
    pass

fair enough -- I wouldn't re-write that now, but as its there already, it may make sense to use it.

Perhaps we need a doc that lays out the prefered testing utilities.

Worthy of a NEP? Or is just a README or something in the code base.

Personally, I think a commitment to pytest is the best way to go -- but there are a lot of legacy tests, so there will be a jumble.

-CHB

-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Stephan Hoyer-2
In reply to this post by Tyler Reddy
On Wed, Aug 8, 2018 at 9:25 AM Tyler Reddy <[hidden email]> wrote:
+1 for pytest parametrization and some of the more straight forward context managers like pytest.raises and so on

+1 from me, too. 
 
+0.5 for the fancy fixture stuff in pytest (have to concede that this stuff can be really hard to reason about sometimes, 
especially with custom scoping decorators, but if it helps to mock something like recently-discovered network dependency
for some unit tests it may be better than adding a new dependency or reimplementing effectively the same thing)

I'm -0 on fixtures, but unfortunately there don't seem to be viable alternatives in pytest. I find the way that pytest automatically match *argument names* with the names of fixtures very distasteful and un-Pythonic.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Juan Nunez-Iglesias
I find the way that pytest automatically match *argument names* with the names of fixtures very distasteful and un-Pythonic.

THIS! I've been wanting to articulate this for a while but couldn't quite put my finger on it. 

Nevertheless, I agree with the sentiment in this thread, specifically that, just like pytest supports unittest tests, I think any framework that comes along to replace it will have to support pytest syntax to be viable.

A NumPy document on preferred testing practices would be very valuable even to projects beyond NumPy, just like HOW_TO_DOCUMENT.txt was valuable to the entire ecosystem.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

mattip


On 08/08/18 17:34, Juan Nunez-Iglesias wrote:
> A NumPy document on preferred testing practices would be very valuable
> even to projects beyond NumPy, just like HOW_TO_DOCUMENT.txt was
> valuable to the entire ecosystem.
>
We have these guidelines
http://www.numpy.org/devdocs/reference/testing.html, or if you prefer
the source https://github.com/numpy/numpy/blob/master/doc/TESTS.rst.txt.
It was updated for pytest in the 1.15 release, but could use some more
editing and refinement.
Matti
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: pytest, fixture and parametrize

Chris Barker - NOAA Federal
> On Aug 8, 2018, at 5:40 PM, Matti Picus <[hidden email]> wrote:
> We have these guidelines http://www.numpy.org/devdocs/reference/testing.html,

Thanks Matti — that’s clearly the place to document best practices.

>  It was updated for pytest in the 1.15 release, but could use some more editing and refinement.

Giving it s quick read right now, it clearly has not embraced pytest (yet?)

Before we try to update the doc, we should have some guidelines as to the goals.

At the broad level — do we want to make use of the pytest testing
Framework, or just use pytest as a test runner?

I would say that it’s a no brainer, except that he numpy testing
framework already has a lot of the key  features of pytest that we
need.

Options:

1) Use pytest only for test running

2) Use pytest features that numpy.testing does not have a good replacement for.

3) prefer pytest features for new tests.

In all cases, I think we should limit ourselves to the less magical
pytest features.

BTW, I see this:

“Setup and teardown functions to functions and methods are known as
“fixtures”, and their use is not encouraged.”

First, is that a typo? I don’t understand what it means. Though I
think I know what a fixture is.

Second — why are fixtures discouraged?

Third, if fixtures are discouraged, rather than banned, we should say
when it is OK to use one.

-CHB

> Matti
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion