PyPy wheels (was: NumPy 1.12.0b1 released)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

PyPy wheels (was: NumPy 1.12.0b1 released)

ralfgommers


On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <[hidden email]> wrote:

Another thing to think about is that 1.12 on pypy won't pass its test suite (though it's close), and we're not yet testing new PRs on pypy, so no guarantees about 1.13 yet. I think on balance these probably aren't reasons *not* to upload wheels, but it's a funny place where we're talking about providing "official" builds even though it's not an "officially supported platform". So we will at least want to be clear about that. And someone will have to handle the bug reports about the test suite failing :-).


Those are good points. We could run PyPy on TravisCI; the PyPy install and numpy build aren't difficult anymore.

Handling bug reports is mostly checking if it's PyPy specific, and if so refer to the PyPy tracker I'd think. It is some work, but given that PyPy has finally chosen a way to support Numpy that's not a dead end and has come quite a long way quite quickly, taking on that bit of extra work as Numpy maintainers is a good time investment imho.

Many bug reports will go straight to PyPy though I expect, because often that is the obvious place to go. This is what I just got from downloading the OS X PyPy binary and pip installing numpy master:

Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
nose version 1.3.7
.................................S.......................................................E........................................................................................S.............................................................RPython traceback:
  File "pypy_interpreter.c", line 43348, in BuiltinCodePassThroughArguments1_funcrun_obj
  File "pypy_module_cpyext_4.c", line 16627, in generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17
Fatal RPython error: AssertionError
Abort trap: 6
 
So guess I'll go find out where that issue tracker is:)

Ralf


On Nov 18, 2016 01:14, "Ralf Gommers" <[hidden email]> wrote:
>
>
>
> On Fri, Nov 18, 2016 at 9:08 PM, Matthew Brett <[hidden email]> wrote:
>>
>> Hi,
>>
>> On Thu, Nov 17, 2016 at 3:24 PM, Matti Picus <[hidden email]> wrote:
>> > Congrats to all on the release.Two questions:
>> >
>> > Is there a guide to building standard wheels for NumPy?
>>
>> I don't think so - there is a repository that we use to build the
>> wheels, that has the Windows, OSX and manyllinux recipes for the
>> standard CPython build:
>>
>> https://github.com/MacPython/numpy-wheelso
>>
>> If you can work out a way to automate the PyPy builds and tests -
>> especially using the same repo - that would be very useful.
>>
>> > Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32 and
>> > OSX64, how can I get them blessed and uploaded to PyPI?
>>
>> If you can automate the build and tests, I'm guessing there will be no
>> objections - but it's not my call...
>
>
> I'm in favor, assuming that the wheel tags and PyPy backwards compatibility situation is OK. Can't really find any examples. What I mean is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels should have tags "pp<something>". Are the PyPy cpyext layer and the <something> defined such that a new PyPy release won't break older wheels?



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PyPy wheels (was: NumPy 1.12.0b1 released)

Nathaniel Smith

On Nov 18, 2016 3:30 PM, "Ralf Gommers" <[hidden email]> wrote:
>
>
>
> On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <[hidden email]> wrote:
>>
>> Another thing to think about is that 1.12 on pypy won't pass its test suite (though it's close), and we're not yet testing new PRs on pypy, so no guarantees about 1.13 yet. I think on balance these probably aren't reasons *not* to upload wheels, but it's a funny place where we're talking about providing "official" builds even though it's not an "officially supported platform". So we will at least want to be clear about that. And someone will have to handle the bug reports about the test suite failing :-).
>
>
> Those are good points. We could run PyPy on TravisCI; the PyPy install and numpy build aren't difficult anymore.

I'm not sure how useful this until the test suite is passing, though, just because of how travis-ci works.

The main outstanding issue needs some work on the numpy side: basically UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy, because it assumes that Python objects get deterministically deallocated as soon as their reference count drops to zero, and pypy doesn't have reference counts.

I think fixing this should be straightforward enough, in case anyone wants to work on it. Every user of UPDATEIFCOPY already has to be aware of the reference counts and know when the pseudo-"view" array is supposed to be deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which acts like the current UPDATEIFCOPY, except that instead of using __del__ to do the writeback, there's an explicit API call you have to make, like PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the writeback if set. Then we should transition to using this internally, and probably deprecate UPDATEIFCOPY (though we may never be able to get rid of it entirely).

-n


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: PyPy wheels (was: NumPy 1.12.0b1 released)

ralfgommers


On Sat, Nov 19, 2016 at 2:53 PM, Nathaniel Smith <[hidden email]> wrote:

On Nov 18, 2016 3:30 PM, "Ralf Gommers" <[hidden email]> wrote:
>
>
>
> On Sat, Nov 19, 2016 at 5:24 AM, Nathaniel Smith <[hidden email]> wrote:
>>
>> Another thing to think about is that 1.12 on pypy won't pass its test suite (though it's close), and we're not yet testing new PRs on pypy, so no guarantees about 1.13 yet. I think on balance these probably aren't reasons *not* to upload wheels, but it's a funny place where we're talking about providing "official" builds even though it's not an "officially supported platform". So we will at least want to be clear about that. And someone will have to handle the bug reports about the test suite failing :-).
>
>
> Those are good points. We could run PyPy on TravisCI; the PyPy install and numpy build aren't difficult anymore.

I'm not sure how useful this until the test suite is passing, though, just because of how travis-ci works.

It's not hard to skip the currently failing tests only on a single run in the build matrix. That would keep TravisCI green and ensure there's no regressions.

Ralf

 

The main outstanding issue needs some work on the numpy side: basically UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy, because it assumes that Python objects get deterministically deallocated as soon as their reference count drops to zero, and pypy doesn't have reference counts.

I think fixing this should be straightforward enough, in case anyone wants to work on it. Every user of UPDATEIFCOPY already has to be aware of the reference counts and know when the pseudo-"view" array is supposed to be deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which acts like the current UPDATEIFCOPY, except that instead of using __del__ to do the writeback, there's an explicit API call you have to make, like PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the writeback if set. Then we should transition to using this internally, and probably deprecate UPDATEIFCOPY (though we may never be able to get rid of it entirely).

-n


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Loading...