Hi,
This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this. The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes:: image = <... Image production ...> ar = numpy.asarray(image) However, when I say:: image = <... Image production ...> print("---") ar = numpy.asarray(image) the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface. The extension type permits reading the pixel values by a method; copying them over by a Python loop works fine. I am ``Py_INCREF``-ing the producer in the C++ buffer view creation function properly. The shapes and strides of the buffer view are ``delete[]``-ed upon releasing the buffer; avoiding this does not prevent the crash. I am catching ``std::exception`` in the view creation function; no such exception occurs. The shapes and strides are allocated by ``new Py_ssize_t[2]``, so they will survive the view creation function. I spent some hours trying to figure out what I am doing wrong. Maybe someone has an idea about this? I double-checked each line of code related to this problem and couldn't find any mistake. Probabaly I am not looking at the right aspect. Best, Friedrich _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hi,
Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt <[hidden email]>: > > [...] The following Python > code crashes:: > > image = <... Image production ...> > ar = numpy.asarray(image) > > However, when I say:: > > image = <... Image production ...> > print("---") > ar = numpy.asarray(image) > > the entire program is executing properly with correct data in the > numpy ndarray produced using the buffer interface. > > [...] Does anyone have an idea about this? By the way, I noticed that this mailing list turned pretty quiet, am I missing something? For completeness, the abovementioned "crash" shows up as just a premature exit of the program. There is no error message whatsoever. The buffer view producing function raises Exceptions properly when something goes wrong; also notice that this code completes without error when the ``print("---")`` statement is in action. So I presume the culprit lies somewhere on the C level. I can only guess that it might be some side-effect unknown to me. Best, Friedrich _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
In reply to this post by Friedrich Romstedt
![]() Hey Friedrich, If you can produce an MVCE that would be really helpful, along with your hardware and environment. Without that, it isn’t possible to be of much help. Best Regards, Hameer Abbasi -- Sent from Canary
_______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Administrator
|
In reply to this post by Friedrich Romstedt
On 2/1/21 9:57 AM, Friedrich Romstedt wrote: > Hi, > > Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt > <[hidden email]>: >> [...] The following Python >> code crashes:: >> >> image = <... Image production ...> >> ar = numpy.asarray(image) >> >> However, when I say:: >> >> image = <... Image production ...> >> print("---") >> ar = numpy.asarray(image) >> >> the entire program is executing properly with correct data in the >> numpy ndarray produced using the buffer interface. >> >> [...] > Does anyone have an idea about this? By the way, I noticed that this > mailing list turned pretty quiet, am I missing something? > > For completeness, the abovementioned "crash" shows up as just a > premature exit of the program. There is no error message whatsoever. > The buffer view producing function raises Exceptions properly when > something goes wrong; also notice that this code completes without > error when the ``print("---")`` statement is in action. So I presume > the culprit lies somewhere on the C level. I can only guess that it > might be some side-effect unknown to me. > > Best, > Friedrich It is very hard to help you from this description. It may be a refcount problem, it may be a buffer protocol problem, it may be something else. Typically, one would create a complete example and then pointing to the code (as repo or pastebin, not as an attachment to a mail here). A few things you might want to check: - Make sure you give instructions how to build your project for Linux, since most of the people on this list do not use windows. - There are tools out there to analyze refcount problems. Python has some built-in tools for switching allocation strategies. - numpy.asarray has a number of strategies to convert instances, which one is it using? Matti _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hello Matti,
Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <[hidden email]>: > > [...] > > It is very hard to help you from this description. It may be a refcount > problem, it may be a buffer protocol problem, it may be something else. Yes, indeed! > Typically, one would create a complete example and then pointing to the > code (as repo or pastebin, not as an attachment to a mail here). https://github.com/friedrichromstedt/bughunting-01 I boiled it down considerably, compared to the program where I stumbled upon the problem. In the abovementioned repo, you find a Python test script in the `test/` folder. Therein, a single `print` statement can be used to trigger or to avoid the error. On Linux, I get a somewhat more precise description than just from the premature exit on Windows: It is a segfault. Certainly it is still asked quite much to skim through my source code, however, I hope that I trimmed it down sufficiently. > - Make sure you give instructions how to build your project for Linux, > since most of the people on this list do not use windows. The code reproducing the segfault can be compiled by `$ python3 setup.py install`, both on Windows as well as on Linux. > - There are tools out there to analyze refcount problems. Python has > some built-in tools for switching allocation strategies. Can you give me some pointer about this? > - numpy.asarray has a number of strategies to convert instances, which > one is it using? I've tried to read about this, but coudn't find anything. What are these different strategies? Many thanks in advance, Friedrich _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hi,
Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt <[hidden email]>: > Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <[hidden email]>: > > Typically, one would create a complete example and then pointing to the > > code (as repo or pastebin, not as an attachment to a mail here). > > https://github.com/friedrichromstedt/bughunting-01 Last week I updated my example code to be more slim. There now exists a single-file extension module: https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp. The corresponding test program https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print`` statement contained in the test file is commented out. My hope to be able to fix my error myself by reducing the code to reproduce the problem has not been fulfillled. I feel that the abovementioned test code is short enough to ask for help with it here. Any hint on how I could solve my problem would be appreciated very much. There are some points which were not clarified yet; I am citing them below. So far, Friedrich > > - There are tools out there to analyze refcount problems. Python has > > some built-in tools for switching allocation strategies. > > Can you give me some pointer about this? > > > - numpy.asarray has a number of strategies to convert instances, which > > one is it using? > > I've tried to read about this, but couldn't find anything. What are > these different strategies? NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:
> Hi, > > Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt > <[hidden email]>: > > Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus < > > [hidden email]>: > > > Typically, one would create a complete example and then pointing > > > to the > > > code (as repo or pastebin, not as an attachment to a mail here). > > > > https://github.com/friedrichromstedt/bughunting-01 > > Last week I updated my example code to be more slim. There now > exists > a single-file extension module: > https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp > . > The corresponding test program > https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py > crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as > well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the > ``print`` > statement contained in the test file is commented out. > > My hope to be able to fix my error myself by reducing the code to > reproduce the problem has not been fulfillled. I feel that the > abovementioned test code is short enough to ask for help with it > here. > Any hint on how I could solve my problem would be appreciated very > much. valgrind), will allow you track down the issue (valgrind reports it from within python, running a python without debug symbols may obfuscate the actual problem; if that is the limiting you, I can post my valgrind output). Since you are running a linux system, I am confident that you can run it in valgrind to find it yourself. (There may be other ways.) Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and ignore some errors e.g. when importing NumPy. Cheers, Sebastian > > There are some points which were not clarified yet; I am citing them > below. > > So far, > Friedrich > > > > - There are tools out there to analyze refcount problems. Python > > > has > > > some built-in tools for switching allocation strategies. > > > > Can you give me some pointer about this? > > > > > - numpy.asarray has a number of strategies to convert instances, > > > which > > > one is it using? > > > > I've tried to read about this, but couldn't find anything. What > > are > > these different strategies? > _______________________________________________ > NumPy-Discussion mailing list > [hidden email] > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hi Friedrich, view->suboffsets = NULL; view->internal = NULL; to Image_getbuffer Best regards, Lev On Mon, Feb 15, 2021 at 10:57 PM Sebastian Berg <[hidden email]> wrote: On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote: _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
In reply to this post by Friedrich Romstedt
On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt
<[hidden email]> wrote: > > Hi, > > This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd > like to be able to convert some C++ extension type to a numpy array by > using ``numpy.asarray``. The extension type implements the Python > buffer interface to support this. > > The extension type, called "Image" here, holds some chunk of > ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; > the buffer is not shared with other objects. The following Python > code crashes:: > > image = <... Image production ...> > ar = numpy.asarray(image) > > However, when I say:: > > image = <... Image production ...> > print("---") > ar = numpy.asarray(image) > > the entire program is executing properly with correct data in the > numpy ndarray produced using the buffer interface. Maybe a dereference bug. Try setting pointers to NULL after freeing, something like this: delete[] view->shape; view->shape = NULL; delete[] view->strides; view->strides = NULL; ... delete[] self->data; self->data = NULL; _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
On Mon, Feb 15, 2021 at 7:35 PM Mansour Moufid <[hidden email]> wrote:
> > On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt > <[hidden email]> wrote: > > > > Hi, > > > > This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd > > like to be able to convert some C++ extension type to a numpy array by > > using ``numpy.asarray``. The extension type implements the Python > > buffer interface to support this. > > > > The extension type, called "Image" here, holds some chunk of > > ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; > > the buffer is not shared with other objects. The following Python > > code crashes:: > > > > image = <... Image production ...> > > ar = numpy.asarray(image) > > > > However, when I say:: > > > > image = <... Image production ...> > > print("---") > > ar = numpy.asarray(image) > > > > the entire program is executing properly with correct data in the > > numpy ndarray produced using the buffer interface. > > Maybe a dereference bug. > > Try setting pointers to NULL after freeing, something like this: > > delete[] view->shape; > view->shape = NULL; > delete[] view->strides; > view->strides = NULL; > > ... > > delete[] self->data; > self->data = NULL; Sorry for two messages in a row, I just noticed: I don't see the type's tp_free member defined? You can set it to PyObject_Free in Init_ImageType: ImageType.tp_free = PyObject_Free; See here: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_free _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
In reply to this post by Sebastian Berg
Hello again,
Am Mo., 15. Feb. 2021 um 16:57 Uhr schrieb Sebastian Berg <[hidden email]>: > > On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote: > > Last week I updated my example code to be more slim. There now > > exists > > a single-file extension module: > > https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp > > . > > The corresponding test program > > https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py > > crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as > > well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the > > ``print`` > > statement contained in the test file is commented out. > > I have tried it out, and can confirm that using debugging tools (namely > valgrind), will allow you track down the issue (valgrind reports it > from within python, running a python without debug symbols may > obfuscate the actual problem; if that is the limiting you, I can post > my valgrind output). > Since you are running a linux system, I am confident that you can run > it in valgrind to find it yourself. (There may be other ways.) > > Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and > ignore some errors e.g. when importing NumPy. From running ``PYTHONMALLOC=malloc valgrind python3 2021-01-11_0909.py`` (with the preceding call of ``print`` in :file:`2021-01-11_0909.py` commented out) I found a few things: - The call might or might not succeed. It doesn't always lead to a segfault. - "at 0x4A64A73: ??? (in /usr/lib/libpython3.9.so.1.0), called by 0x4A64914: PyMemoryView_FromObject (in /usr/lib/libpython3.9.so.1.0)", a "Conditional jump or move depends on uninitialised value(s)". After one more block of valgrind output ("Use of uninitialised value of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0)"), it finally leads either to "Invalid read of size 8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0) [...] Address 0x1 is not stack'd, malloc'd or (recently) free'd", resulting in a segfault, or just to another "Use of uninitialised value of size 8 at 0x48EEA15: ??? (in /usr/lib/libpython3.9.so.1.0)", after which the program completes successfully. - All this happens within "PyMemoryView_FromObject". So I can only guess that the "uninitialised value" is compared to 0x0, and when it is different (e.g. 0x1), it leads via "Address 0x1 is not stack'd, malloc'd or (recently) free'd" to the segfault observed. I suppose I need to compile Python and numpy myself to see the debug symbols instead of the "???" marks? Maybe even with ``-O0``? Furthermore, the shared object belonging to my code isn't involved directly in any way, so the segfault possibly has to do with some data I am leaving "uninitialised" at the moment. Thanks for the other replies as well; for the moment I feel that going the valgrind way might teach me how to debug errors of this kind myself. So far, Friedrich _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
I've reproduced the error you've described and got rid of it without valgrind. Those two lines are enough to avoid the segfault. But feel free to find it yourself :) Best regards, Lev On Tue, Feb 16, 2021 at 5:02 PM Friedrich Romstedt <[hidden email]> wrote: Hello again, _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hi Lev,
Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov <[hidden email]>: > > I've reproduced the error you've described and got rid of it without valgrind. > Those two lines are enough to avoid the segfault. Okay, good to know, I'll try it! Thanks for looking into it. > But feel free to find it yourself :) Yes :-D Best wishes, Friedrich _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
On Tue, 2021-02-16 at 12:40 +0100, Friedrich Romstedt wrote:
> Hi Lev, > > Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov < > [hidden email]>: > > > > I've reproduced the error you've described and got rid of it > > without valgrind. > > Those two lines are enough to avoid the segfault. > > Okay, good to know, I'll try it! Thanks for looking into it. valgrind in that case is often helpful and typically quick (it runs slow, but not much preparation needed). Especially because you reported it succeeding sometimes, where "uninitialized" might help, although I guess a `gdb` backtrace in the crash case might have been just as clear. With debugging symbols in Python (a full debug build makes sense), it mentioned "suboffsets" in a function name for me (maybe when a crash happened), a debug Python will also default to a debug malloc: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONMALLOC Which would not have been very useful here, but could be if you access a Python object after it was free'd for example. Uninitialized + "suboffsets" seemed fairly clear, but I may have underestimated it alot because I recognize "suboffsets" for buffers immediately. Cheers, Sebastian > > > But feel free to find it yourself :) > > Yes :-D > > Best wishes, > Friedrich > _______________________________________________ > NumPy-Discussion mailing list > [hidden email] > https://mail.python.org/mailman/listinfo/numpy-discussion > _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Free forum by Nabble | Edit this page |