Unreliable crash when converting using numpy.asarray via C buffer interface

[Friedrich Romstedt]

Hi,

This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10.  I'd
like to be able to convert some C++ extension type to a numpy array by
using ``numpy.asarray``.  The extension type implements the Python
buffer interface to support this.

The extension type, called "Image" here, holds some chunk of
``double``, C order, contiguous, 2 dimensions.  It "owns" the buffer;
the buffer is not shared with other objects.  The following Python
code crashes::

    image = <... Image production ...>
    ar = numpy.asarray(image)

However, when I say::

    image = <... Image production ...>
    print("---")
    ar = numpy.asarray(image)

the entire program is executing properly with correct data in the
numpy ndarray produced using the buffer interface.

The extension type permits reading the pixel values by a method;
copying them over by a Python loop works fine.  I am ``Py_INCREF``-ing
the producer in the C++ buffer view creation function properly.  The
shapes and strides of the buffer view are ``delete[]``-ed upon
releasing the buffer; avoiding this does not prevent the crash.  I am
catching ``std::exception`` in the view creation function; no such
exception occurs.  The shapes and strides are allocated by ``new
Py_ssize_t[2]``, so they will survive the view creation function.

I spent some hours trying to figure out what I am doing wrong.  Maybe
someone has an idea about this?  I double-checked each line of code
related to this problem and couldn't find any mistake.  Probabaly I am
not looking at the right aspect.

Best,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hi,

Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt
<[hidden email]>:

>
> [...] The following Python
> code crashes::
>
>     image = <... Image production ...>
>     ar = numpy.asarray(image)
>
> However, when I say::
>
>     image = <... Image production ...>
>     print("---")
>     ar = numpy.asarray(image)
>
> the entire program is executing properly with correct data in the
> numpy ndarray produced using the buffer interface.
>
> [...]

Does anyone have an idea about this?  By the way, I noticed that this
mailing list turned pretty quiet, am I missing something?

For completeness, the abovementioned "crash" shows up as just a
premature exit of the program.  There is no error message whatsoever.
The buffer view producing function raises Exceptions properly when
something goes wrong; also notice that this code completes without
error when the ``print("---")`` statement is in action.  So I presume
the culprit lies somewhere on the C level.  I can only guess that it
might be some side-effect unknown to me.

Best,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Hameer Abbasi-2]

Hey Friedrich,

If you can produce an MVCE that would be really helpful, along with your hardware and environment. Without that, it isn’t possible to be of much help.

https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

Best Regards,

Hameer Abbasi

--
Sent from Canary

On Dienstag, Jan. 26, 2021 at 9:49 AM, Friedrich Romstedt <[hidden email]> wrote:

Hi,

This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd
like to be able to convert some C++ extension type to a numpy array by
using ``numpy.asarray``. The extension type implements the Python
buffer interface to support this.

The extension type, called "Image" here, holds some chunk of
``double``, C order, contiguous, 2 dimensions. It "owns" the buffer;
the buffer is not shared with other objects. The following Python
code crashes::

image = <... Image production ...>
ar = numpy.asarray(image)

However, when I say::

image = <... Image production ...>
print("---")
ar = numpy.asarray(image)

the entire program is executing properly with correct data in the
numpy ndarray produced using the buffer interface.

The extension type permits reading the pixel values by a method;
copying them over by a Python loop works fine. I am ``Py_INCREF``-ing
the producer in the C++ buffer view creation function properly. The
shapes and strides of the buffer view are ``delete[]``-ed upon
releasing the buffer; avoiding this does not prevent the crash. I am
catching ``std::exception`` in the view creation function; no such
exception occurs. The shapes and strides are allocated by ``new
Py_ssize_t[2]``, so they will survive the view creation function.

I spent some hours trying to figure out what I am doing wrong. Maybe
someone has an idea about this? I double-checked each line of code
related to this problem and couldn't find any mistake. Probabaly I am
not looking at the right aspect.

Best,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[mattip]

On 2/1/21 9:57 AM, Friedrich Romstedt wrote:

> Hi,
>
> Am Di., 26. Jan. 2021 um 09:48 Uhr schrieb Friedrich Romstedt
> <[hidden email]>:
>> [...] The following Python
>> code crashes::
>>
>>      image = <... Image production ...>
>>      ar = numpy.asarray(image)
>>
>> However, when I say::
>>
>>      image = <... Image production ...>
>>      print("---")
>>      ar = numpy.asarray(image)
>>
>> the entire program is executing properly with correct data in the
>> numpy ndarray produced using the buffer interface.
>>
>> [...]
> Does anyone have an idea about this?  By the way, I noticed that this
> mailing list turned pretty quiet, am I missing something?
>
> For completeness, the abovementioned "crash" shows up as just a
> premature exit of the program.  There is no error message whatsoever.
> The buffer view producing function raises Exceptions properly when
> something goes wrong; also notice that this code completes without
> error when the ``print("---")`` statement is in action.  So I presume
> the culprit lies somewhere on the C level.  I can only guess that it
> might be some side-effect unknown to me.
>
> Best,
> Friedrich

It is very hard to help you from this description. It may be a refcount
problem, it may be a buffer protocol problem, it may be something else.
Typically, one would create a complete example and then pointing to the
code (as repo or pastebin, not as an attachment to a mail here). A few
things you might want to check:

- Make sure you give instructions how to build your project for Linux,
since most of the people on this list do not use windows.

- There are tools out there to analyze refcount problems. Python has
some built-in tools for switching allocation strategies.

- numpy.asarray has a number of strategies to convert instances, which
one is it using?


Matti

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hello Matti,

Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <[hidden email]>:
>
> [...]
>
> It is very hard to help you from this description. It may be a refcount
> problem, it may be a buffer protocol problem, it may be something else.

Yes, indeed!

> Typically, one would create a complete example and then pointing to the
> code (as repo or pastebin, not as an attachment to a mail here).

https://github.com/friedrichromstedt/bughunting-01

I boiled it down considerably, compared to the program where I
stumbled upon the problem.  In the abovementioned repo, you find a
Python test script in the `test/` folder.  Therein, a single `print`
statement can be used to trigger or to avoid the error.  On Linux, I
get a somewhat more precise description than just from the premature
exit on Windows: It is a segfault.

Certainly it is still asked quite much to skim through my source code,
however, I hope that I trimmed it down sufficiently.

> - Make sure you give instructions how to build your project for Linux,
> since most of the people on this list do not use windows.

The code reproducing the segfault can be compiled by `$ python3
setup.py install`, both on Windows as well as on Linux.

> - There are tools out there to analyze refcount problems. Python has
> some built-in tools for switching allocation strategies.

Can you give me some pointer about this?

> - numpy.asarray has a number of strategies to convert instances, which
> one is it using?

I've tried to read about this, but coudn't find anything.  What are
these different strategies?

Many thanks in advance,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hi,

Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt
<[hidden email]>:
> Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <[hidden email]>:
> > Typically, one would create a complete example and then pointing to the
> > code (as repo or pastebin, not as an attachment to a mail here).
>
> https://github.com/friedrichromstedt/bughunting-01

Last week I updated my example code to be more slim.  There now exists
a single-file extension module:
https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp.
The corresponding test program
https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the ``print``
statement contained in the test file is commented out.

My hope to be able to fix my error myself by reducing the code to
reproduce the problem has not been fulfillled.  I feel that the
abovementioned test code is short enough to ask for help with it here.
Any hint on how I could solve my problem would be appreciated very
much.

There are some points which were not clarified yet; I am citing them below.

So far,
Friedrich

> > - There are tools out there to analyze refcount problems. Python has
> > some built-in tools for switching allocation strategies.
>
> Can you give me some pointer about this?
>
> > - numpy.asarray has a number of strategies to convert instances, which
> > one is it using?
>
> I've tried to read about this, but couldn't find anything.  What are
> these different strategies?

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Sebastian Berg]

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:

> Hi,
>
> Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt
> <[hidden email]>:
> > Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <
> > [hidden email]>:
> > > Typically, one would create a complete example and then pointing
> > > to the
> > > code (as repo or pastebin, not as an attachment to a mail here).
> >
> > https://github.com/friedrichromstedt/bughunting-01
>
> Last week I updated my example code to be more slim.  There now
> exists
> a single-file extension module:
> https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp
> .
> The corresponding test program
> https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
> crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
> well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the
> ``print``
> statement contained in the test file is commented out.
>
> My hope to be able to fix my error myself by reducing the code to
> reproduce the problem has not been fulfillled.  I feel that the
> abovementioned test code is short enough to ask for help with it
> here.
> Any hint on how I could solve my problem would be appreciated very
> much.

I have tried it out, and can confirm that using debugging tools (namely
valgrind), will allow you track down the issue (valgrind reports it
from within python, running a python without debug symbols may
obfuscate the actual problem; if that is the limiting you, I can post
my valgrind output).
Since you are running a linux system, I am confident that you can run
it in valgrind to find it yourself.  (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and
ignore some errors e.g. when importing NumPy.

Cheers,

Sebastian

>
> There are some points which were not clarified yet; I am citing them
> below.
>
> So far,
> Friedrich
>
> > > - There are tools out there to analyze refcount problems. Python
> > > has
> > > some built-in tools for switching allocation strategies.
> >
> > Can you give me some pointer about this?
> >
> > > - numpy.asarray has a number of strategies to convert instances,
> > > which
> > > one is it using?
> >
> > I've tried to read about this, but couldn't find anything.  What
> > are
> > these different strategies?
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[axil]

Hi Friedrich,

Try adding 
    view->suboffsets = NULL;
    view->internal = NULL;

to Image_getbuffer

Best regards,

Lev

On Mon, Feb 15, 2021 at 10:57 PM Sebastian Berg <[hidden email]> wrote:

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:
> Hi,
>
> Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt
> <[hidden email]>:
> > Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <
> > [hidden email]>:
> > > Typically, one would create a complete example and then pointing
> > > to the
> > > code (as repo or pastebin, not as an attachment to a mail here).
> >
> > https://github.com/friedrichromstedt/bughunting-01
>
> Last week I updated my example code to be more slim.  There now
> exists
> a single-file extension module:
> https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp
> .
> The corresponding test program
> https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
> crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
> well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the
> ``print``
> statement contained in the test file is commented out.
>
> My hope to be able to fix my error myself by reducing the code to
> reproduce the problem has not been fulfillled.  I feel that the
> abovementioned test code is short enough to ask for help with it
> here.
> Any hint on how I could solve my problem would be appreciated very
> much.

I have tried it out, and can confirm that using debugging tools (namely
valgrind), will allow you track down the issue (valgrind reports it
from within python, running a python without debug symbols may
obfuscate the actual problem; if that is the limiting you, I can post
my valgrind output).
Since you are running a linux system, I am confident that you can run
it in valgrind to find it yourself.  (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and
ignore some errors e.g. when importing NumPy.

Cheers,

Sebastian


>
> There are some points which were not clarified yet; I am citing them
> below.
>
> So far,
> Friedrich
>
> > > - There are tools out there to analyze refcount problems. Python
> > > has
> > > some built-in tools for switching allocation strategies.
> >
> > Can you give me some pointer about this?
> >
> > > - numpy.asarray has a number of strategies to convert instances,
> > > which
> > > one is it using?
> >
> > I've tried to read about this, but couldn't find anything.  What
> > are
> > these different strategies?
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Mansour Moufid]

On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt
<[hidden email]> wrote:

>
> Hi,
>
> This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10.  I'd
> like to be able to convert some C++ extension type to a numpy array by
> using ``numpy.asarray``.  The extension type implements the Python
> buffer interface to support this.
>
> The extension type, called "Image" here, holds some chunk of
> ``double``, C order, contiguous, 2 dimensions.  It "owns" the buffer;
> the buffer is not shared with other objects.  The following Python
> code crashes::
>
>     image = <... Image production ...>
>     ar = numpy.asarray(image)
>
> However, when I say::
>
>     image = <... Image production ...>
>     print("---")
>     ar = numpy.asarray(image)
>
> the entire program is executing properly with correct data in the
> numpy ndarray produced using the buffer interface.

Maybe a dereference bug.

Try setting pointers to NULL after freeing, something like this:

    delete[] view->shape;
    view->shape = NULL;
    delete[] view->strides;
    view->strides = NULL;

...

    delete[] self->data;
    self->data = NULL;
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Mansour Moufid]

On Mon, Feb 15, 2021 at 7:35 PM Mansour Moufid <[hidden email]> wrote:

>
> On Tue, Jan 26, 2021 at 3:50 AM Friedrich Romstedt
> <[hidden email]> wrote:
> >
> > Hi,
> >
> > This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10.  I'd
> > like to be able to convert some C++ extension type to a numpy array by
> > using ``numpy.asarray``.  The extension type implements the Python
> > buffer interface to support this.
> >
> > The extension type, called "Image" here, holds some chunk of
> > ``double``, C order, contiguous, 2 dimensions.  It "owns" the buffer;
> > the buffer is not shared with other objects.  The following Python
> > code crashes::
> >
> >     image = <... Image production ...>
> >     ar = numpy.asarray(image)
> >
> > However, when I say::
> >
> >     image = <... Image production ...>
> >     print("---")
> >     ar = numpy.asarray(image)
> >
> > the entire program is executing properly with correct data in the
> > numpy ndarray produced using the buffer interface.
>
> Maybe a dereference bug.
>
> Try setting pointers to NULL after freeing, something like this:
>
>     delete[] view->shape;
>     view->shape = NULL;
>     delete[] view->strides;
>     view->strides = NULL;
>
> ...
>
>     delete[] self->data;
>     self->data = NULL;

Sorry for two messages in a row, I just noticed:

I don't see the type's tp_free member defined?

You can set it to PyObject_Free in Init_ImageType:

    ImageType.tp_free = PyObject_Free;

See here: https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_free
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hello again,

Am Mo., 15. Feb. 2021 um 16:57 Uhr schrieb Sebastian Berg
<[hidden email]>:

>
> On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:
> > Last week I updated my example code to be more slim.  There now
> > exists
> > a single-file extension module:
> > https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp
> > .
> > The corresponding test program
> > https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
> > crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
> > well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the
> > ``print``
> > statement contained in the test file is commented out.
>
> I have tried it out, and can confirm that using debugging tools (namely
> valgrind), will allow you track down the issue (valgrind reports it
> from within python, running a python without debug symbols may
> obfuscate the actual problem; if that is the limiting you, I can post
> my valgrind output).
> Since you are running a linux system, I am confident that you can run
> it in valgrind to find it yourself.  (There may be other ways.)
>
> Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and
> ignore some errors e.g. when importing NumPy.

From running ``PYTHONMALLOC=malloc valgrind python3
2021-01-11_0909.py`` (with the preceding call of ``print`` in
:file:`2021-01-11_0909.py` commented out) I found a few things:

-   The call might or might not succeed.  It doesn't always lead to a segfault.
-   "at 0x4A64A73: ??? (in /usr/lib/libpython3.9.so.1.0), called by
0x4A64914: PyMemoryView_FromObject (in /usr/lib/libpython3.9.so.1.0)",
a "Conditional jump or move depends on uninitialised value(s)".  After
one more block of valgrind output ("Use of uninitialised value of size
8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0)"), it finally
leads either to "Invalid read of size 8 at 0x48EEA1B: ??? (in
/usr/lib/libpython3.9.so.1.0) [...] Address 0x1 is not stack'd,
malloc'd or (recently) free'd", resulting in a segfault, or just to
another "Use of uninitialised value of size 8 at 0x48EEA15: ??? (in
/usr/lib/libpython3.9.so.1.0)", after which the program completes
successfully.
-   All this happens within "PyMemoryView_FromObject".

So I can only guess that the "uninitialised value" is compared to 0x0,
and when it is different (e.g. 0x1), it leads via "Address 0x1 is not
stack'd, malloc'd or (recently) free'd" to the segfault observed.

I suppose I need to compile Python and numpy myself to see the debug
symbols instead of the "???" marks? Maybe even with ``-O0``?

Furthermore, the shared object belonging to my code isn't involved
directly in any way, so the segfault possibly has to do with some data
I am leaving "uninitialised" at the moment.

Thanks for the other replies as well; for the moment I feel that going
the valgrind way might teach me how to debug errors of this kind
myself.

So far,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[axil]

I've reproduced the error you've described and got rid of it without valgrind.

Those two lines are enough to avoid the segfault.

But feel free to find it yourself :)

Best regards,

Lev

On Tue, Feb 16, 2021 at 5:02 PM Friedrich Romstedt <[hidden email]> wrote:

Hello again,

Am Mo., 15. Feb. 2021 um 16:57 Uhr schrieb Sebastian Berg
<[hidden email]>:
>
> On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:
> > Last week I updated my example code to be more slim.  There now
> > exists
> > a single-file extension module:
> > https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bughuntingfrmod/bughuntingfrmod.cpp
> > .
> > The corresponding test program
> > https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
> > crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
> > well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the
> > ``print``
> > statement contained in the test file is commented out.
>
> I have tried it out, and can confirm that using debugging tools (namely
> valgrind), will allow you track down the issue (valgrind reports it
> from within python, running a python without debug symbols may
> obfuscate the actual problem; if that is the limiting you, I can post
> my valgrind output).
> Since you are running a linux system, I am confident that you can run
> it in valgrind to find it yourself.  (There may be other ways.)
>
> Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and
> ignore some errors e.g. when importing NumPy.

From running ``PYTHONMALLOC=malloc valgrind python3
2021-01-11_0909.py`` (with the preceding call of ``print`` in
:file:`2021-01-11_0909.py` commented out) I found a few things:

-   The call might or might not succeed.  It doesn't always lead to a segfault.
-   "at 0x4A64A73: ??? (in /usr/lib/libpython3.9.so.1.0), called by
0x4A64914: PyMemoryView_FromObject (in /usr/lib/libpython3.9.so.1.0)",
a "Conditional jump or move depends on uninitialised value(s)".  After
one more block of valgrind output ("Use of uninitialised value of size
8 at 0x48EEA1B: ??? (in /usr/lib/libpython3.9.so.1.0)"), it finally
leads either to "Invalid read of size 8 at 0x48EEA1B: ??? (in
/usr/lib/libpython3.9.so.1.0) [...] Address 0x1 is not stack'd,
malloc'd or (recently) free'd", resulting in a segfault, or just to
another "Use of uninitialised value of size 8 at 0x48EEA15: ??? (in
/usr/lib/libpython3.9.so.1.0)", after which the program completes
successfully.
-   All this happens within "PyMemoryView_FromObject".

So I can only guess that the "uninitialised value" is compared to 0x0,
and when it is different (e.g. 0x1), it leads via "Address 0x1 is not
stack'd, malloc'd or (recently) free'd" to the segfault observed.

I suppose I need to compile Python and numpy myself to see the debug
symbols instead of the "???" marks? Maybe even with ``-O0``?

Furthermore, the shared object belonging to my code isn't involved
directly in any way, so the segfault possibly has to do with some data
I am leaving "uninitialised" at the moment.

Thanks for the other replies as well; for the moment I feel that going
the valgrind way might teach me how to debug errors of this kind
myself.

So far,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hi Lev,

Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov <[hidden email]>:
>
> I've reproduced the error you've described and got rid of it without valgrind.
> Those two lines are enough to avoid the segfault.

Okay, good to know, I'll try it! Thanks for looking into it.

> But feel free to find it yourself :)

Yes :-D

Best wishes,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Sebastian Berg]

On Tue, 2021-02-16 at 12:40 +0100, Friedrich Romstedt wrote:

> Hi Lev,
>
> Am Di., 16. Feb. 2021 um 11:50 Uhr schrieb Lev Maximov <
> [hidden email]>:
> >
> > I've reproduced the error you've described and got rid of it
> > without valgrind.
> > Those two lines are enough to avoid the segfault.
>
> Okay, good to know, I'll try it! Thanks for looking into it.

Yeah, sorry if I was too fuzzy.  Your error was random, and checking
valgrind in that case is often helpful and typically quick (it runs
slow, but not much preparation needed).
Especially because you reported it succeeding sometimes, where
"uninitialized" might help, although I guess a `gdb` backtrace in the
crash case might have been just as clear.

With debugging symbols in Python (a full debug build makes sense), it
mentioned "suboffsets" in a function name for me (maybe when a crash
happened), a debug Python will also default to a debug malloc:
https://docs.python.org/3/using/cmdline.html#envvar-PYTHONMALLOC
Which would not have been very useful here, but could be if you access
a Python object after it was free'd for example.

Uninitialized + "suboffsets" seemed fairly clear, but I may have
underestimated it alot because I recognize "suboffsets" for buffers
immediately.

Cheers,

Sebastian
 

>
> > But feel free to find it yourself :)
>
> Yes :-D
>
> Best wishes,
> Friedrich
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Friedrich Romstedt]

Hi Matti, Sebastian and Lev,

Am Mo., 15. Feb. 2021 um 18:50 Uhr schrieb Lev Maximov <[hidden email]>:
>
> Try adding
>     view->suboffsets = NULL;
>     view->internal = NULL;
> to Image_getbuffer

finally I got it working easily using Lev's pointer cited above.  I
didn't follow the valgrind approach furthermore, since I found it
likely that it'd produce the same finding.

This is just to let you know; I applied the fix several weeks ago.

Many thanks,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[axil]

I'm glad you sorted it out as the subject line sounded quite horrifying )

Best regards,

Lev

On Mon, Mar 29, 2021 at 2:54 PM Friedrich Romstedt <[hidden email]> wrote:

Hi Matti, Sebastian and Lev,

Am Mo., 15. Feb. 2021 um 18:50 Uhr schrieb Lev Maximov <[hidden email]>:
>
> Try adding
>     view->suboffsets = NULL;
>     view->internal = NULL;
> to Image_getbuffer

finally I got it working easily using Lev's pointer cited above.  I
didn't follow the valgrind approach furthermore, since I found it
likely that it'd produce the same finding.

This is just to let you know; I applied the fix several weeks ago.

Many thanks,
Friedrich
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

[Gunter Meissner]

Aloha Numpy Community,

I am just writing a book on "How to Cheat in Statistics - And get Away with It". 

I noticed there is no built-in syntax for the 'Adjusted R-squared' in any library (do correct me if I am wrong)

I think it would be a good idea to program it. The math is straight forward, I can 

provide it if desired. 

Thank you,

Gunter  

On Mon, Feb 15, 2021 at 5:56 AM Sebastian Berg <[hidden email]> wrote:

On Mon, 2021-02-15 at 10:12 +0100, Friedrich Romstedt wrote:
> Hi,
>
> Am Do., 4. Feb. 2021 um 09:07 Uhr schrieb Friedrich Romstedt
> <[hidden email]>:
> > Am Mo., 1. Feb. 2021 um 09:46 Uhr schrieb Matti Picus <
> > [hidden email]>:
> > > Typically, one would create a complete example and then pointing
> > > to the
> > > code (as repo or pastebin, not as an attachment to a mail here).
> >
> > https://github.com/friedrichromstedt/bughunting-01
>
> Last week I updated my example code to be more slim.  There now
> exists
> a single-file extension module:
> https://github.com/friedrichromstedt/bughunting-01/blob/master/lib/bugIhuntingfrmod/bughuntingfrmod.cpp
> .
> The corresponding test program
> https://github.com/friedrichromstedt/bughunting-01/blob/master/test/2021-02-11_0909.py
> crashes "properly" both on Windows 10 (Python 3.8.2, numpy 1.19.2) as
> well as on Arch Linux (Python 3.9.1, numpy 1.20.0), when the
> ``print``
> statement contained in the test file is commented out.
>
> My hope to be able to fix my error myself by reducing the code to
> reproduce the problem has not been fulfillled.  I feel that the
> abovementioned test code is short enough to ask for help with it
> here.
> Any hint on how I could solve my problem would be appreciated very
> much.

I have tried it out, and can confirm that using debugging tools (namely
valgrind), will allow you track down the issue (valgrind reports it
from within python, running a python without debug symbols may
obfuscate the actual problem; if that is the limiting you, I can post
my valgrind output).
Since you are running a linux system, I am confident that you can run
it in valgrind to find it yourself.  (There may be other ways.)

Just remember to run valgrind with `PYTHONMALLOC=malloc valgrind` and
ignore some errors e.g. when importing NumPy.

Cheers,

Sebastian


>
> There are some points which were not clarified yet; I am citing them
> below.
>
> So far,
> Friedrich
>
> > > - There are tools out there to analyze refcount problems. Python
> > > has
> > > some built-in tools for switching allocation strategies.
> >
> > Can you give me some pointer about this?
> >
> > > - numpy.asarray has a number of strategies to convert instances,
> > > which
> > > one is it using?
> >
> > I've tried to read about this, but couldn't find anything.  What
> > are
> > these different strategies?
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

--

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion