embedded PyArray_FromDimsAndData Segmentation Fault

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

embedded PyArray_FromDimsAndData Segmentation Fault

Thomas Hrabe-2
embedded PyArray_FromDimsAndData Segmentation Fault

Hi all,

the PyArray_FromDimsAndData is still cousing me headaches.

Is there anybody out there finding the error of the following code?

#include "Python.h"
#include <numpy/ndarrayobject.h>
int main(int argc,char** argv)
{

        int dimensions = 2;

        void* value = malloc(sizeof(double)*100);

        int* size = (int*)malloc(sizeof(int)*2);

        size[0] = 10;
        size[1] = 10;

        for(int i=0;i<100;i++)
                ((double*)value)[i] = 1.0;

        for(int i=0;i<100;i++)
                printf("%e ",((double*)value)[i]);

        printf("\n%d %d\n",dimensions,size[0]);
        PyArray_FromDimsAndData(dimensions,size,NPY_DOUBLELTR,(char*)value); //TROUBLE HERE

        return 0;
}

I allway get a segmentation fault at the PyArray_FromDimsAndData call.
I want to create copies of c arrays, copy them into a running python interpreter as nd-arrays and modify them with some python functions.

If I did this in a module, I would have to call the
import_array();
function, I know. However, this is all outside of any module and when I add it before PyArray_FromDimsAndData I get the following compilation error:
src/test.cpp:24: error: return-statement with no value, in function returning 'int'

Does anybody have a clue?
Best,
Thomas




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndData Segmentation Fault

Robert Kern-2
On Wed, May 14, 2008 at 5:42 PM, Thomas Hrabe <[hidden email]> wrote:

>
> Hi all,
>
> the PyArray_FromDimsAndData is still cousing me headaches.
>
> Is there anybody out there finding the error of the following code?
>
> #include "Python.h"
> #include <numpy/ndarrayobject.h>
> int main(int argc,char** argv)
> {
>
>         int dimensions = 2;
>
>         void* value = malloc(sizeof(double)*100);
>
>         int* size = (int*)malloc(sizeof(int)*2);
>
>         size[0] = 10;
>         size[1] = 10;
>
>         for(int i=0;i<100;i++)
>                 ((double*)value)[i] = 1.0;
>
>         for(int i=0;i<100;i++)
>                 printf("%e ",((double*)value)[i]);
>
>         printf("\n%d %d\n",dimensions,size[0]);
>         PyArray_FromDimsAndData(dimensions,size,NPY_DOUBLELTR,(char*)value);
> //TROUBLE HERE
>
>         return 0;
> }
>
> I allway get a segmentation fault at the PyArray_FromDimsAndData call.
> I want to create copies of c arrays, copy them into a running python
> interpreter as nd-arrays and modify them with some python functions.
>
> If I did this in a module, I would have to call the
> import_array();
> function, I know. However, this is all outside of any module and when I add
> it before PyArray_FromDimsAndData I get the following compilation error:
> src/test.cpp:24: error: return-statement with no value, in function
> returning 'int'

You can't use numpy outside of Python. Put your code into a Python
extension module. I can explain the proximate causes of the error
messages if you really want, but they aren't really relevant. The
ultimate problem is that you aren't in a Python extension module.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndData Segmentation Fault

Stéfan van der Walt
In reply to this post by Thomas Hrabe-2
Hi Thomas

2008/5/15 Thomas Hrabe <[hidden email]>:
> PyArray_FromDimsAndData(dimensions,size,NPY_DOUBLELTR,(char*)value);
> //TROUBLE HERE

I didn't know a person could write a stand-alone program using NumPy
this way (can you?);  but what I do know is that FromDimsAndData is
deprecated, and that it can be replaced here by

        PyArray_SimpleNewFromData(dimensions, size, NPY_CDOUBLE, value);

where

        npy_intp* size = malloc(sizeof(npy_intp)*2);

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndDataSegmentation Fault

Thomas Hrabe-2
>I didn't know a person could write a stand-alone program using NumPy
>this way (can you?)

Well, this is possible when you embed python and use the "simple" objects such as ints, strings, ....
Why should it be impossible to do it for numpy then?
My plan is to send multidimensional arrays from C to python and to apply some python specific functions to them.




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

winmail.dat (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndDataSegmentation Fault

Robert Kern-2
On Wed, May 14, 2008 at 6:40 PM, Thomas Hrabe <[hidden email]> wrote:
>>I didn't know a person could write a stand-alone program using NumPy
>>this way (can you?)
>
> Well, this is possible when you embed python and use the "simple" objects such as ints, strings, ....
> Why should it be impossible to do it for numpy then?

numpy exposes its API as a pointer to an array which contains function
pointers. import_array() imports the extension module, accesses the
PyCObject that contains this pointer, and sets a global pointer
appropriately. There are #defines macros to emulate the functions by
dereferencing the appropriate element of the array and calling it with
the given macro arguments. The reason you get the error about
returning nothing when the return type of main() is declared int is
because this macro is only intended to work inside of an initmodule()
function of an extension module, whose return type is void.
import_array() includes error handling logic and will return if there
is an error. You get the segfault without import_array() because all
of the functions you try to call are trying to dereference an array
which has not been initialized.

> My plan is to send multidimensional arrays from C to python and to apply some python specific functions to them.

Well, first you need to call Py_Initialize() to start the VM.
Otherwise, you can't import numpy to begin with. I guess you could
write a "void load_numpy(void)" function which just exists to call
import_array(). Just be sure to check the exception state
appropriately after it returns.

But for the most part, it's much better to drive your C code using
Python than the other around.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndDataSegmentationFault

Thomas Hrabe-2

>But for the most part, it's much better to drive your C code using
>Python than the other around.

True, the other way arround works fine.
Will investigate further tomorrow, because its a must have feature...
Thanks

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

winmail.dat (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndDataSegmentationFault

Thomas Hrabe-2
In reply to this post by Robert Kern-2
RE: [Numpy-discussion] embedded PyArray_FromDimsAndDataSegmentationFault

After all, I figured how to create an numpy in C with the help below.

If called in C, import_array() but actually _import_array() successfully creates all the instances needed for the array.
However, once I call this function from another environment such as Matlab,
  PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
in __import_array() returns NULL, because numpy.core.multiarray is not found?
Do you think it might depend on the path settings?

As I said, the code works fine from plain C but its odd from within Matlab.

Best,
Thomas



-----Original Message-----
From: [hidden email] on behalf of Robert Kern
Sent: Wed 5/14/2008 5:12 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] embedded PyArray_FromDimsAndDataSegmentationFault

On Wed, May 14, 2008 at 6:40 PM, Thomas Hrabe <[hidden email]> wrote:
>>I didn't know a person could write a stand-alone program using NumPy
>>this way (can you?)
>
> Well, this is possible when you embed python and use the "simple" objects such as ints, strings, ....
> Why should it be impossible to do it for numpy then?

numpy exposes its API as a pointer to an array which contains function
pointers. import_array() imports the extension module, accesses the
PyCObject that contains this pointer, and sets a global pointer
appropriately. There are #defines macros to emulate the functions by
dereferencing the appropriate element of the array and calling it with
the given macro arguments. The reason you get the error about
returning nothing when the return type of main() is declared int is
because this macro is only intended to work inside of an initmodule()
function of an extension module, whose return type is void.
import_array() includes error handling logic and will return if there
is an error. You get the segfault without import_array() because all
of the functions you try to call are trying to dereference an array
which has not been initialized.

> My plan is to send multidimensional arrays from C to python and to apply some python specific functions to them.

Well, first you need to call Py_Initialize() to start the VM.
Otherwise, you can't import numpy to begin with. I guess you could
write a "void load_numpy(void)" function which just exists to call
import_array(). Just be sure to check the exception state
appropriately after it returns.

But for the most part, it's much better to drive your C code using
Python than the other around.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embedded PyArray_FromDimsAndDataSegmentationFault

Robert Kern-2
On Tue, May 20, 2008 at 2:01 PM, Thomas Hrabe <[hidden email]> wrote:
> After all, I figured how to create an numpy in C with the help below.
>
> If called in C, import_array() but actually _import_array() successfully
> creates all the instances needed for the array.
> However, once I call this function from another environment such as Matlab,
>   PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
> in __import_array() returns NULL, because numpy.core.multiarray is not
> found?

Something like that. Call PyErr_Print() do display the full traceback
so you can find out what the actual problem is.

> Do you think it might depend on the path settings?

You've called Py_Initialize() before you do anything else with Py*
functions, right?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: embeddedPyArray_FromDimsAndDataSegmentationFault

Thomas Hrabe-2
Thats what PyErr_Print() prints.
Python is initialised, for sure

Traceback (most recent call last):
  File "/usr/global/python32/lib/python2.4/site-packages/numpy/__init__.py", lin
e 34, in ?
    import testing
  File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/__init__.
py", line 3, in ?
    from numpytest import *
  File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/numpytest
.py", line 8, in ?
    import unittest
  File "/usr/global/python32/lib/python2.4/unittest.py", line 51, in ?
    import time
ImportError: /usr/global/python32/lib/python2.4/lib-dynload/time.so: undefined s
ymbol: PyExc_IOErro

why does it work in C and not in C started within Matlab?

-----Original Message-----
From: [hidden email] on behalf of Robert Kern
Sent: Tue 5/20/2008 12:23 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] embeddedPyArray_FromDimsAndDataSegmentationFault
 
On Tue, May 20, 2008 at 2:01 PM, Thomas Hrabe <[hidden email]> wrote:
> After all, I figured how to create an numpy in C with the help below.
>
> If called in C, import_array() but actually _import_array() successfully
> creates all the instances needed for the array.
> However, once I call this function from another environment such as Matlab,
>   PyObject *numpy = PyImport_ImportModule("numpy.core.multiarray");
> in __import_array() returns NULL, because numpy.core.multiarray is not
> found?

Something like that. Call PyErr_Print() do display the full traceback
so you can find out what the actual problem is.

> Do you think it might depend on the path settings?

You've called Py_Initialize() before you do anything else with Py*
functions, right?

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

winmail.dat (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: embeddedPyArray_FromDimsAndDataSegmentationFault

Robert Kern-2
On Tue, May 20, 2008 at 3:36 PM, Thomas Hrabe <[hidden email]> wrote:

> Thats what PyErr_Print() prints.
> Python is initialised, for sure
>
> Traceback (most recent call last):
>  File "/usr/global/python32/lib/python2.4/site-packages/numpy/__init__.py", lin
> e 34, in ?
>    import testing
>  File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/__init__.
> py", line 3, in ?
>    from numpytest import *
>  File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/numpytest
> .py", line 8, in ?
>    import unittest
>  File "/usr/global/python32/lib/python2.4/unittest.py", line 51, in ?
>    import time
> ImportError: /usr/global/python32/lib/python2.4/lib-dynload/time.so: undefined s
> ymbol: PyExc_IOErro
>
> why does it work in C and not in C started within Matlab?

It depends on how you linked everything. Presumably, you linked in
libpython24 for the program but not the MEX.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: (SPAM) Re: embeddedPyArray_FromDimsAndDataSegmentationFault

Pauli Virtanen-3
In reply to this post by Thomas Hrabe-2
ti, 2008-05-20 kello 13:36 -0700, Thomas Hrabe kirjoitti:

> Thats what PyErr_Print() prints.
> Python is initialised, for sure
>
> Traceback (most recent call last):
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/__init__.py", lin
> e 34, in ?
>     import testing
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/__init__.
> py", line 3, in ?
>     from numpytest import *
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/numpytest
> py", line 8, in ?
>     import unittest
>   File "/usr/global/python32/lib/python2.4/unittest.py", line 51, in ?
>     import time
> ImportError: /usr/global/python32/lib/python2.4/lib-dynload/time.so: undefined s
> ymbol: PyExc_IOErro
>
> why does it work in C and not in C started within Matlab?

It's probably that it has something to do with the way Matlab loads its
extension libraries:

http://www.iki.fi/pav/software/pythoncall/index.html#compilation-important

--
Pauli Virtanen


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: (SPAM) Re:embeddedPyArray_FromDimsAndDataSegmentationFault

Thomas Hrabe-2
RE: [Numpy-discussion] (SPAM) Re:embeddedPyArray_FromDimsAndDataSegmentationFault

Hi Pauli

my makefile compiles with the following options

/usr/global/matlab/bin/mex -cxx -g
-I./include/ -I/home/global/python32/include/python2.4
-I/home/global/python32/lib/python2.4/site-packages/numpy/core/include/
-DLIBPYTHON=\"/usr/global/python32/lib/libpython2.4.so\"
....
-lpython2.4
With this options compilation is processed but many symbols remain unsolved

undefined symbol: PyExc_RuntimeError    (./bin/mp.mexglx)
....

When I use -lpython2.4 the linker woun't find the lib
/usr/bin/ld: cannot find -lpython2.4

But I bet you have managed to solve this problem. By the way, what Matlab version did you use?
Best,
Thomas


-----Original Message-----
From: [hidden email] on behalf of Pauli Virtanen
Sent: Tue 5/20/2008 2:10 PM
To: Discussion of Numerical Python
Subject: Re: [Numpy-discussion] (SPAM) Re:embeddedPyArray_FromDimsAndDataSegmentationFault

ti, 2008-05-20 kello 13:36 -0700, Thomas Hrabe kirjoitti:
> Thats what PyErr_Print() prints.
> Python is initialised, for sure
>
> Traceback (most recent call last):
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/__init__.py", lin
> e 34, in ?
>     import testing
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/__init__.
> py", line 3, in ?
>     from numpytest import *
>   File "/usr/global/python32/lib/python2.4/site-packages/numpy/testing/numpytest
> py", line 8, in ?
>     import unittest
>   File "/usr/global/python32/lib/python2.4/unittest.py", line 51, in ?
>     import time
> ImportError: /usr/global/python32/lib/python2.4/lib-dynload/time.so: undefined s
> ymbol: PyExc_IOErro
>
> why does it work in C and not in C started within Matlab?

It's probably that it has something to do with the way Matlab loads its
extension libraries:

http://www.iki.fi/pav/software/pythoncall/index.html#compilation-important

--
Pauli Virtanen


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion