NumPtr vs NumPy.i to access C

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

NumPtr vs NumPy.i to access C

maitai

Hi, I'd like to access a C function from python, and the function takes input/output arrays. I'd probably use SWIG to do the interface to the C code. I found 2 options:
-NumPtr module, to access Numeric arrays as pointers
http://www.penzilla.net/tutorials/python/numptr/
- numpy.i, a SWIG interface file for NumPy that defines typemaps
http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/swig/doc/numpy_swig.html

I'm not sure if there is significant differences between the 2 options (besides using either NumPy or Numeric). Does numpy.i interface file use pointers to access NumPy arrays? or does it make a copy of the array to pass it to/from the C function?

I'm new to programming and I'd like to make sure of this. I need to use in C very large arrays frequently, so I want to avoid making copies of it, because speed will be an important factor.

Thanks in advance!



_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Dag Sverre Seljebotn
Jose Martin wrote:

>
> Hi, I'd like to access a C function from python, and the function takes
> input/output arrays. I'd probably use SWIG to do the interface to the C
> code. I found 2 options:
> -NumPtr module, to access Numeric arrays as pointers
> http://www.penzilla.net/tutorials/python/numptr/
> - numpy.i, a SWIG interface file for NumPy that defines typemaps
> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/swig/doc/numpy_swig.html
>
> I'm not sure if there is significant differences between the 2 options
> (besides using either NumPy or Numeric). Does numpy.i interface file use
> pointers to access NumPy arrays? or does it make a copy of the array to
> pass it to/from the C function?

I'll use the opportunity to recommend Cython, http://cython.org.
(Disclaimer: I am not objective as I am very involved with Cython
development, in particular I'll spend this summer improving NumPy support
in it. You should listen to others on SWIG.)

Cython is a different approach from SWIG (see
http://wiki.cython.org/WrappingCorCpp; in particular SWIG uses more layers
of indirection).

To wrap a function, say, dealing with a 2-D float64 NumPy array, you'd
write the following Cython code (Cython is a seperate language, sort of a
mix of Python and C):

import numpy
cimport numpy

import_array()

cdef extern from "your_c_header.h":
    int your_func(numpy.float64* buf, unsigned int width, unsigned int
height)

def call_your_func(numpy.ndarray arr):
    if arr.ndim != 2: raise ValueError("Need 2D array")
    # ... and so on. Better make sure array is C-contiguous too!
    # (or convert it)

    return your_func(<numpy.float64*>arr.data, arr.shape[0], arr.shape[1])

After Cython compilation you get a C file, and after compiling the C file
you can import it into Python like a normal module and call
"call_your_func" directly.

That's an overview. Basically, "arr" behaves exactly like a Python object,
except for when accessing the fields in which case values from the
underlying C implementation will be returned instead. More info can be
found on cython.org etc. (or ask!).

Dag Sverre

PS. You'll need an "numpy.pxd" (put it in the same directory, the
"cimport" pulls it in). Here's mine:

cdef extern from "Python.h":
    ctypedef int Py_intptr_t

cdef extern from "numpy/arrayobject.h":
    ctypedef class numpy.ndarray [object PyArrayObject]:
        cdef char *data
        cdef int ndim "nd"
        cdef Py_intptr_t *shape "dimensions"
        cdef Py_intptr_t *strides

    cdef void import_array()

    ctypedef float npy_float64
    ctypedef float npy_float80
    ctypedef int npy_int8
    ctypedef int npy_uint8
    ctypedef int npy_int32
    ctypedef int npy_uint32
    ctypedef int npy_int64
    ctypedef int npy_uint64
    ...


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Nathan Bell-4
In reply to this post by maitai
On Sat, May 17, 2008 at 5:55 PM, Jose Martin <[hidden email]> wrote:

>
> Hi, I'd like to access a C function from python, and the function takes input/output arrays. I'd probably use SWIG to do the interface to the C code. I found 2 options:
> -NumPtr module, to access Numeric arrays as pointers
> http://www.penzilla.net/tutorials/python/numptr/
> - numpy.i, a SWIG interface file for NumPy that defines typemaps
> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/swig/doc/numpy_swig.html
>
> I'm not sure if there is significant differences between the 2 options (besides using either NumPy or Numeric). Does numpy.i interface file use pointers to access NumPy arrays? or does it make a copy of the array to pass it to/from the C function?
>
> I'm new to programming and I'd like to make sure of this. I need to use in C very large arrays frequently, so I want to avoid making copies of it, because speed will be an important factor.
>

I'm fairly confident that numpy.i *will not* copy an array unless it
is necessary to do so.  Nice contiguous arrays should be passed
through efficiently.

I use SWIG extensively in scipy.sparse:
http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools

One nice aspect of SWIG is that you can have it automatically dispatch
the right function.  For instance, if you have
    foo(float * arr)  and foo(double * arr)
then you can use SWIG typemaps pick the correct function to use.

You can also make SWIG upcast to the appropriate types.  For example,
if in Python you passed an int array and a double array to:
   foo(double * arr1, double * arr2)
then you can have SWIG automatically upcast the int array to double
and delete it before returning to Python.

--
Nathan Bell [hidden email]
http://graphics.cs.uiuc.edu/~wnbell/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Nathan Bell-4
In reply to this post by Dag Sverre Seljebotn
On Sat, May 17, 2008 at 6:42 PM, Dag Sverre Seljebotn
<[hidden email]> wrote:
>
> Cython is a different approach from SWIG (see
> http://wiki.cython.org/WrappingCorCpp; in particular SWIG uses more layers
> of indirection).
>

>From the link:
"[SWIG] Can wrap almost any C and C++ code, including templates etc.
Disadvantage is that it produces a C file, this compiles to .so, but
then it also produces a Python wrapper on top of this .so file. So
it's messy and it's slow. Also SWIG is not only targeting Python, but
other languages as well."

I really wish that people didn't spread FUD about SWIG.  For
reference, here's the "messy and slow" Python wrapper for a function
in scipy.sparse:

52 def expandptr(*args):
53    """expandptr(int n_row, int Ap, int Bi)"""
54    return _csr.expandptr(*args)

From:  http://projects.scipy.org/scipy/scipy/browser/trunk/scipy/sparse/sparsetools/csr.py#L52

I understand that scipy.sparse does not use all features of SWIG
(which may produce uglier wrappers), but I think the above case is a
fair example of what to expect when wrapping typical C/C++ libraries.

More disingenuous FUD here: http://www.sagemath.org/doc/html/prog/node36.html

--
Nathan Bell [hidden email]
http://graphics.cs.uiuc.edu/~wnbell/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Robert Kern-2
On Sat, May 17, 2008 at 7:39 PM, Nathan Bell <[hidden email]> wrote:

> More disingenuous FUD here: http://www.sagemath.org/doc/html/prog/node36.html

For the purposes to which SWIG was applied in that case, the findings
are accurate. The wording is overly general, though; it doesn't talk
about other use cases for which the findings are not applicable. When
the amount of computation in C/C++ is fairly small (multiplying 2 GMP
integers), and the Python-level wrapper function needs to be called
many times, then the overhead of the various SWIG layers can become
significant. When the amount of computation in C/C++ is fairly large
(matmult on sparse matrices), the Python function call overhead will
probably be insignificant.

Making a blanket statement either way is incorrect.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Nathan Bell-4
On Sat, May 17, 2008 at 7:48 PM, Robert Kern <[hidden email]> wrote:
> For the purposes to which SWIG was applied in that case, the findings
> are accurate.

IMO it's deliberately misleading.  The following three layers are
spurious and have no analog on the Cython stack:
   Python code to provide a clean interface
   Handcode C++ Integer class
   GMP's C++ Interface

A more honest comparison would list 3 layers for SWIG vs. 2 layers for Cython.

I don't have a hard time believing that Cython is a better choice for
fine-grained access to C/C++ code.  However contrived scenarios like
the above don't inspire my confidence either.  I have yet to see a
benchmark that reveals the claimed benefits of Cython.

--
Nathan Bell [hidden email]
http://graphics.cs.uiuc.edu/~wnbell/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Robert Kern-2
On Sat, May 17, 2008 at 8:13 PM, Nathan Bell <[hidden email]> wrote:
> On Sat, May 17, 2008 at 7:48 PM, Robert Kern <[hidden email]> wrote:
>> For the purposes to which SWIG was applied in that case, the findings
>> are accurate.
>
> IMO it's deliberately misleading.  The following three layers are
> spurious and have no analog on the Cython stack:
>   Python code to provide a clean interface
>   Handcode C++ Integer class
>   GMP's C++ Interface

If you want to end up with a class, that's more or less what you would
do in SWIG. Some bits in Python, because they're dealing with Python
types and exceptions and such. Some bits in C++ because you need to
touch C/C++ structures. Just wrapping the functions in SWIG and then
making the class in pure Python can sometimes work, but often you need
to make a C++ class sitting on top of your library.

Except for possibly the GMP C++ layer (perhaps the handwritten C++
class could just have used the GMP C API, I don't know), they're not
spurious. All of that functionality that was implemented in those
layers were implemented in the single Cython layer.

> A more honest comparison would list 3 layers for SWIG vs. 2 layers for Cython.
>
> I don't have a hard time believing that Cython is a better choice for
> fine-grained access to C/C++ code.  However contrived scenarios like
> the above don't inspire my confidence either.

It was not contrived. It's production code. It was a real and
perfectly valid use case.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Brian Granger-2
In reply to this post by Nathan Bell-4
>> Cython is a different approach from SWIG (see
>> http://wiki.cython.org/WrappingCorCpp; in particular SWIG uses more layers
>> of indirection).
>>
>
> >From the link:
> "[SWIG] Can wrap almost any C and C++ code, including templates etc.
> Disadvantage is that it produces a C file, this compiles to .so, but
> then it also produces a Python wrapper on top of this .so file. So
> it's messy and it's slow. Also SWIG is not only targeting Python, but
> other languages as well."
>
> I really wish that people didn't spread FUD about SWIG.  For
> reference, here's the "messy and slow" Python wrapper for a function
> in scipy.sparse:

I have taken the liberty of making the Cython wiki a little more
specific and fair to SWIG.  Here is the changed text:

SWIG is one of the oldest and most mature methods of wrapping C or C++
code into Python (SWIG works for other target languages as well). It
can wrap almost any C and C++ code, including complex templated C++
code. If you have a large (so that hand wrapping is prohibitive) and
course grained API, SWIG will likely be your best choice. However,
SWIG does have a number or disadvantages compared with Cython. First,
SWIG produces a C file, which gets compiled to a .so, but then it also
produces a Python wrapper on top of this .so file. For fine grained
APIs (where not much is done in each C/C++ call), the overhead of this
additional Python wrapper can be significant. Second, with SWIG, the
Python wrappers are written for you, so if their design is not exactly
what you want, you end up doing more work to create your final Python
API.

Please correct any new errors I have introduced.

Cheers,

Brian
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Nathan Bell-4
On Sat, May 17, 2008 at 9:30 PM, Brian Granger <[hidden email]> wrote:
>
> Please correct any new errors I have introduced.
>

Thanks Brian, I think that's a fair representation.

Minor typo "course grained" -> "coarse-grained"

--
Nathan Bell [hidden email]
http://graphics.cs.uiuc.edu/~wnbell/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Brian Granger-2
In reply to this post by maitai
Jose,

As you can see, people have different preferences for wrapping C/C++
code.  I should also mention that one of the easiest methods if numpy
arrays are involved is ctypes.  numpy arrays already have more-or-less
built-in support for talking to ctypes.  Details are available here:

http://www.scipy.org/Cookbook/Ctypes

The good news is that there are _many_ good options.  I also know that
the numpy.i file for handling numpy arrays in SWIG is of high quality
and has decent documentation.

Brian


On Sat, May 17, 2008 at 4:55 PM, Jose Martin <[hidden email]> wrote:

>
> Hi, I'd like to access a C function from python, and the function takes input/output arrays. I'd probably use SWIG to do the interface to the C code. I found 2 options:
> -NumPtr module, to access Numeric arrays as pointers
> http://www.penzilla.net/tutorials/python/numptr/
> - numpy.i, a SWIG interface file for NumPy that defines typemaps
> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/swig/doc/numpy_swig.html
>
> I'm not sure if there is significant differences between the 2 options (besides using either NumPy or Numeric). Does numpy.i interface file use pointers to access NumPy arrays? or does it make a copy of the array to pass it to/from the C function?
>
> I'm new to programming and I'd like to make sure of this. I need to use in C very large arrays frequently, so I want to avoid making copies of it, because speed will be an important factor.
>
> Thanks in advance!
>
>
>
> _______________________________________________
> Join Excite! - http://www.excite.com
> The most personalized portal on the Web!
>
>
> _______________________________________________
> Numpy-discussion mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

Brian Granger-2
In reply to this post by Nathan Bell-4
On Sat, May 17, 2008 at 8:35 PM, Nathan Bell <[hidden email]> wrote:
> On Sat, May 17, 2008 at 9:30 PM, Brian Granger <[hidden email]> wrote:
>>
>> Please correct any new errors I have introduced.
>>
>
> Thanks Brian, I think that's a fair representation.
>
> Minor typo "course grained" -> "coarse-grained"

Fixed

> --
> Nathan Bell [hidden email]
> http://graphics.cs.uiuc.edu/~wnbell/
> _______________________________________________
> Numpy-discussion mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

wfspotz-sandia
In reply to this post by maitai
Just to make sure the original question gets answered, yes, numpy.i  
avoids copies as much as possible.

A special case is when your C code provides you with a view of its  
internal data and does not require any memory to be allocated by the  
(python) user.  This can be dangerous, but if it is your use case, be  
sure to use the ARGOUTVIEW_* typemaps.

Oh, and Brian's description of SWIG is an eminently fair one.

AND, if NumPtr is only for Numeric, you should know that Numeric is no  
longer developed or supported.

On May 17, 2008, at 4:55 PM, Jose Martin wrote:

> Hi, I'd like to access a C function from python, and the function  
> takes input/output arrays. I'd probably use SWIG to do the interface  
> to the C code. I found 2 options:
> -NumPtr module, to access Numeric arrays as pointers
> http://www.penzilla.net/tutorials/python/numptr/
> - numpy.i, a SWIG interface file for NumPy that defines typemaps
> http://projects.scipy.org/scipy/numpy/browser/trunk/numpy/doc/swig/doc/numpy_swig.html
>
> I'm not sure if there is significant differences between the 2  
> options (besides using either NumPy or Numeric). Does numpy.i  
> interface file use pointers to access NumPy arrays? or does it make  
> a copy of the array to pass it to/from the C function?
>
> I'm new to programming and I'd like to make sure of this. I need to  
> use in C very large arrays frequently, so I want to avoid making  
> copies of it, because speed will be an important factor.
>
> Thanks in advance!

** Bill Spotz                                              **
** Sandia National Laboratories  Voice: (505)845-0170      **
** P.O. Box 5800                 Fax:   (505)284-0154      **
** Albuquerque, NM 87185-0370    Email: [hidden email] **






_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: NumPtr vs NumPy.i to access C

maitai
In reply to this post by maitai

Thanks everyone for all the comments! It helped to understand better the advantages/disadvantages of the various options to interact with C.

Jose.


 --- On Sat 05/17, Bill Spotz < [hidden email] > wrote:

Just to make sure the original question gets answered, yes, numpy.i
avoids copies as much as possible.

A special case is when your C code provides you with a view of its
internal data and does not require any memory to be allocated by the
(python) user. This can be dangerous, but if it is your use case, be
sure to use the ARGOUTVIEW_* typemaps.

Oh, and Brian's description of SWIG is an eminently fair one.

AND, if NumPtr is only for Numeric, you should know that Numeric is no
longer developed or supported.


_______________________________________________
Join Excite! - http://www.excite.com
The most personalized portal on the Web!


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion