a possible way to implement a plogin system

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

a possible way to implement a plogin system

Lisandro Dalcin
David, in order to put clear what I was proposing to you in previous
mail regarding to implementing plugin systems for numpy, please take a
look at the attached tarball.

The plugins are in charge of implementing the action of generic foo()
and bar() functions in C. The example actually implements two
different plugins in Cython. There is also an include file with some
support declarations and some macro definitions wrap_foo and wrap_bar

Next, in mymodule.pyx, I define python side foo() and bar() functions
(calling funtins pointers in a struct via the macros wrap_xxx) and a
use() for selecting the plugin at runtime.

Finally, the test.py script is pure python code showing all this in
action. If you want to give a try, then install Cython, next do
'make', and finally 'python test.py'

Please note that I've implemented this is Cython just for convenience,
it does not actually depend in any special Cython feature, and could
be translated to pure C code with Python C/API calls.

IMHO, this is the easier and cleaner way to deal inside python with
plugins written in low-level C. It does not depend explicitely on
'dlopen' stuff from your side.

Now tell me. What is in your mind that this is not general/robust
enough as to have to explicitely deal with dlopen? It even works on
systems with no dynload, provided that all is put inside the python
executable.



--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: a possible way to implement a plogin system

Lisandro Dalcin
Sorry, I forgot to  attach the code...

On 4/30/08, Lisandro Dalcin <[hidden email]> wrote:

> David, in order to put clear what I was proposing to you in previous
>  mail regarding to implementing plugin systems for numpy, please take a
>  look at the attached tarball.
>
>  The plugins are in charge of implementing the action of generic foo()
>  and bar() functions in C. The example actually implements two
>  different plugins in Cython. There is also an include file with some
>  support declarations and some macro definitions wrap_foo and wrap_bar
>
>  Next, in mymodule.pyx, I define python side foo() and bar() functions
>  (calling funtins pointers in a struct via the macros wrap_xxx) and a
>  use() for selecting the plugin at runtime.
>
>  Finally, the test.py script is pure python code showing all this in
>  action. If you want to give a try, then install Cython, next do
>  'make', and finally 'python test.py'
>
>  Please note that I've implemented this is Cython just for convenience,
>  it does not actually depend in any special Cython feature, and could
>  be translated to pure C code with Python C/API calls.
>
>  IMHO, this is the easier and cleaner way to deal inside python with
>  plugins written in low-level C. It does not depend explicitely on
>  'dlopen' stuff from your side.
>
>  Now tell me. What is in your mind that this is not general/robust
>  enough as to have to explicitely deal with dlopen? It even works on
>  systems with no dynload, provided that all is put inside the python
>  executable.
>
>
>
>
>  --
>  Lisandro Dalcín
>  ---------------
>  Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
>  Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
>  Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
>  PTLC - Güemes 3450, (3000) Santa Fe, Argentina
>  Tel/Fax: +54-(0)342-451.1594
>

--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

stubs.tar (27K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: a possible way to implement a plogin system

David Cournapeau-4
In reply to this post by Lisandro Dalcin
On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote:
> David, in order to put clear what I was proposing to you in previous
> mail regarding to implementing plugin systems for numpy, please take a
> look at the attached tarball.

Thanks for looking at this Lisandro.

The problem I see with the approach of struct vs direct function
pointers is of ABI compatibility: it is easy to mess things up with
structures. There is the advantage of using only one dlsym (or
equivalent) with the struct, which may be much faster than using
hundreds of dlsym for each function. Linux does not seem to have
problems with that, but mac os X for example felt slow when I tried
doing this for several thousand functions. I did not go really far on
that, though. I don't really see why using a struct is cleaner, though.
That's really the same thing (function pointers), in both cases the name
namespace pollution will be the same, and in both cases there will be a
need to generate source code.

>
> IMHO, this is the easier and cleaner way to deal inside python with
> plugins written in low-level C. It does not depend explicitely on
> 'dlopen' stuff from your side.

Concerning the loading mechanism, I don't understand the point of using
PyCObject_Import. By quickly looking at the code of the function, the
plugin needs to be a python module in that case, which is not needed in
our case; I don't like the fact that it is not documented either.

The code for dlopen/dlclose is really short. For each platform, it is
like 50 lines of code, and we have a better control on what we can do
(which may be needed; for example, you want want to load symbols from a
dll A, but the symbols are only in dll B, which dll A depends on; that's
a problem that does not happen for python extensions, I think; I don't
really know to be honest).

The one thing which may have been useful by using python mechanism is
the binary location detection, but that won't be needed in our case,
since the plugins will always be distributed at the same time as
numpy/scipy, hence we will have tight control on their location.

> It even works on
> systems with no dynload, provided that all is put inside the python
> executable.

That won't be needed: the point is to have several implementations with
the same functions. If you link everything statically, you will get name
clashes anyway (if you want to stay portable). On those platforms,
plugins should be disabled when building numpy/scipy.

cheers,

David

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: a possible way to implement a plogin system

Lisandro Dalcin
On 5/1/08, David Cournapeau <[hidden email]> wrote:

> On Wed, 2008-04-30 at 16:44 -0300, Lisandro Dalcin wrote:
>  > David, in order to put clear what I was proposing to you in previous
>  > mail regarding to implementing plugin systems for numpy, please take a
>  > look at the attached tarball.
>
>
> Thanks for looking at this Lisandro.
>
>  The problem I see with the approach of struct vs direct function
>  pointers is of ABI compatibility: it is easy to mess things up with
>  structures.

That's a very valid point.

> There is the advantage of using only one dlsym (or
>  equivalent) with the struct, which may be much faster than using
>  hundreds of dlsym for each function. Linux does not seem to have
>  problems with that, but mac os X for example felt slow when I tried
>  doing this for several thousand functions. I did not go really far on
>  that, though. I don't really see why using a struct is cleaner, though.
>  That's really the same thing (function pointers), in both cases the name
>  namespace pollution will be the same, and in both cases there will be a
>  need to generate source code.

Perhaps cleaner is not the right word. I actually believe that is far
more portable, regarding the oddities of dlopening in different
platforms.

> Concerning the loading mechanism, I don't understand the point of using
>  PyCObject_Import. By quickly looking at the code of the function, the
>  plugin needs to be a python module in that case, which is not needed in
>  our case; I don't like the fact that it is not documented either.

Well, that is the same mechanism NumPy uses for exporting its C API to
extensions modules. Look at generated header  '__multiarray_api.h' in
function '_import_array()' in your numpy installation. This is the
standard, portable, and (as Python doc says) recommended way to expose
C API's from extensions modules.

>  The code for dlopen/dlclose is really short. For each platform, it is
>  like 50 lines of code, and we have a better control on what we can do
>  (which may be needed; for example, you want want to load symbols from a
>  dll A, but the symbols are only in dll B, which dll A depends on; that's
>  a problem that does not happen for python extensions, I think; I don't
>  really know to be honest).

Please note that all my concerns about recommending you not to use
dlopen, is just to save you from future headaches!!!.  See yourself at
row 17 of table here
<http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html> . And that
table does not cover stuff like RTLD_GLOBAL flags to dlopen (or
equivalent).


--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: a possible way to implement a plogin system

cdavid
Lisandro Dalcin wrote:
>
> Perhaps cleaner is not the right word. I actually believe that is far
> more portable, regarding the oddities of dlopening in different
> platforms.

I am sorry, I don't understand why a struct is less sensitive to
platform issues than function pointers when dynamically loading them.
Could you give more details ?

>
> Well, that is the same mechanism NumPy uses for exporting its C API to
> extensions modules. Look at generated header  '__multiarray_api.h' in
> function '_import_array()' in your numpy installation. This is the
> standard, portable, and (as Python doc says) recommended way to expose
> C API's from extensions modules.

Yes, but this is not for exposing C python extensions ! This is for pure
C plugins, and as such, we have different requirements; in particular,
we don't need to hide api (the struct thing is partly useful because iso
C does not have a namespace concept outside the compilation unit with
static). Again, the use cases I have in mind are multiple
implementations of the same core functions,  selected at runtime (sse,
sse2, mmx, etc...). Adding all the python mechanism just to load symbols
does not sound like simplification to me: using the python C API is
heavy, you have to start thinking about all kind of issues (ref
counting, etc...).

>
> Please note that all my concerns about recommending you not to use
> dlopen, is just to save you from future headaches!!!

I implemented and tested the API for mac os x, windows and dlopen which
covers 99 % of our users I think. Note that we can copy the actual code
in python sources if we want to.

> .  See yourself at
> row 17 of table here
> <http://www.fortran-2000.com/ArnaudRecipes/sharedlib.html> . And that
> table does not cover stuff like RTLD_GLOBAL flags to dlopen (or
> equivalent).

I do not pretend to know about all the platforms idiosyncrasies, but I
am aware of the different aspects. In particular, note that most of the
table is related to building, which I already took care of when starting
numscons :) Also, I don't think we will use the system outside a few
majors platforms (typically,  I don'tsee anyone implementing ufunc core
loops in alpha assembly for True64 :) ).

cheers,

David

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion