Controlling the way a module is imported

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Controlling the way a module is imported

cdavid
Hi,
   
    This is not numpy specific, but I need it for numpy/scipy. More
specifically, I would like to be able to have one module interface which
load imp1, imp2, imp3, etc... depending on some options. I see two
obvious solutions: monkey patching, and file configuration, but I try to
avoid the former, and there is no mechanism for the later in scipy. I
read that import hooks could be used for this purpose, but I don't quite
understand the pep:

http://www.python.org/dev/peps/pep-0302/

Has anyone played with it ?

cheers,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

Robert Kern-2
On Sun, May 25, 2008 at 4:28 AM, David Cournapeau
<[hidden email]> wrote:
> Hi,
>
>    This is not numpy specific, but I need it for numpy/scipy. More
> specifically, I would like to be able to have one module interface which
> load imp1, imp2, imp3, etc... depending on some options.

Can you be more specific?

> I see two
> obvious solutions: monkey patching, and file configuration, but I try to
> avoid the former, and there is no mechanism for the later in scipy. I
> read that import hooks could be used for this purpose, but I don't quite
> understand the pep:
>
> http://www.python.org/dev/peps/pep-0302/
>
> Has anyone played with it ?

Avoid custom import hooks at all costs. They are very fragile and
interfere with each other.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

cdavid
Robert Kern wrote:
>
> Can you be more specific?

Sure: in my branch to refactor fftpack,  every non default backend (that
is everything but fftpack) is a separate python module, which implements
some fft functions, and is 'importable'.

So in scipy.fftpack, I have a function which:
    - tries to find one of the backend from a list (by using __import__)
    - if no backend is found, just use the default backend
    - if one is found, pull the functions from the found backend, and
use the default backend as a fallback for non available functions (each
backend can implement a different subset of fftpack).

This is done in the following prototype:

http://projects.scipy.org/scipy/scipy/browser/branches/refactor_fft/scipy/fftpack/common.py

I would like to add the possibility to  control which backend is used:
the problem is how to communicate the name of the backend to
scipy.fftpack in a persistent way (such as if you want to use mkl, it
will remember it at the next python session), and in a dynamic way (such
as you can decide which one to use before importing scipy.fftpack, or
even changing the backend on the fly).

thanks,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

Timmie
Administrator
In reply to this post by cdavid
Hello,
I posted a similar question to Tutor:

loading modules only when needed and PEP 008 -
http://article.gmane.org/gmane.comp.python.tutor/46969

I wanted to use this for thridparty libs that I deliver with my application.

Would like to see how you gonna solve it.

Kind regards,
Timmie

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

Lisandro Dalcin
In reply to this post by cdavid
David, I've implemented something like this for petsc4py.

Basically, I want to be able to build extension modules accessing
PETSc for different configuration options (for example, optimized
versus debug), when using PETSc, it is normal to select the build
variant by a $PETSC_ARCH environ var.

All this is implemented in pure Python code, and tanking advantage of
the built-in 'imp' module. I believe it is robust enough, I had never
received any report of this causing trouble from petsc4py users out
there.

If you think this fits your needs, then I'll help you. Buth then I'll
need to know a bit more about the directory tree structure of your
package and your extension modules.



On 5/25/08, David Cournapeau <[hidden email]> wrote:

> Hi,
>
>     This is not numpy specific, but I need it for numpy/scipy. More
>  specifically, I would like to be able to have one module interface which
>  load imp1, imp2, imp3, etc... depending on some options. I see two
>  obvious solutions: monkey patching, and file configuration, but I try to
>  avoid the former, and there is no mechanism for the later in scipy. I
>  read that import hooks could be used for this purpose, but I don't quite
>  understand the pep:
>
>  http://www.python.org/dev/peps/pep-0302/
>
>  Has anyone played with it ?
>
>  cheers,
>
>  David
>  _______________________________________________
>  Numpy-discussion mailing list
>  [hidden email]
>  http://projects.scipy.org/mailman/listinfo/numpy-discussion
>


--
Lisandro Dalcín
---------------
Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC)
Instituto de Desarrollo Tecnológico para la Industria Química (INTEC)
Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET)
PTLC - Güemes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

cdavid
Lisandro Dalcin wrote:
> David, I've implemented something like this for petsc4py.
>
> Basically, I want to be able to build extension modules accessing
> PETSc for different configuration options (for example, optimized
> versus debug), when using PETSc, it is normal to select the build
> variant by a $PETSC_ARCH environ var.
>  

Yes, that's another solution I forgot to mention.

> If you think this fits your needs, then I'll help you. Buth then I'll
> need to know a bit more about the directory tree structure of your
> package and your extension modules.
>  

Thanks, but I already have the code to dynamically load a module from a
string. My problem is how to control it, and how we want to do this kind
of things. Up to know, scipy have never depended on env variables or
configuration files, so I don't want to introduce this kind of things
without discussing it first.

cheers,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

Robert Kern-2
On Mon, May 26, 2008 at 11:01 PM, David Cournapeau
<[hidden email]> wrote:

> Lisandro Dalcin wrote:
>> David, I've implemented something like this for petsc4py.
>>
>> Basically, I want to be able to build extension modules accessing
>> PETSc for different configuration options (for example, optimized
>> versus debug), when using PETSc, it is normal to select the build
>> variant by a $PETSC_ARCH environ var.
>
> Yes, that's another solution I forgot to mention.
>
>> If you think this fits your needs, then I'll help you. Buth then I'll
>> need to know a bit more about the directory tree structure of your
>> package and your extension modules.
>
> Thanks, but I already have the code to dynamically load a module from a
> string. My problem is how to control it, and how we want to do this kind
> of things. Up to know, scipy have never depended on env variables or
> configuration files, so I don't want to introduce this kind of things
> without discussing it first.

There are actually a couple of places, like scipy.misc.imshow(). I
tend to dislike libraries automatically reading configuration
(including environment variables). However, imshow() is not really a
library function so much as a function for the interactive
interpreter, so I don't mind so much.

I would definitely *not* use environment variables as the sole control
of FFT backend selection. If I have to restart my IPython session just
because I forgot to set the right environment variable, I will be very
unhappy.

For FFTs, I would probably keep the backend functions in a
module-level list. On import, the package would try to import the
backends it knows could be bundled in scipy.fftpack and insert them
into the list. Probably, these would just be the optimized versions;
the default FFTPACK versions would be kept separate. If an import
fails, it would be a good idea to format the traceback into a string
and store it in an accessible location for debugging unintentional
ImportErrors. Each API function (e.g. rfft()) would check its list for
an optimized implementation and use it or else fall back to the
default. Each implementation module (e.g. scipy.fftpack._fftw3, or
whatever you have named them; I haven't reviewed your code, yet) would
have an explicit registration function that puts its implementations
(all or a subset, e.g. _fftw3.register() or _fftw3.register('fft',
'rfft')) at the top of the lists. One needs to think about how to
handle power-of-two only implementations. This is probably the only
real special case we have to handle, so we can probably do it
explicitly instead of coming up with an overly generic solution. I
wouldn't bother trying to persist this information. If anyone wants to
persist their preferences, then can write a utility function that does
exactly what they need.

That's the five-minute overview of what I would try.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

Jon Wright
In reply to this post by cdavid
David Cournapeau wrote:
> Thanks, but I already have the code to dynamically load a module from a
> string. My problem is how to control it, and how we want to do this kind
> of things. Up to know, scipy have never depended on env variables or
> configuration files, so I don't want to introduce this kind of things
> without discussing it first.
>  
Both matplotlib and ipython are using rc files already. For your fftpack
work, it would be great to be able to compare different backends within
the same session, both for benchmarking and accuracy checking. In that
case the user would probably want to control what is loaded at runtime.

Thanks!

Jon


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Controlling the way a module is imported

David Cournapeau
In reply to this post by Robert Kern-2
On Tue, May 27, 2008 at 3:14 PM, Robert Kern <[hidden email]> wrote:

>
> I would definitely *not* use environment variables as the sole control
> of FFT backend selection. If I have to restart my IPython session just
> because I forgot to set the right environment variable, I will be very
> unhappy.

Oh yes, I certainly agree it should not be the sole way to control it.
It could, maybe, used to force it, and even then, I am not sure it is
that useful.

>
> For FFTs, I would probably keep the backend functions in a
> module-level list. On import, the package would try to import the
> backends it knows could be bundled in scipy.fftpack and insert them
> into the list. Probably, these would just be the optimized versions;
> the default FFTPACK versions would be kept separate. If an import
> fails, it would be a good idea to format the traceback into a string
> and store it in an accessible location for debugging unintentional
> ImportErrors. Each API function (e.g. rfft()) would check its list for
> an optimized implementation and use it or else fall back to the
> default. Each implementation module (e.g. scipy.fftpack._fftw3, or
> whatever you have named them; I haven't reviewed your code, yet) would
> have an explicit registration function that puts its implementations
> (all or a subset, e.g. _fftw3.register() or _fftw3.register('fft',
> 'rfft')) at the top of the lists

Registration aside, that's more or less what I have done so far.

> handle power-of-two only implementations. This is probably the only
> real special case we have to handle, so we can probably do it
> explicitly instead of coming up with an overly generic solution.

I thought about changing the C Api to return an error code for invalid
input, and writing a python wrapper which calls the default backend
when an error is returned. This way, I can treat any kind of error
without too much burden.

cheers,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion