numpy setup.py too restrictive, prevents use of fblas with cblas

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

numpy setup.py too restrictive, prevents use of fblas with cblas

George Nurser-2
Apologies for coming out of the woodwork so late here..

For blas/atlas etc in numpy & scipy on an opteron I use the AMD
libraries (which only have fblas) together with cblas. This works very
well.

Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py  has:

            if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]):
                return None # dotblas needs ATLAS, Fortran compiled
blas will not be sufficient

To get my AMD fblas/cblas approach to work, I have to comment out
these two lines.
Can this restriction preventing use of AMD fblas be removed for v1.1?

Should I file a bug here?

Regards, George Nurser.
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Travis Oliphant-5
George Nurser wrote:

> Apologies for coming out of the woodwork so late here..
>
> For blas/atlas etc in numpy & scipy on an opteron I use the AMD
> libraries (which only have fblas) together with cblas. This works very
> well.
>
> Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py  has:
>
>             if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]):
>                 return None # dotblas needs ATLAS, Fortran compiled
> blas will not be sufficient
>
> To get my AMD fblas/cblas approach to work, I have to comment out
> these two lines.
> Can this restriction preventing use of AMD fblas be removed for v1.1?
>  
> Should I file a bug here?
>  
Yes, you should put a ticket together.   The more information you can
provide, the more likely someone will act on the ticket.  Some kind of
info on why you had to comment out those two lines would be helpful.

-Travis

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Robert Kern-2
On Wed, Apr 16, 2008 at 9:11 AM, Travis E. Oliphant
<[hidden email]> wrote:

> George Nurser wrote:
>  > Apologies for coming out of the woodwork so late here..
>  >
>  > For blas/atlas etc in numpy & scipy on an opteron I use the AMD
>  > libraries (which only have fblas) together with cblas. This works very
>  > well.
>  >
>  > Current-ish SVN (v4779) in line 295-296 of numpy/core/setup.py  has:
>  >
>  >             if ('NO_ATLAS_INFO',1) in blas_info.get('define_macros',[]):
>  >                 return None # dotblas needs ATLAS, Fortran compiled
>  > blas will not be sufficient
>  >
>  > To get my AMD fblas/cblas approach to work, I have to comment out
>  > these two lines.
>  > Can this restriction preventing use of AMD fblas be removed for v1.1?
>  >
>  > Should I file a bug here?
>  >
>  Yes, you should put a ticket together.   The more information you can
>  provide, the more likely someone will act on the ticket.  Some kind of
>  info on why you had to comment out those two lines would be helpful.

I am guessing that he built the cblas interface from here:

  http://www.netlib.org/blas/blast-forum/cblas.tgz

Note that this is same C interface provided by ATLAS, the MKL, and
GOTO and is *not* the version simply converted from the FORTRAN using
f2c. It calls the (hopefully accelerated) FORTRAN subroutine where
possible after manipulating the arguments to tell the subroutine that
the matrices are transposed if row-major.

The correct fix needs to be more sophisticated than removing those two
lines. We need to recognize the MKL and the GOTO BLAS and allow them,
too. It might also be worth including the appropriate subset of the
cblas code provided in the tarball such that we can use any
accelerated FORTRAN BLAS without the standard cblas interface. Then
George wouldn't have the build the cblas library himself, either.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Stéfan van der Walt
Hi Robert

On 16/04/2008, Robert Kern <[hidden email]> wrote:
>  The correct fix needs to be more sophisticated than removing those two
>  lines. We need to recognize the MKL and the GOTO BLAS and allow them,
>  too. It might also be worth including the appropriate subset of the
>  cblas code provided in the tarball such that we can use any
>  accelerated FORTRAN BLAS without the standard cblas interface. Then
>  George wouldn't have the build the cblas library himself, either.

The inclusion of those cblas routines sounds like a good idea.  Could
you describe which we need, and what would be required to get this
done?

Cheers
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Robert Kern-2
On Wed, Apr 16, 2008 at 3:37 PM, Stéfan van der Walt <[hidden email]> wrote:

> Hi Robert
>
>  On 16/04/2008, Robert Kern <[hidden email]> wrote:
>  >  The correct fix needs to be more sophisticated than removing those two
>  >  lines. We need to recognize the MKL and the GOTO BLAS and allow them,
>  >  too. It might also be worth including the appropriate subset of the
>  >  cblas code provided in the tarball such that we can use any
>  >  accelerated FORTRAN BLAS without the standard cblas interface. Then
>  >  George wouldn't have the build the cblas library himself, either.
>
>  The inclusion of those cblas routines sounds like a good idea.  Could
>  you describe which we need, and what would be required to get this
>  done?

Basically all of the files defining functions used in _dotblas.c and
their dependencies; grepping for "cblas_" should find them all. The
tricky part, naturally, is the build configuration. We need to
recognize when we have an accelerated fblas and cblas (do not use our
own cblas routines), when we have an accelerated fblas only (do use
our own cblas routines), and when we don't have anything accelerated
(do not build dotblas at all). I don't have a particularly clear idea
on how to do that cleanly.

One potential problem that I've just noticed is that there are some
Fortran files in the cblas. For the most part, these seem to be used
to convert Fortran function calls which return complex scalars to
Fortran subroutines which can be easily interfaced to C. For example
cblas_cdotu_sub.c requires cdotc.f. dotblas uses four such functions:
cblas_[cz]dot[cu]_sub(). We would have to find another way to do
these; possibly, we can just apply f2c to the Fortran files.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Andreas Klöckner-3
In reply to this post by Stéfan van der Walt
On Mittwoch 16 April 2008, Stéfan van der Walt wrote:
> The inclusion of those cblas routines sounds like a good idea.  Could
> you describe which we need, and what would be required to get this
> done?

Suppose cblas gets included in numpy, but for some reason someone decides to
link another copy of cblas with their (separate) extension. Can we be certain
that this does not lead to crashes on any platform supported by numpy?

Andreas

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

Robert Kern-2
On Wed, Apr 16, 2008 at 4:56 PM, Andreas Klöckner
<[hidden email]> wrote:
> On Mittwoch 16 April 2008, Stéfan van der Walt wrote:
>  > The inclusion of those cblas routines sounds like a good idea.  Could
>  > you describe which we need, and what would be required to get this
>  > done?
>
>  Suppose cblas gets included in numpy, but for some reason someone decides to
>  link another copy of cblas with their (separate) extension. Can we be certain
>  that this does not lead to crashes on any platform supported by numpy?

I imagine we would have the same problem with the f2c'ed default
LAPACK routines we include to support numpy.linalg. I haven't heard of
this ever causing a problem.

However, if there is a conflict, we can get around it with clever
#defines such that the actual symbols in the extension module would
be, say, numpy_cblas_dgemm, etc.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: numpy setup.py too restrictive, prevents use of fblas with cblas

George Nurser-2
In reply to this post by Robert Kern-2
> I am guessing that he built the cblas interface from here:
>
>   http://www.netlib.org/blas/blast-forum/cblas.tgz
>
>  Note that this is same C interface provided by ATLAS, the MKL, and
>  GOTO and is *not* the version simply converted from the FORTRAN using
>  f2c. It calls the (hopefully accelerated) FORTRAN subroutine where
>  possible after manipulating the arguments to tell the subroutine that
>  the matrices are transposed if row-major.
>
>  The correct fix needs to be more sophisticated than removing those two
>  lines. We need to recognize the MKL and the GOTO BLAS and allow them,
>  too. It might also be worth including the appropriate subset of the
>  cblas code provided in the tarball such that we can use any
>  accelerated FORTRAN BLAS without the standard cblas interface. Then
>  George wouldn't have the build the cblas library himself, either.

I have filed a ticket on this.

Yes, it is the cblas interface from
http://www.netlib.org/blas/blast-forum/cblas.tgz.
Not having to build the interface would indeed make it a whole lot easier.

George Nurser.
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion