Type declaration to include all valid numerical NumPy types for Cython

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Type declaration to include all valid numerical NumPy types for Cython

Ilhan Polat
Hi all,

As you might have seen my recent mails in Cython list, I'm trying to cook up an input validator for the linalg.solve() function. The machinery of SciPy linalg is as follows:

Some input comes in passes through np.asarray() then depending on the resulting dtype of the numpy array we choose a LAPACK flavor (s,d,c,z) and off it goes through f2py to lalaland and comes back with some result.

For the backslash polyalgorithm I need the arrays to be contiguous (C- or F- doesn't matter) and any of the four (possibly via making new copies) float, double, float complex, double complex after the intake because we are using wrapped fortran code (LAPACK) in SciPy. So my difficulty is how to type such function input, say,

ctypedef fused numeric_numpy_t:
    bint
    cnp.npy_bool
    cnp.int_t      
    cnp.intp_t      
    cnp.int8_t      
    cnp.int16_t    
    cnp.int32_t    
    cnp.int64_t    
    cnp.uint8_t    
    cnp.uint16_t    
    cnp.uint32_t    
    cnp.uint64_t
    cnp.float32_t  
    cnp.float64_t  
    cnp.complex64_t
    cnp.complex128_t


Is this acceptable or something else needs to be used? Then there is the storyof np.complex256 and mysterious np.float16. Then there is the Linux vs Windows platform dependence issue and possibly some more that I can't comprehend. Then there are datetime, str, unicode etc. that need to be rejected. So this is quickly getting out of hand for my small brain.

To be honest, I am a bit running out of steam working with this issue even though I managed to finish the actual difficult algorithmic part but got stuck here. I am quite surprised how fantastically complicated and confusing both NumPy and Cython docs about this stuff. Shouldn't we keep a generic fused type for such usage? Or maybe there already exists but I don't know and would be really grateful for pointers.

Here I wrote a dummy typed Cython function just for type checking:

cpdef inline bint ncc( numeric_numpy_t[:, :] a):
    print(a.is_f_contig())
    print(a.is_c_contig())

    return a.is_f_contig() or a.is_c_contig()

And this is a dummy loop (with aliases) just to check whether fused type is working or not (on windows I couldn't make it work for float16).

for x in (np.uint, np.uintc, np.uintp, np.uint0, np.uint8, np.uint16, np.uint32,
          np.uint64, np.int, np.intc, np.intp, np.int0, np.int8, np.int16,
          np.int32,np.int64, np.float, np.float32, np.float64, np.float_,
          np.complex, np.complex64, np.complex128, np.complex_):
    print(x)
    C = np.arange(25., dtype=x).reshape(5, 5)
    ncc(C)


Thanks in advance,
ilhan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Type declaration to include all valid numerical NumPy types for Cython

Eric Moore-11
If that is really all you need, then the version in python is: 

def convert_one(a):
    """
    Converts input with arbitrary layout and dtype to a blas/lapack
    compatible dtype with either C or F order.  Acceptable objects are passed
    through without making copies.
    """

    a_arr = np.asarray(a)
    dtype = np.result_type(a_arr, 1.0)

    # need to handle these separately
    if dtype == np.longdouble:
        dtype = np.dtype('d')
    elif dtype == np.clongdouble:
        dtype = np.dtype('D')
    elif dtype == np.float16:
        dtype = np.dtype('f')

    # explicitly force a copy if a_arr isn't one segment
    return np.array(a_arr, dtype, copy=not a_arr.flags.forc, order='K')

In Cython, you could just run exactly this code and it's probably fine.  The could also be rewritten using the C calls if you really wanted. 

You need to either provide your own or use a casting table and the copy / conversion routines from somewhere.  Cython, to my knowledge, doesn't provide these things, but Numpy does.

Eric

On Sun, Aug 9, 2020 at 6:16 PM Ilhan Polat <[hidden email]> wrote:
Hi all,

As you might have seen my recent mails in Cython list, I'm trying to cook up an input validator for the linalg.solve() function. The machinery of SciPy linalg is as follows:

Some input comes in passes through np.asarray() then depending on the resulting dtype of the numpy array we choose a LAPACK flavor (s,d,c,z) and off it goes through f2py to lalaland and comes back with some result.

For the backslash polyalgorithm I need the arrays to be contiguous (C- or F- doesn't matter) and any of the four (possibly via making new copies) float, double, float complex, double complex after the intake because we are using wrapped fortran code (LAPACK) in SciPy. So my difficulty is how to type such function input, say,

ctypedef fused numeric_numpy_t:
    bint
    cnp.npy_bool
    cnp.int_t      
    cnp.intp_t      
    cnp.int8_t      
    cnp.int16_t    
    cnp.int32_t    
    cnp.int64_t    
    cnp.uint8_t    
    cnp.uint16_t    
    cnp.uint32_t    
    cnp.uint64_t
    cnp.float32_t  
    cnp.float64_t  
    cnp.complex64_t
    cnp.complex128_t


Is this acceptable or something else needs to be used? Then there is the storyof np.complex256 and mysterious np.float16. Then there is the Linux vs Windows platform dependence issue and possibly some more that I can't comprehend. Then there are datetime, str, unicode etc. that need to be rejected. So this is quickly getting out of hand for my small brain.

To be honest, I am a bit running out of steam working with this issue even though I managed to finish the actual difficult algorithmic part but got stuck here. I am quite surprised how fantastically complicated and confusing both NumPy and Cython docs about this stuff. Shouldn't we keep a generic fused type for such usage? Or maybe there already exists but I don't know and would be really grateful for pointers.

Here I wrote a dummy typed Cython function just for type checking:

cpdef inline bint ncc( numeric_numpy_t[:, :] a):
    print(a.is_f_contig())
    print(a.is_c_contig())

    return a.is_f_contig() or a.is_c_contig()

And this is a dummy loop (with aliases) just to check whether fused type is working or not (on windows I couldn't make it work for float16).

for x in (np.uint, np.uintc, np.uintp, np.uint0, np.uint8, np.uint16, np.uint32,
          np.uint64, np.int, np.intc, np.intp, np.int0, np.int8, np.int16,
          np.int32,np.int64, np.float, np.float32, np.float64, np.float_,
          np.complex, np.complex64, np.complex128, np.complex_):
    print(x)
    C = np.arange(25., dtype=x).reshape(5, 5)
    ncc(C)


Thanks in advance,
ilhan

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Type declaration to include all valid numerical NumPy types for Cython

Ilhan Polat
Yes it seems like I don't have any other option anyways. There is a bit of a penalty but I guess this should do the trick.

Thanks Eric (again! :D)

On Mon, Aug 10, 2020 at 2:51 AM Eric Moore <[hidden email]> wrote:
If that is really all you need, then the version in python is: 

def convert_one(a):
    """
    Converts input with arbitrary layout and dtype to a blas/lapack
    compatible dtype with either C or F order.  Acceptable objects are passed
    through without making copies.
    """

    a_arr = np.asarray(a)
    dtype = np.result_type(a_arr, 1.0)

    # need to handle these separately
    if dtype == np.longdouble:
        dtype = np.dtype('d')
    elif dtype == np.clongdouble:
        dtype = np.dtype('D')
    elif dtype == np.float16:
        dtype = np.dtype('f')

    # explicitly force a copy if a_arr isn't one segment
    return np.array(a_arr, dtype, copy=not a_arr.flags.forc, order='K')

In Cython, you could just run exactly this code and it's probably fine.  The could also be rewritten using the C calls if you really wanted. 

You need to either provide your own or use a casting table and the copy / conversion routines from somewhere.  Cython, to my knowledge, doesn't provide these things, but Numpy does.

Eric

On Sun, Aug 9, 2020 at 6:16 PM Ilhan Polat <[hidden email]> wrote:
Hi all,

As you might have seen my recent mails in Cython list, I'm trying to cook up an input validator for the linalg.solve() function. The machinery of SciPy linalg is as follows:

Some input comes in passes through np.asarray() then depending on the resulting dtype of the numpy array we choose a LAPACK flavor (s,d,c,z) and off it goes through f2py to lalaland and comes back with some result.

For the backslash polyalgorithm I need the arrays to be contiguous (C- or F- doesn't matter) and any of the four (possibly via making new copies) float, double, float complex, double complex after the intake because we are using wrapped fortran code (LAPACK) in SciPy. So my difficulty is how to type such function input, say,

ctypedef fused numeric_numpy_t:
    bint
    cnp.npy_bool
    cnp.int_t      
    cnp.intp_t      
    cnp.int8_t      
    cnp.int16_t    
    cnp.int32_t    
    cnp.int64_t    
    cnp.uint8_t    
    cnp.uint16_t    
    cnp.uint32_t    
    cnp.uint64_t
    cnp.float32_t  
    cnp.float64_t  
    cnp.complex64_t
    cnp.complex128_t


Is this acceptable or something else needs to be used? Then there is the storyof np.complex256 and mysterious np.float16. Then there is the Linux vs Windows platform dependence issue and possibly some more that I can't comprehend. Then there are datetime, str, unicode etc. that need to be rejected. So this is quickly getting out of hand for my small brain.

To be honest, I am a bit running out of steam working with this issue even though I managed to finish the actual difficult algorithmic part but got stuck here. I am quite surprised how fantastically complicated and confusing both NumPy and Cython docs about this stuff. Shouldn't we keep a generic fused type for such usage? Or maybe there already exists but I don't know and would be really grateful for pointers.

Here I wrote a dummy typed Cython function just for type checking:

cpdef inline bint ncc( numeric_numpy_t[:, :] a):
    print(a.is_f_contig())
    print(a.is_c_contig())

    return a.is_f_contig() or a.is_c_contig()

And this is a dummy loop (with aliases) just to check whether fused type is working or not (on windows I couldn't make it work for float16).

for x in (np.uint, np.uintc, np.uintp, np.uint0, np.uint8, np.uint16, np.uint32,
          np.uint64, np.int, np.intc, np.intp, np.int0, np.int8, np.int16,
          np.int32,np.int64, np.float, np.float32, np.float64, np.float_,
          np.complex, np.complex64, np.complex128, np.complex_):
    print(x)
    C = np.arange(25., dtype=x).reshape(5, 5)
    ncc(C)


Thanks in advance,
ilhan

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion