failed to add routine to the core module

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

failed to add routine to the core module

marc

Dear Numpy developers,

I'm trying to add a routine to calculate the sum of a product of two arrays (a dot product). But that would not increase the memory (from what I saw np.dot is increasing the memory while it should not be necessary). The idea is to avoid the use of the temporary array in the calculation of the variance ( numpy/numpy/core/_methods.py line 112).

The routine that I want to implement look like this in python,

arr = np.random.rand(100000)
mean = arr.mean()
var = 0.0
for ai in arr: var += (ai-mean)**2

I would like to implement it in the umath module. As a first step, I tried to reproduce the divmod function of umath, but I did not manage to do it, you can find my fork here (the branch with the changes is call looking_around). During compilation I get the following error,

gcc: numpy/core/src/multiarray/number.c
In file included from numpy/core/src/multiarray/number.c:17:0: numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’:
numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka struct <anonymous>}’ has no member named ‘nb_sum_multiply’ (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
                            ^
numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
        ^
numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply, array_sum_multiply);

Sorry if my question seems basic, but I'm new in Numpy development.
Any help?

Thank you in advance,

Marc Barbry

PS: I opened an issues as well on the github repository
https://github.com/numpy/numpy/issues/9130 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: failed to add routine to the core module

Marten van Kerkwijk
Hi Marc,

ufuncs are quite tricky to compile. Part of your problem is that, I
think, you started a bit too high up: `divmod` is also a binary
operation, so that part you do not need at all. It may be an idea to
start instead with a PR that implemented a new ufunc, e.g.,
https://github.com/numpy/numpy/pull/8795, so that you can see what is
involved.

All the best,

Marten



On Thu, May 18, 2017 at 9:04 AM, marc <[hidden email]> wrote:

> Dear Numpy developers,
>
> I'm trying to add a routine to calculate the sum of a product of two arrays
> (a dot product). But that would not increase the memory (from what I saw
> np.dot is increasing the memory while it should not be necessary). The idea
> is to avoid the use of the temporary array in the calculation of the
> variance ( numpy/numpy/core/_methods.py line 112).
>
> The routine that I want to implement look like this in python,
>
> arr = np.random.rand(100000)
> mean = arr.mean()
> var = 0.0
> for ai in arr: var += (ai-mean)**2
>
> I would like to implement it in the umath module. As a first step, I tried
> to reproduce the divmod function of umath, but I did not manage to do it,
> you can find my fork here (the branch with the changes is call
> looking_around). During compilation I get the following error,
>
> gcc: numpy/core/src/multiarray/number.c
> In file included from numpy/core/src/multiarray/number.c:17:0:
> numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’:
> numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka
> struct <anonymous>}’ has no member named ‘nb_sum_multiply’
> (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
>                             ^
> numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro
> ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
>         ^
> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
> ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply,
> array_sum_multiply);
>
> Sorry if my question seems basic, but I'm new in Numpy development.
> Any help?
>
> Thank you in advance,
>
> Marc Barbry
>
> PS: I opened an issues as well on the github repository
> https://github.com/numpy/numpy/issues/9130
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: failed to add routine to the core module

marc
Hello Marten,

Thank you for your help, effectively, the example that you propose is
much easier to imitate, I can now continue further.

Thanks,
Marc

On 05/18/2017 04:32 PM, Marten van Kerkwijk wrote:

> Hi Marc,
>
> ufuncs are quite tricky to compile. Part of your problem is that, I
> think, you started a bit too high up: `divmod` is also a binary
> operation, so that part you do not need at all. It may be an idea to
> start instead with a PR that implemented a new ufunc, e.g.,
> https://github.com/numpy/numpy/pull/8795, so that you can see what is
> involved.
>
> All the best,
>
> Marten
>
>
>
> On Thu, May 18, 2017 at 9:04 AM, marc <[hidden email]> wrote:
>> Dear Numpy developers,
>>
>> I'm trying to add a routine to calculate the sum of a product of two arrays
>> (a dot product). But that would not increase the memory (from what I saw
>> np.dot is increasing the memory while it should not be necessary). The idea
>> is to avoid the use of the temporary array in the calculation of the
>> variance ( numpy/numpy/core/_methods.py line 112).
>>
>> The routine that I want to implement look like this in python,
>>
>> arr = np.random.rand(100000)
>> mean = arr.mean()
>> var = 0.0
>> for ai in arr: var += (ai-mean)**2
>>
>> I would like to implement it in the umath module. As a first step, I tried
>> to reproduce the divmod function of umath, but I did not manage to do it,
>> you can find my fork here (the branch with the changes is call
>> looking_around). During compilation I get the following error,
>>
>> gcc: numpy/core/src/multiarray/number.c
>> In file included from numpy/core/src/multiarray/number.c:17:0:
>> numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’:
>> numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka
>> struct <anonymous>}’ has no member named ‘nb_sum_multiply’
>> (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func))
>>                              ^
>> numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro
>> ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \
>>          ^
>> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
>> ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply,
>> array_sum_multiply);
>>
>> Sorry if my question seems basic, but I'm new in Numpy development.
>> Any help?
>>
>> Thank you in advance,
>>
>> Marc Barbry
>>
>> PS: I opened an issues as well on the github repository
>> https://github.com/numpy/numpy/issues/9130
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: failed to add routine to the core module

Sebastian Berg
In reply to this post by marc
On Thu, 2017-05-18 at 15:04 +0200, marc wrote:
> Dear Numpy developers,
> I'm trying to add a routine to calculate the sum of a product of two
> arrays (a dot product). But that would not increase the memory (from
> what I saw np.dot is increasing the memory while it should not be
> necessary). The idea is to avoid the use of the temporary array in
> the calculation of the variance ( numpy/numpy/core/_methods.py line
> 112).

np.dot should only increase memory in some cases (such as non-
contiguous arrays) and be much faster in most cases (unless e.g. you do
not have a BLAS compatible type). You might also want to check out
np.einsum, which is pretty slick and can handle these kind of
operations as well. Note that `np.dot` calls into BLAS so that it is in
general much faster then np.einsum.

- Sebastian

> The routine that I want to implement look like this in python,
> arr = np.random.rand(100000)
> mean = arr.mean()
> var = 0.0
> for ai in arr: var += (ai-mean)**2
> I would like to implement it in the umath module. As a first step, I
> tried to reproduce the divmod function of umath, but I did not manage
> to do it, you can find my fork here (the branch with the changes is
> call looking_around). During compilation I get the following error,
> gcc: numpy/core/src/multiarray/number.c 
> In file included from numpy/core/src/multiarray/number.c:17:0:
> numpy/core/src/multiarray/number.c: In function
> ‘array_sum_multiply’: 
> numpy/core/src/private/binop_override.h:176:39: error:
> ‘PyNumberMethods {aka struct <anonymous>}’ has no member named
> ‘nb_sum_multiply’ (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) !=
> (void*)(test_func)) 
>                             ^ 
> numpy/core/src/private/binop_override.h:180:13: note: in expansion of
> macro ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr,
> test_func) && \
>         ^ 
> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro
> ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2,
> nb_sum_multiply, array_sum_multiply);
> Sorry if my question seems basic, but I'm new in Numpy development.
> Any help?
> Thank you in advance,
> Marc Barbry
>
> PS: I opened an issues as well on the github repository
> https://github.com/numpy/numpy/issues/9130 
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (817 bytes) Download Attachment
Loading...