Dear Numpy developers, I'm trying to add a routine to calculate the sum of a product of two arrays (a dot product). But that would not increase the memory (from what I saw np.dot is increasing the memory while it should not be necessary). The idea is to avoid the use of the temporary array in the calculation of the variance ( numpy/numpy/core/_methods.py line 112). The routine that I want to implement look like this in python,
I would like to implement it in the umath module. As a first step, I tried to reproduce the divmod function of umath, but I did not manage to do it, you can find my fork here (the branch with the changes is call looking_around). During compilation I get the following error, gcc: numpy/core/src/multiarray/number.c Sorry if my question seems basic, but I'm new in Numpy
development. Thank you in advance, PS: I opened an issues as well on the github repository https://github.com/numpy/numpy/issues/9130 _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hi Marc,
ufuncs are quite tricky to compile. Part of your problem is that, I think, you started a bit too high up: `divmod` is also a binary operation, so that part you do not need at all. It may be an idea to start instead with a PR that implemented a new ufunc, e.g., https://github.com/numpy/numpy/pull/8795, so that you can see what is involved. All the best, Marten On Thu, May 18, 2017 at 9:04 AM, marc <[hidden email]> wrote: > Dear Numpy developers, > > I'm trying to add a routine to calculate the sum of a product of two arrays > (a dot product). But that would not increase the memory (from what I saw > np.dot is increasing the memory while it should not be necessary). The idea > is to avoid the use of the temporary array in the calculation of the > variance ( numpy/numpy/core/_methods.py line 112). > > The routine that I want to implement look like this in python, > > arr = np.random.rand(100000) > mean = arr.mean() > var = 0.0 > for ai in arr: var += (ai-mean)**2 > > I would like to implement it in the umath module. As a first step, I tried > to reproduce the divmod function of umath, but I did not manage to do it, > you can find my fork here (the branch with the changes is call > looking_around). During compilation I get the following error, > > gcc: numpy/core/src/multiarray/number.c > In file included from numpy/core/src/multiarray/number.c:17:0: > numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’: > numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka > struct <anonymous>}’ has no member named ‘nb_sum_multiply’ > (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func)) > ^ > numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro > ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \ > ^ > numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro > ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply, > array_sum_multiply); > > Sorry if my question seems basic, but I'm new in Numpy development. > Any help? > > Thank you in advance, > > Marc Barbry > > PS: I opened an issues as well on the github repository > https://github.com/numpy/numpy/issues/9130 > > _______________________________________________ > NumPy-Discussion mailing list > [hidden email] > https://mail.python.org/mailman/listinfo/numpy-discussion > NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
Hello Marten,
Thank you for your help, effectively, the example that you propose is much easier to imitate, I can now continue further. Thanks, Marc On 05/18/2017 04:32 PM, Marten van Kerkwijk wrote: > Hi Marc, > > ufuncs are quite tricky to compile. Part of your problem is that, I > think, you started a bit too high up: `divmod` is also a binary > operation, so that part you do not need at all. It may be an idea to > start instead with a PR that implemented a new ufunc, e.g., > https://github.com/numpy/numpy/pull/8795, so that you can see what is > involved. > > All the best, > > Marten > > > > On Thu, May 18, 2017 at 9:04 AM, marc <[hidden email]> wrote: >> Dear Numpy developers, >> >> I'm trying to add a routine to calculate the sum of a product of two arrays >> (a dot product). But that would not increase the memory (from what I saw >> np.dot is increasing the memory while it should not be necessary). The idea >> is to avoid the use of the temporary array in the calculation of the >> variance ( numpy/numpy/core/_methods.py line 112). >> >> The routine that I want to implement look like this in python, >> >> arr = np.random.rand(100000) >> mean = arr.mean() >> var = 0.0 >> for ai in arr: var += (ai-mean)**2 >> >> I would like to implement it in the umath module. As a first step, I tried >> to reproduce the divmod function of umath, but I did not manage to do it, >> you can find my fork here (the branch with the changes is call >> looking_around). During compilation I get the following error, >> >> gcc: numpy/core/src/multiarray/number.c >> In file included from numpy/core/src/multiarray/number.c:17:0: >> numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’: >> numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka >> struct <anonymous>}’ has no member named ‘nb_sum_multiply’ >> (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func)) >> ^ >> numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro >> ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \ >> ^ >> numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro >> ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply, >> array_sum_multiply); >> >> Sorry if my question seems basic, but I'm new in Numpy development. >> Any help? >> >> Thank you in advance, >> >> Marc Barbry >> >> PS: I opened an issues as well on the github repository >> https://github.com/numpy/numpy/issues/9130 >> >> _______________________________________________ >> NumPy-Discussion mailing list >> [hidden email] >> https://mail.python.org/mailman/listinfo/numpy-discussion >> > _______________________________________________ > NumPy-Discussion mailing list > [hidden email] > https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion |
In reply to this post by marc
On Thu, 2017-05-18 at 15:04 +0200, marc wrote:
> Dear Numpy developers, > I'm trying to add a routine to calculate the sum of a product of two > arrays (a dot product). But that would not increase the memory (from > what I saw np.dot is increasing the memory while it should not be > necessary). The idea is to avoid the use of the temporary array in > the calculation of the variance ( numpy/numpy/core/_methods.py line > 112). np.dot should only increase memory in some cases (such as non- contiguous arrays) and be much faster in most cases (unless e.g. you do not have a BLAS compatible type). You might also want to check out np.einsum, which is pretty slick and can handle these kind of operations as well. Note that `np.dot` calls into BLAS so that it is in general much faster then np.einsum. - Sebastian > The routine that I want to implement look like this in python, > arr = np.random.rand(100000) > mean = arr.mean() > var = 0.0 > for ai in arr: var += (ai-mean)**2 > I would like to implement it in the umath module. As a first step, I > tried to reproduce the divmod function of umath, but I did not manage > to do it, you can find my fork here (the branch with the changes is > call looking_around). During compilation I get the following error, > gcc: numpy/core/src/multiarray/number.c > In file included from numpy/core/src/multiarray/number.c:17:0: > numpy/core/src/multiarray/number.c: In function > ‘array_sum_multiply’: > numpy/core/src/private/binop_override.h:176:39: error: > ‘PyNumberMethods {aka struct <anonymous>}’ has no member named > ‘nb_sum_multiply’ (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != > (void*)(test_func)) > ^ > numpy/core/src/private/binop_override.h:180:13: note: in expansion of > macro ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, > test_func) && \ > ^ > numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro > ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, > nb_sum_multiply, array_sum_multiply); > Sorry if my question seems basic, but I'm new in Numpy development. > Any help? > Thank you in advance, > Marc Barbry > > PS: I opened an issues as well on the github repository > https://github.com/numpy/numpy/issues/9130 > _______________________________________________ > NumPy-Discussion mailing list > [hidden email] > https://mail.python.org/mailman/listinfo/numpy-discussion NumPy-Discussion mailing list [hidden email] https://mail.python.org/mailman/listinfo/numpy-discussion signature.asc (817 bytes) Download Attachment |
Free forum by Nabble | Edit this page |