Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

YueCompl
Hello folks,

Myself a long time Numpy user, and recently done some Haskell stuff,


Best regards,
Compl


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

Matthew Brett
Hi,

On Mon, Jan 20, 2020 at 3:00 PM YueCompl <[hidden email]> wrote:
>
> Hello folks,
>
> Myself a long time Numpy user, and recently done some Haskell stuff,
>
> Please have a look at  https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell,
> with http://hackage.haskell.org/package/accelerate  and/or  http://hackage.haskell.org/package/repa doing the number crunching?

Interesting - do you have any feeling for the differences in speed for
Haskell accelerate array processing as compared to Numpy with default
OpenBLAS?

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

YueCompl
Hi Matthew,

I hate to say I'm still a noob in the implementation details of Numpy, I
just got an interpreter for sth else working, and suddenly realized it
can make seemingly better magic methods in Python spirit.

Repa and Accelerate per my learning so far, are focused on loop-fusion,
i.e. optimization after all intended computations have been expressed
and composed together, this seems some different from Numpy/Pandas
tradition to implement doable things ex ante, and let the users do
things interactively  ad-hoc. That said, so I feel Repa will do composed
computations more speedy, once properly organized. And with Accelerate,
since it does GPU natively, I do feel a sure thing for it to excel CPU
implementations.

btw I'm not a native English speaker, please don't hesitate to ask me to
make myself clearer in my wordings, I'm aware of my flaws in English skills.

btw I'm a believer that Brain-Computer-Interface ought to happen sooner,
especially non-intrusive ones, I can't wait to program such interfaces
for great good and fun. I see you are pioneering in this area, take my
admiring, and I started following you on github, looking forward to hear
from you more.

btw I'm behind China's Great Fire Wall, access to some western sites are
banned badly at times, that will explain why I can be unresponsive some
times.

Cheers,

Compl


On 2020/1/21 下午10:20, Matthew Brett wrote:

> Hi,
>
> On Mon, Jan 20, 2020 at 3:00 PM YueCompl <[hidden email]> wrote:
>> Hello folks,
>>
>> Myself a long time Numpy user, and recently done some Haskell stuff,
>>
>> Please have a look at  https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell,
>> with http://hackage.haskell.org/package/accelerate  and/or  http://hackage.haskell.org/package/repa doing the number crunching?
> Interesting - do you have any feeling for the differences in speed for
> Haskell accelerate array processing as compared to Numpy with default
> OpenBLAS?
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

Frederic Bastien
The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up.



[1] https://github.com/pydata/numexpr

-----Original Message-----
From: NumPy-Discussion <numpy-discussion-bounces+fbastien=[hidden email]> On Behalf Of Compl Yue
Sent: Tuesday, January 21, 2020 10:33 AM
To: [hidden email]
Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

External email: Use caution opening links or attachments


Hi Matthew,

I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit.

Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively  ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations.

btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills.

btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more.

btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times.

Cheers,

Compl


On 2020/1/21 下午10:20, Matthew Brett wrote:

> Hi,
>
> On Mon, Jan 20, 2020 at 3:00 PM YueCompl <[hidden email]> wrote:
>> Hello folks,
>>
>> Myself a long time Numpy user, and recently done some Haskell stuff,
>>
>> Please have a look at  
>> https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate  and/or  http://hackage.haskell.org/package/repa doing the number crunching?
> Interesting - do you have any feeling for the differences in speed for
> Haskell accelerate array processing as compared to Numpy with default
> OpenBLAS?
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

YueCompl
Yes, I do feel NumExpr does very similar things as Repa, while Repa works rather statically for Haskell code, NumExpr is much more interactive.

Well, honestly as a long time Numpy user, I'm aware of NumExpr since early days I learning Numpy, but never ever resorted to NumExpr, it's neither as complete as Numpy, nor as flexible as Numba.

I'm neither proficient in Haskell at data exploration for now, attempting improving the tooling (e.g. [hadui](https://github.com/complyue/hadui) ), while still working on an ndarray interface for Haskell to an in house array database (mmap'ed virtual large datafile per se). I miss very much the workflow with Pandas/Numpy/Numba, the interactive aspect is crucial to me and my team's daily work. We have to resort to Haskell because some time ago we needed to start processing irregular event streams (no common dimension along time axis), for thousands of series (instruments in the stock market). While in handling this many number of inputs (passing in a tuple of 1d arrays, Numba treats each array as a separate input variable), Numba will swallow 20+GB ram with 10+ hours in jit compilation without a result.

I'm aware of Julia, and feels very pity for my org not able to adopt it for hiring difficulties, while luckily Haskell is agreed to be the base of our next gen internal platform. So I'm currently working hard to bring sufficient interaction abilities with a Haskell tool chain.

On 2020-01-22, at 01:58, Frederic Bastien <[hidden email]> wrote:

The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up.



[1] https://github.com/pydata/numexpr

-----Original Message-----
From: NumPy-Discussion <[hidden email]> On Behalf Of Compl Yue
Sent: Tuesday, January 21, 2020 10:33 AM
To: [hidden email]
Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?

External email: Use caution opening links or attachments


Hi Matthew,

I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit.

Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively  ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations.

btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills.

btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more.

btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times.

Cheers,

Compl


On 2020/1/21 下午10:20, Matthew Brett wrote:
Hi,

On Mon, Jan 20, 2020 at 3:00 PM YueCompl <[hidden email]> wrote:
Hello folks,

Myself a long time Numpy user, and recently done some Haskell stuff,

Please have a look at  
https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods and https://github.com/e-wrks/edh/tree/master/Tour#indexing , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate  and/or  http://hackage.haskell.org/package/repa doing the number crunching?
Interesting - do you have any feeling for the differences in speed for
Haskell accelerate array processing as compared to Numpy with default
OpenBLAS?

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion