How to speed up array generating

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

How to speed up array generating

klark--kent
Hello. There is a random 1D array m_0 with size 3000, for example:
m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

Joseph Fox-Rabinovitz
What other ways have you tried?

On Sat, Jan 9, 2021 at 2:15 PM <[hidden email]> wrote:
Hello. There is a random 1D array m_0 with size 3000, for example:
m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

klark--kent
np.meshgrid, indexing, reshape

09.01.2021, 22:30, "Joseph Fox-Rabinovitz" <[hidden email]>:
What other ways have you tried?

On Sat, Jan 9, 2021 at 2:15 PM <[hidden email]> wrote:
Hello. There is a random 1D array m_0 with size 3000, for example:
m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
,

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

bashtage
What about arange and then an integer divide or mod?

Kevin


On Sat, Jan 9, 2021, 20:54 <[hidden email]> wrote:
np.meshgrid, indexing, reshape

09.01.2021, 22:30, "Joseph Fox-Rabinovitz" <[hidden email]>:
What other ways have you tried?

On Sat, Jan 9, 2021 at 2:15 PM <[hidden email]> wrote:
Hello. There is a random 1D array m_0 with size 3000, for example:
m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
,

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

bashtage
Actually I would try a broadcast multiply followed by ravel first.  

Kevin


On Sat, Jan 9, 2021, 21:12 Kevin Sheppard <[hidden email]> wrote:
What about arange and then an integer divide or mod?

Kevin


On Sat, Jan 9, 2021, 20:54 <[hidden email]> wrote:
np.meshgrid, indexing, reshape

09.01.2021, 22:30, "Joseph Fox-Rabinovitz" <[hidden email]>:
What other ways have you tried?

On Sat, Jan 9, 2021 at 2:15 PM <[hidden email]> wrote:
Hello. There is a random 1D array m_0 with size 3000, for example:
m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
,

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

Sebastian Berg
In reply to this post by klark--kent
On Sat, 2021-01-09 at 23:45 +0300, [hidden email] wrote:
> np.meshgrid, indexing, reshape

I would expect e.g. reshape helps if you make use of the fact that this
is very clear in 2-D. The other thing is ensuring to use the methods
`arr.reshape`, `arr.repeat`, which avoids overheads (some of which may
be optimized away in the future though).

That said, with this type of code and the small arrays in the example,
there are fairly significant overheads.  So if  you want to get serious
input or a good solution, I would suggest to give more context and
include how those lines of code are used in your program/function.

Chances are that there is a much faster and possibly more elegant way
to write it, e.g. by solving many of these at the same time, but nobody
will be able to tell unless they know how you need it. Also with more
context, the other possibility is that somone may hint at cython or
numba/transonic.

Cheers,

Sebastian


>
> 09.01.2021, 22:30, "Joseph Fox-Rabinovitz"
> <[hidden email]>:
> What other ways have you tried?
>
> On Sat, Jan 9, 2021 at 2:15 PM <[hidden email]> wrote:
> > Hello. There is a random 1D array m_0 with size 3000, for example:
> > m_0 = np.array([0, 1, 2])
> > I need to generate two 1D arrays:
> > m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
> > m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])
> > Is there faster way to do it than this one:
> > import numpy as np
> > import time
> > N = 3
> > m_0 = np.arange(N)
> >
> > t = time.time()
> > m_1 = np.tile(m_0, N)
> > m_2 = np.repeat(m_0, N)
> > t = time.time() - t
> > I tried other ways but they are slower or have the same time. Other
> > NumPy operations in my code 10-100 times faster. Why the repeating
> > an array is so slow? I need 10 times speed up. Thank you for your
> > attantion to my problem.
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> ,
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

V. Armando Sole
In reply to this post by klark--kent

I guess the speed up, if any, will be machine dependent, but you can give a try at:

import numpy as np
import time
N = 3000
m_0 = np.arange(N)
t = time.time()
a = np.ones(N)
m_1 = np.outer(m_0, a).ravel()
m_2 = np.outer(a, m_0).ravel()
t = time.time() - t

 


On 2021-01-09 20:07, [hidden email] wrote:

Hello. There is a random 1D array m_0 with size 3000, for example:

m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

klark--kent
Thank you but the same time:
   import numpy as np
    import time
    N = 3000
    m_0 = np.arange(N)
    a = np.ones(N)
#===== first =========
    t1 = time.time()
    m_1 = np.outer(m_0, a).ravel()
    m_2 = np.outer(a, m_0).ravel()
    t1 = time.time() - t1
#===== second =======
    t2 = time.time()
    m_3 = np.tile(m_0, N)
    m_4  = np.repeat(m_0, N)
    t2 = time.time() - t2
#==================
    np.sum(m_1 - m_3)
    np.sum(m_2 - m_4)
    t1
    t2
 
10.01.2021, 08:37, "V. Armando Sole" <[hidden email]>:

I guess the speed up, if any, will be machine dependent, but you can give a try at:

import numpy as np
import time
N = 3000
m_0 = np.arange(N)
t = time.time()
a = np.ones(N)
m_1 = np.outer(m_0, a).ravel()
m_2 = np.outer(a, m_0).ravel()
t = time.time() - t

 

 

On 2021-01-09 20:07, [hidden email] wrote:

Hello. There is a random 1D array m_0 with size 3000, for example:

m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
 
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
,

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: How to speed up array generating

D.S. McNeil

As Sebastian said, it's going to take more information about context.  It's very unlikely that you actually need these particular arrays-- probably, you're interested in some more important result which you're using these arrays as an intermediate step to obtain.  Their structure makes that even more likely than usual.

Without knowing how they're being used, no one can know whether they should be replaced by a nested numba or cython loop, or a clever use of einsum, or who knows what else.

The minimum time code can take is no time. :-)


Doug

On Sun, 10 Jan 2021 at 02:00, <[hidden email]> wrote:
Thank you but the same time:
   import numpy as np
    import time
    N = 3000
    m_0 = np.arange(N)
    a = np.ones(N)
#===== first =========
    t1 = time.time()
    m_1 = np.outer(m_0, a).ravel()
    m_2 = np.outer(a, m_0).ravel()
    t1 = time.time() - t1
#===== second =======
    t2 = time.time()
    m_3 = np.tile(m_0, N)
    m_4  = np.repeat(m_0, N)
    t2 = time.time() - t2
#==================
    np.sum(m_1 - m_3)
    np.sum(m_2 - m_4)
    t1
    t2
 
10.01.2021, 08:37, "V. Armando Sole" <[hidden email]>:

I guess the speed up, if any, will be machine dependent, but you can give a try at:

import numpy as np
import time
N = 3000
m_0 = np.arange(N)
t = time.time()
a = np.ones(N)
m_1 = np.outer(m_0, a).ravel()
m_2 = np.outer(a, m_0).ravel()
t = time.time() - t

 

 

On 2021-01-09 20:07, [hidden email] wrote:

Hello. There is a random 1D array m_0 with size 3000, for example:

m_0 = np.array([0, 1, 2])

I need to generate two 1D arrays:

m_1 = np.array([0, 1, 2, 0, 1, 2, 0, 1, 2])
m_2 = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

Is there faster way to do it than this one:

import numpy as np
import time
N = 3
m_0 = np.arange(N)

t = time.time()
m_1 = np.tile(m_0, N)
m_2 = np.repeat(m_0, N)
t = time.time() - t
I tried other ways but they are slower or have the same time. Other NumPy operations in my code 10-100 times faster. Why the repeating an array is so slow? I need 10 times speed up. Thank you for your attantion to my problem.
 
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
,

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion