Segmentation fault

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Segmentation fault

Anne-Sophie Sertier
Hello !

I'm a new user of Numpy. I want to use it to work with matrices (very
huge matrices), select column, make product, etc ...
Configuration : Debian, python 2.4.4 and numpy 1.0.1

So here is my problem:
I initialize a matrix which will contain only 0 and 1

 >>> from numpy import *
>>> matCons = zeros((194844,267595)),dtype=int8)

then I put the 1s according to my conditions

>>> for i in range(194844):
        try:
                clusters = consensus[namesP[i]]
        except KeyError:
                continue
        for clus in clusters:
                ind = argmax(namesC.find(clus)) # search of index
                matrix[i,ind] = 1

finaly I want to select some columns in this matrix.

So I did for example:

>>> matCons[:,(1,2)]

and I obtain the segmentation fault.
I try to select one column by one, which is possible :

>>> a = matCons[:,1]

but if I try to put it in a new matrix, I have again te segmentation fault

>>> tmp = zeros((194844,2),int8)
>>> tmp[:,0] = a

I make some tests like this one :
----------------------------------------------------------
>>> matrix = zeros((194844,267595),int8)
>>> a = matrix[:,0]
>>> for i in range(len(a)):
        print i, a[i]
...

...
2221 0
2222 0
2223 0
2224 0
2225 0
2226 0
2227 0
2228 0
2229 0
2230 0
2231 0
2232 0
2233 0
2234 0
2235 0
2236 0
2237 0
2238 0
2239 0
2240 0
2241 95
2242 0
2243 0
2244 101
2245 0
2246 15
2247 -112
2248 -115
2249 -1
2250 84
2251 -99
2252 -1
2253 60
2254 -49
2255 -1
2256 17
2257 -35
2258 -119
2259 -35
2260 -1
2261 52
2262 -1
2263 -1
2264 -35
2265 -1
2266 121
2267 1
2268 0
2269 0
2270 0
2271 0
2272 28
2273 50
2274 4
2275 -1
2276 -24
2277 0
2278 0
2279 0
2280 -58
2281 0
2282 0
2283 0
2284 0
2285 -1
2286 -69
2287 12
2288 -36
2289 8
2290 66
Erreur de segmentation
------------------------------------------
But if I do the same things on a matrix with dimension 194844*300, it
works well.

So if someone can explain me why it does not work in my case...

Thanks in advance.

Anne-Sophie



--
Anne-Sophie Sertier, Doctorante monitrice
UMR 5558, Laboratoire de Biométrie et Biologie Evolutive
Université Lyon1, bât G. Mendel 2è étage
43 bd du 11 novembre 1918
69622 Villeurbanne cedex
France

Tel: 04 26 23 44 76

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Segmentation fault

cdavid
Anne-Sophie Sertier wrote:

> Hello !
>
> I'm a new user of Numpy. I want to use it to work with matrices (very
> huge matrices), select column, make product, etc ...
> Configuration : Debian, python 2.4.4 and numpy 1.0.1
>
> So here is my problem:
> I initialize a matrix which will contain only 0 and 1
>
>  >>> from numpy import *
>  
>>>> matCons = zeros((194844,267595)),dtype=int8)
>>>>        

Unless you are executing this on a gigantic computer, this won't work
very well: you are asking to create an array which has ~ 2e5^2 elements,
that is around 40 Gb.

There is a bug, but the bug happens at the above line: the zeros call
did not fail whereas it should have. It is likely caused because the
number of elements cannot fit into a 32 bits integers, which means it
overflows:

import numpy as np
n , m = 1e5,1e5
a = np.zeros((n, m), np.int8)
assert a.size == n * m

Will raise an assertion error (n * m is just big enough to not fit into
a 32 bits integer in this case).

cheers,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Segmentation fault

cdavid
David Cournapeau wrote:

>
> Unless you are executing this on a gigantic computer, this won't work
> very well: you are asking to create an array which has ~ 2e5^2 elements,
> that is around 40 Gb.
>
> There is a bug, but the bug happens at the above line: the zeros call
> did not fail whereas it should have. It is likely caused because the
> number of elements cannot fit into a 32 bits integers, which means it
> overflows:
>
> import numpy as np
> n , m = 1e5,1e5
> a = np.zeros((n, m), np.int8)
> assert a.size == n * m
>
> Will raise an assertion error (n * m is just big enough to not fit into
> a 32 bits integer in this case).
>  

I forgot to add: this is a bug in numpy, you should not get a
segmentation fault, but it would not work anyway (because you are asking
for a too big array) :)

cheers,

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Segmentation fault

Anne-Sophie Sertier
Thanks a lot for your answer ! I will try another way !

Anne-Sophie

David Cournapeau a écrit :

> David Cournapeau wrote:
>> Unless you are executing this on a gigantic computer, this won't work
>> very well: you are asking to create an array which has ~ 2e5^2 elements,
>> that is around 40 Gb.
>>
>> There is a bug, but the bug happens at the above line: the zeros call
>> did not fail whereas it should have. It is likely caused because the
>> number of elements cannot fit into a 32 bits integers, which means it
>> overflows:
>>
>> import numpy as np
>> n , m = 1e5,1e5
>> a = np.zeros((n, m), np.int8)
>> assert a.size == n * m
>>
>> Will raise an assertion error (n * m is just big enough to not fit into
>> a 32 bits integer in this case).
>>  
>
> I forgot to add: this is a bug in numpy, you should not get a
> segmentation fault, but it would not work anyway (because you are asking
> for a too big array) :)
>
> cheers,
>
> David
> _______________________________________________
> Numpy-discussion mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>

--
Anne-Sophie Sertier, Doctorante monitrice
UMR 5558, Laboratoire de Biométrie et Biologie Evolutive
Université Lyon1, bât G. Mendel 2è étage
43 bd du 11 novembre 1918
69622 Villeurbanne cedex
France

Tel: 04 26 23 44 76
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion