question about optimizing

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

question about optimizing

Brian Blais
Hello,

I have a custom array, which contains custom objects (I give a stripped down example below), and I want to loop over all of the elements of the array and call a method of the object.  I can do it like:

    a=MyArray((5,5),MyObject,10)
    
    for obj in a.flat:
        obj.update()

but I was wondering if there is a faster way, especially if obj.update is a cython-ed function.  I was thinking something like apply_along_axis, but without having an input array at all. 

Is there a better way to do this?  While I'm asking, is there a better way to overload ndarray than what I am doing below?  I tried to follow code I found online, but the examples of this are few and far between.


thanks you for any help!


Brian Blais


from numpy import ndarray,prod,array

class MyObject(object):
    def __init__(self,value):
        self.value=value
        
    def update(self):
        self.value*=2

    def __repr__(self):
        return "My value is %d." % self.value
        
class MyArray(ndarray):
    
    def __new__(subtype, shape,obj, *args,**kwargs):
        
        if isinstance(shape,int):
            N=shape
            shape=(shape,)
        else:
            N=prod(shape)

        objs=[]
        for i in range(N):
            objs.append(obj(*args,**kwargs))
            
            
        arr = array(objs, dtype=None, copy=False)
        arr = arr.view(subtype)
        arr.shape=shape
        
        return arr


if __name__=="__main__":
    
    a=MyArray((5,5),MyObject,10)
    
    for obj in a.flat:
        obj.update()


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Anne Archibald
2008/5/16 Brian Blais <[hidden email]>:

> I have a custom array, which contains custom objects (I give a stripped down
> example below), and I want to loop over all of the elements of the array and
> call a method of the object.  I can do it like:
>     a=MyArray((5,5),MyObject,10)
>
>     for obj in a.flat:
>         obj.update()
> but I was wondering if there is a faster way, especially if obj.update is a
> cython-ed function.  I was thinking something like apply_along_axis, but
> without having an input array at all.
> Is there a better way to do this?  While I'm asking, is there a better way
> to overload ndarray than what I am doing below?  I tried to follow code I
> found online, but the examples of this are few and far between.

Unfortunately, the loop overhead isn't very big compared to the
overhead of method dispatch, so there's no way you're going to make
things fast. For convenience you can do something like

update = vectorize(lambda object: object.update())

and then later

update(a)

But if you really want things to go quickly, I think the best approach
is to take advantage of the main feature of arrays: they hold
homogeneous items efficiently. So use the array to store the contents
of your object, and put any special behaviour in the array object. For
example, if I wanted to efficiently compute with arrays of numbers
carrying units, I would attach a unit to the array as a whole, and
have the array store doubles. With some cleverness, you can even have
accesses to the array return a freshly-created object whose contents
are based on the values you looked up in the array. But storing actual
python objects in an array is probably not a good idea.

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Alan G Isaac
On Fri, 16 May 2008, Anne Archibald apparently wrote:
> storing actual python objects in an array is probably not
> a good idea

I have been wondering what people use object arrays for.
I have been guessing that it is for indexing convenience?
Are there other core motivations?
Alan Isaac



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris


On Sat, May 17, 2008 at 9:52 AM, Alan G Isaac <[hidden email]> wrote:
On Fri, 16 May 2008, Anne Archibald apparently wrote:
> storing actual python objects in an array is probably not
> a good idea

I have been wondering what people use object arrays for.
I have been guessing that it is for indexing convenience?
Are there other core motivations?

You can always define an object array of matrices, which solves Tim's problem of matrix stacks, albeit in not the most efficient manner and not the easiest thing to specify due to the current limitations of array(...). I do think it would be nice to have arrays or arrays, but this needs one more array type so that one can do something like array(list_of_arrays, dtype=matrix((2,3),float)), i.e., we could use a fancier dtype.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Brian Blais
In reply to this post by Alan G Isaac
On May 17, 2008, at May 17:11:52 AM, Alan G Isaac wrote:

On Fri, 16 May 2008, Anne Archibald apparently wrote:
storing actual python objects in an array is probably not 
a good idea

I have been wondering what people use object arrays for.
I have been guessing that it is for indexing convenience?

at least for me, that was the motivation.  I am trying to build a simulation framework for part of the brain, which requires connected layers of nodes.  A layer is either a 1D or 2D structure of nodes, with each node a relatively complex beast.  Rather than reinvent the indexing (1D, 2D, slicing, etc...), I just inherited from ndarray.  I thought, after the fact, that some numpy functions on arrays would help speed up the code, which consists mostly of calling an update function on all nodes, passing each them an input vector.  I wasn't sure if there would be any speed up for this, compared to 

for n in self.flat:
   n.update(input_vector)

From the response, the answer seems to be no, and that I should stick with the python loops for clarity.  But also, the words of Anne Archibald, makes me think that I have made a bad choice by inheriting from ndarray, although I am not sure what a convenient alternative would be.



bb




_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Anne Archibald
In reply to this post by Charles R Harris
2008/5/17 Charles R Harris <[hidden email]>:

>
>
> On Sat, May 17, 2008 at 9:52 AM, Alan G Isaac <[hidden email]> wrote:
>>
>> On Fri, 16 May 2008, Anne Archibald apparently wrote:
>> > storing actual python objects in an array is probably not
>> > a good idea
>>
>> I have been wondering what people use object arrays for.
>> I have been guessing that it is for indexing convenience?
>> Are there other core motivations?
>
> You can always define an object array of matrices, which solves Tim's
> problem of matrix stacks, albeit in not the most efficient manner and not
> the easiest thing to specify due to the current limitations of array(...). I
> do think it would be nice to have arrays or arrays, but this needs one more
> array type so that one can do something like array(list_of_arrays,
> dtype=matrix((2,3),float)), i.e., we could use a fancier dtype.

I think if you're going to be writing code to do this, it would be
better not to use object arrays. After all, there no reason the
underlysing storage for an array of matrices shouldn't be one big
block of memory rather than a lot of scattered python objects.

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris


On Sat, May 17, 2008 at 1:02 PM, Anne Archibald <[hidden email]> wrote:
2008/5/17 Charles R Harris <[hidden email]>:
>
>
> On Sat, May 17, 2008 at 9:52 AM, Alan G Isaac <[hidden email]> wrote:
>>
>> On Fri, 16 May 2008, Anne Archibald apparently wrote:
>> > storing actual python objects in an array is probably not
>> > a good idea
>>
>> I have been wondering what people use object arrays for.
>> I have been guessing that it is for indexing convenience?
>> Are there other core motivations?
>
> You can always define an object array of matrices, which solves Tim's
> problem of matrix stacks, albeit in not the most efficient manner and not
> the easiest thing to specify due to the current limitations of array(...). I
> do think it would be nice to have arrays or arrays, but this needs one more
> array type so that one can do something like array(list_of_arrays,
> dtype=matrix((2,3),float)), i.e., we could use a fancier dtype.

I think if you're going to be writing code to do this, it would be
better not to use object arrays. After all, there no reason the
underlysing storage for an array of matrices shouldn't be one big
block of memory rather than a lot of scattered python objects.

Exactly, which is why I suggested an extension to dtype so that it could also specify arraya. In that way matrix multiplication on matrix stacks would simply go over to an element wise matrix multiplication. In that case, dtype would also be nestable so that one could have arrays of arrays of arrays ...

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Anne Archibald
In reply to this post by Brian Blais
2008/5/17 Brian Blais <[hidden email]>:

> at least for me, that was the motivation.  I am trying to build a simulation
> framework for part of the brain, which requires connected layers of nodes.
>  A layer is either a 1D or 2D structure of nodes, with each node a
> relatively complex beast.  Rather than reinvent the indexing (1D, 2D,
> slicing, etc...), I just inherited from ndarray.  I thought, after the fact,
> that some numpy functions on arrays would help speed up the code, which
> consists mostly of calling an update function on all nodes, passing each
> them an input vector.  I wasn't sure if there would be any speed up for
> this, compared to
> for n in self.flat:
>    n.update(input_vector)
> From the response, the answer seems to be no, and that I should stick with
> the python loops for clarity.  But also, the words of Anne Archibald, makes
> me think that I have made a bad choice by inheriting from ndarray, although
> I am not sure what a convenient alternative would be.

Well, it doesn't exist yet, but a handy tool would be a factory
function "ArrayOf"; you would pass it a class, and it would produce a
subclass of ndarray designed to contain that class. That is, the
underlying storage would be a record array, but the getitem and
setitem would automatically handle conversion to and from the class
you supplied it, where appropriate.

myarray = ArrayOf(Node,dtype=...)
A = myarray.array([Node(...), Node(...), Node(...)])
n = A[1]
A[2] = Node(...)
A.C.update() # python-loop-based update of all elements

You could also design it so that it was easy to derive a class from
it, since that's probably the best way to handle vectorized methods:

class myarray(ArrayOf(Node, dtype=...)):
    def update(self):
        self.underlying["node_attribute"] += 1


I should say, if you can get away with treating your nodes more like C
structures and writing (possibly vectorized) functions to act on them,
you can avoid all this mumbo jumbo:

node_dtype = [("node_attribute",np.int),("weight", np.float)]
A = np.zeros(10,dtype=node_dtype)

def nodes_update(A):
    A["node_attribute"] += 1

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris


On Sat, May 17, 2008 at 1:18 PM, Anne Archibald <[hidden email]> wrote:
2008/5/17 Brian Blais <[hidden email]>:

> at least for me, that was the motivation.  I am trying to build a simulation
> framework for part of the brain, which requires connected layers of nodes.
>  A layer is either a 1D or 2D structure of nodes, with each node a
> relatively complex beast.  Rather than reinvent the indexing (1D, 2D,
> slicing, etc...), I just inherited from ndarray.  I thought, after the fact,
> that some numpy functions on arrays would help speed up the code, which
> consists mostly of calling an update function on all nodes, passing each
> them an input vector.  I wasn't sure if there would be any speed up for
> this, compared to
> for n in self.flat:
>    n.update(input_vector)
> From the response, the answer seems to be no, and that I should stick with
> the python loops for clarity.  But also, the words of Anne Archibald, makes
> me think that I have made a bad choice by inheriting from ndarray, although
> I am not sure what a convenient alternative would be.

Well, it doesn't exist yet, but a handy tool would be a factory
function "ArrayOf"; you would pass it a class, and it would produce a
subclass of ndarray designed to contain that class.

Subclasses should generally be avoided unless they satisfy the "is a" criterion, which I don't think a matrix stack does. That is to say, a subclass should behave as an ndarray in *all* ways except for added functionality. All we want is an item type with dimensions.

That is, the
underlying storage would be a record array, but the getitem and
setitem would automatically handle conversion to and from the class
you supplied it, where appropriate.

I don't think getitem and setitem should be overloaded in a subclass, that is where the matrix class went wrong. Those methods should not be considered "virtual". So if you have an array of special elements, then you are allowed a[0][i, j], but you aren't allowed  a[0, i, j]. I think this has the virtue of sense and consistency: you can overload the operators of the type anyway you please, but you can't overload the operators of ndarray.

I suppose we could specify which operators in ndarray *could* be considered virtual. That, after all, is what designing a base class is all about. But we haven't done that.



myarray = ArrayOf(Node,dtype=...)
A = myarray.array([Node(...), Node(...), Node(...)])
n = A[1]
A[2] = Node(...)
A.C.update() # python-loop-based update of all elements

You could also design it so that it was easy to derive a class from
it, since that's probably the best way to handle vectorized methods:

class myarray(ArrayOf(Node, dtype=...)):
   def update(self):
       self.underlying["node_attribute"] += 1


I should say, if you can get away with treating your nodes more like C
structures and writing (possibly vectorized) functions to act on them,
you can avoid all this mumbo jumbo:

Yes, what we want is something like an object array with efficient contiguous storage.  Record types are a step in that direction, they just aren't complete enough. Hmm, this is an argument for using methods instead of functions, so that you could sort on the columns of the matrices in a stack by doing something like a[...].sort(axis=1).


node_dtype = [("node_attribute",np.int),("weight", np.float)]
A = np.zeros(10,dtype=node_dtype)

def nodes_update(A):
   A["node_attribute"] += 1

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Bryan Cole-2
In reply to this post by Brian Blais

>
>
> From the response, the answer seems to be no, and that I should stick
> with the python loops for clarity.  But also, the words of Anne
> Archibald, makes me think that I have made a bad choice by inheriting
> from ndarray, although I am not sure what a convenient alternative
> would be.

It may be worth bearing in mind that indexing a python list is faster
than a ndarray (mostly because ndarrays must support many more
features), so if you don't require (or exploit) the features of ndarrays
(i.e. you don't need fancy indexing or element-wise operations), you may
be better off with lists.

BC

>
>
>
>
>
>
> bb
>
>
>
>
>
> --
> Brian Blais
> [hidden email]
> http://web.bryant.edu/~bblais
>
>
>
>
>
> _______________________________________________
> Numpy-discussion mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris
In reply to this post by Charles R Harris


On Sat, May 17, 2008 at 1:45 PM, Charles R Harris <[hidden email]> wrote:


On Sat, May 17, 2008 at 1:18 PM, Anne Archibald <[hidden email]> wrote:
2008/5/17 Brian Blais <[hidden email]>:

> at least for me, that was the motivation.  I am trying to build a simulation
> framework for part of the brain, which requires connected layers of nodes.
>  A layer is either a 1D or 2D structure of nodes, with each node a
> relatively complex beast.  Rather than reinvent the indexing (1D, 2D,
> slicing, etc...), I just inherited from ndarray.  I thought, after the fact,
> that some numpy functions on arrays would help speed up the code, which
> consists mostly of calling an update function on all nodes, passing each
> them an input vector.  I wasn't sure if there would be any speed up for
> this, compared to
> for n in self.flat:
>    n.update(input_vector)
> From the response, the answer seems to be no, and that I should stick with
> the python loops for clarity.  But also, the words of Anne Archibald, makes
> me think that I have made a bad choice by inheriting from ndarray, although
> I am not sure what a convenient alternative would be.

Well, it doesn't exist yet, but a handy tool would be a factory
function "ArrayOf"; you would pass it a class, and it would produce a
subclass of ndarray designed to contain that class.

Subclasses should generally be avoided unless they satisfy the "is a" criterion, which I don't think a matrix stack does. That is to say, a subclass should behave as an ndarray in *all* ways except for added functionality. All we want is an item type with dimensions.

Base classes also tend to have limited functionality that will be common to all derived types. The object type in Python has only a few methods and attributes:

In [4]: dir(object)
Out[4]:
['__class__',
 '__delattr__',
 '__doc__',
 '__getattribute__',
 '__hash__',
 '__init__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__str__']

And overloading any of these is likely to cause trouble.

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Robert Kern-2
On Sat, May 17, 2008 at 3:18 PM, Charles R Harris
<[hidden email]> wrote:

> Base classes also tend to have limited functionality that will be common to
> all derived types. The object type in Python has only a few methods and
> attributes:
>
> In [4]: dir(object)
> Out[4]:
> ['__class__',
>  '__delattr__',
>  '__doc__',
>  '__getattribute__',
>  '__hash__',
>  '__init__',
>  '__new__',
>  '__reduce__',
>  '__reduce_ex__',
>  '__repr__',
>  '__setattr__',
>  '__str__']
>
> And overloading any of these is likely to cause trouble.

Nonsense. *Most* of those are intended to be overloaded. Especially on object.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris


On Sat, May 17, 2008 at 2:25 PM, Robert Kern <[hidden email]> wrote:
On Sat, May 17, 2008 at 3:18 PM, Charles R Harris
<[hidden email]> wrote:
> Base classes also tend to have limited functionality that will be common to
> all derived types. The object type in Python has only a few methods and
> attributes:
>
> In [4]: dir(object)
> Out[4]:
> ['__class__',
>  '__delattr__',
>  '__doc__',
>  '__getattribute__',
>  '__hash__',
>  '__init__',
>  '__new__',
>  '__reduce__',
>  '__reduce_ex__',
>  '__repr__',
>  '__setattr__',
>  '__str__']
>
> And overloading any of these is likely to cause trouble.

Nonsense. *Most* of those are intended to be overloaded. Especially on object.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Charles R Harris
In reply to this post by Robert Kern-2


On Sat, May 17, 2008 at 2:25 PM, Robert Kern <[hidden email]> wrote:
On Sat, May 17, 2008 at 3:18 PM, Charles R Harris
<[hidden email]> wrote:
> Base classes also tend to have limited functionality that will be common to
> all derived types. The object type in Python has only a few methods and
> attributes:
>
> In [4]: dir(object)
> Out[4]:
> ['__class__',
>  '__delattr__',
>  '__doc__',
>  '__getattribute__',
>  '__hash__',
>  '__init__',
>  '__new__',
>  '__reduce__',
>  '__reduce_ex__',
>  '__repr__',
>  '__setattr__',
>  '__str__']
>
> And overloading any of these is likely to cause trouble.

Nonsense. *Most* of those are intended to be overloaded. Especially on object.

My bad. What I should have said is, if you change the semantics of these methods and attributes, then things will go wrong. The Python documentation says as much. For instance, if __new__  returns the time of day. So my point would be: those things intended to be overloading are well defined and what they do is part of the contract.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Robert Kern-2
On Sat, May 17, 2008 at 3:33 PM, Charles R Harris
<[hidden email]> wrote:

>
> On Sat, May 17, 2008 at 2:25 PM, Robert Kern <[hidden email]> wrote:
>>
>> On Sat, May 17, 2008 at 3:18 PM, Charles R Harris
>> <[hidden email]> wrote:
>> > Base classes also tend to have limited functionality that will be common
>> > to
>> > all derived types. The object type in Python has only a few methods and
>> > attributes:
>> >
>> > In [4]: dir(object)
>> > Out[4]:
>> > ['__class__',
>> >  '__delattr__',
>> >  '__doc__',
>> >  '__getattribute__',
>> >  '__hash__',
>> >  '__init__',
>> >  '__new__',
>> >  '__reduce__',
>> >  '__reduce_ex__',
>> >  '__repr__',
>> >  '__setattr__',
>> >  '__str__']
>> >
>> > And overloading any of these is likely to cause trouble.
>>
>> Nonsense. *Most* of those are intended to be overloaded. Especially on
>> object.
>
> My bad. What I should have said is, if you change the semantics of these
> methods and attributes, then things will go wrong. The Python documentation
> says as much. For instance, if __new__  returns the time of day. So my point
> would be: those things intended to be overloading are well defined and what
> they do is part of the contract.

Gotcha.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Anne Archibald
In reply to this post by Anne Archibald
2008/5/17 Anne Archibald <[hidden email]>:

> 2008/5/17 Brian Blais <[hidden email]>:
>
>> at least for me, that was the motivation.  I am trying to build a simulation
>> framework for part of the brain, which requires connected layers of nodes.
>>  A layer is either a 1D or 2D structure of nodes, with each node a
>> relatively complex beast.  Rather than reinvent the indexing (1D, 2D,
>> slicing, etc...), I just inherited from ndarray.  I thought, after the fact,
>> that some numpy functions on arrays would help speed up the code, which
>> consists mostly of calling an update function on all nodes, passing each
>> them an input vector.  I wasn't sure if there would be any speed up for
>> this, compared to
>> for n in self.flat:
>>    n.update(input_vector)
>> From the response, the answer seems to be no, and that I should stick with
>> the python loops for clarity.  But also, the words of Anne Archibald, makes
>> me think that I have made a bad choice by inheriting from ndarray, although
>> I am not sure what a convenient alternative would be.
>
> Well, it doesn't exist yet, but a handy tool would be a factory
> function "ArrayOf"; you would pass it a class, and it would produce a
> subclass of ndarray designed to contain that class. That is, the
> underlying storage would be a record array, but the getitem and
> setitem would automatically handle conversion to and from the class
> you supplied it, where appropriate.
>
> myarray = ArrayOf(Node,dtype=...)
> A = myarray.array([Node(...), Node(...), Node(...)])
> n = A[1]
> A[2] = Node(...)
> A.C.update() # python-loop-based update of all elements
>
> You could also design it so that it was easy to derive a class from
> it, since that's probably the best way to handle vectorized methods:
>
> class myarray(ArrayOf(Node, dtype=...)):
>    def update(self):
>        self.underlying["node_attribute"] += 1

So just as an experiment I implemented some of this:
http://www.scipy.org/Cookbook/Obarray

Anne
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: question about optimizing

Robin-62
In reply to this post by Brian Blais
On Sat, May 17, 2008 at 7:22 PM, Brian Blais <[hidden email]> wrote:

> at least for me, that was the motivation.  I am trying to build a simulation
> framework for part of the brain, which requires connected layers of nodes.
>  A layer is either a 1D or 2D structure of nodes, with each node a
> relatively complex beast.  Rather than reinvent the indexing (1D, 2D,
> slicing, etc...), I just inherited from ndarray.  I thought, after the fact,
> that some numpy functions on arrays would help speed up the code, which
> consists mostly of calling an update function on all nodes, passing each
> them an input vector.  I wasn't sure if there would be any speed up for
> this, compared to
> for n in self.flat:
>    n.update(input_vector)
> From the response, the answer seems to be no, and that I should stick with
> the python loops for clarity.  But also, the words of Anne Archibald, makes
> me think that I have made a bad choice by inheriting from ndarray, although
> I am not sure what a convenient alternative would be.
>
>
> bb

Hello,

It depends on what you are doing but to really exploit the performance
gains of numpy it can be better to change the way you are doing the
model to have arrays representing the various properties of each node
seperately.

For example, instead of having each node with a voltage and a number
of channels node.V node.C (list of n channel conductances), you have
big arrays holding the values for all of them. So V would be an array
1xn_nodes and C would be an array of n_nodes x max number of channels
(some of them could be zero if not all nodes have the same number of
channels).
Another global type array n_nodes x n_nodes could hold all the connections...

Then instead of updating each node individually you can update all
nodes together with vectorised operations utilising ATLAS etc.

The indexing can get a bit complicated and it may not be possible
depending on how your nodes interact (but usually even if a node
depends on values of other nodes it does so only at the previous
timestep so you can store the full previous state and reference it in
the update function).

Just a suggestion - it was much more efficient for me to do it this
way with integrate and fire type neural networks... Also I hope I've
clearly expressed what I mean - it's getting late here.

Cheers

Robin
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion