I am not exactly an expert on data storage, but I use PyTables a lot for

all kinds of scientific data sets and am very happy with it. Indeed it

has many advanced capabilities; so it may seem overkill at first glance.

But for simple tasks such as the one you describe the api is simple;

indeed I also use it for small data sets because it is such a quick way

of storing data in a portable way. Regarding speed and overhead: I don't

know in general what the penalties or gains are for very small files. On

my system an empty file is 1032 bytes, and if I fill it with an array of

3 by 30000 random float64's it is 723080. Not so bad.

Just try it out yourself:

>>> import numpy, tables

>>> ta = numpy.random.random((3,30000))

>>> f = tables.openFile('test.h5','w')

>>> f.createArray('/','testarray',ta)

>>> f.close()

With most real data file size can be smaller because you have the

option of enabling compression.

But I must admit that I haven't tried reading HDF5 in Matlab or C (and

never will); I know it is possible, but I don't know how difficult it

is.

Cheers, Gabriel

On Tue, 2008-05-20 at 12:11 -0400, Gary Pajer wrote:

> On Tue, May 20, 2008 at 10:26 AM, Gabriel J.L. Beckers

> <

[hidden email]> wrote:

> > PyTables is an efficient way of doing it (

http://www.pytables.org). You

> > essentially write data to a HDF5 file, which is portable and can be read

> > in Matlab or in a C program (using the HDF5 library).

> >

> > Gabriel

>

> I thought about that. It seems to have much more than I need, so I

> wonder if it's got more overhead / less speed / more complex API than

> I need. But big isn't necessarily bad, but it might be. Is pytables

> overkill?

>

>

> >

> > On Tue, 2008-05-20 at 09:32 -0400, Gary Pajer wrote:

> >> I want to store data in a way that can be read by a C or Matlab program.

> >>

> >> Not too much data, not too complicated: a dozen or so floats, a few

> >> integers, a few strings, and a (3, x) numpy array where typically 500

> >> < x < 30000.

> >>

> >> I was about to create my own format for storage when it occurred to me

> >> that I might want to use XML or some other standard format. Like

> >> JSON, perhaps. Can anyone comment, esp relating to numpy

> >> implementation issues, or offer suggestions?

> >>

> >> Thanks,

> >> Gary

> >> _______________________________________________

> >> Numpy-discussion mailing list

> >>

[hidden email]
> >>

http://projects.scipy.org/mailman/listinfo/numpy-discussion> >

> >

> > _______________________________________________

> > Numpy-discussion mailing list

> >

[hidden email]
> >

http://projects.scipy.org/mailman/listinfo/numpy-discussion> >

> _______________________________________________

> Numpy-discussion mailing list

>

[hidden email]
>

http://projects.scipy.org/mailman/listinfo/numpy-discussion_______________________________________________

Numpy-discussion mailing list

[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion