out parameter for np.fromfile

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

out parameter for np.fromfile

hmaarrfk

I was wondering what would your thoughts be on adding an output parameter to np.fromfile?

The advantage would be when interfacing with executables like ffmpeg which are arguably easier to use by calling them as a subprocess compared to a shared library in python.

Having the output parameter in np.fromfile would enable pre-allocation of large arrays that are reused during the computation of new image frames when decoding large video files.

Thoughts are appreciated!

Mark


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: out parameter for np.fromfile

Marten van Kerkwijk
Hi Mark,

Having an `out` might make sense. With present numpy, if you are really dealing with a file or file-like object, you might consider using `np.memmap` to access the data more directly. If it is something that looks more like a buffer, `np.frombuffer` may be useful (that doesn't copy data, but points the array at the memory that holds the buffer).

All the best,

Marten


On Sun, Nov 4, 2018 at 10:35 PM Mark Harfouche <[hidden email]> wrote:

I was wondering what would your thoughts be on adding an output parameter to np.fromfile?

The advantage would be when interfacing with executables like ffmpeg which are arguably easier to use by calling them as a subprocess compared to a shared library in python.

Having the output parameter in np.fromfile would enable pre-allocation of large arrays that are reused during the computation of new image frames when decoding large video files.

Thoughts are appreciated!

Mark

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: out parameter for np.fromfile

hmaarrfk
Thanks Marten.

I tried memap for a few things but it seemed to create an other OS level buffer in specific situations. I think the `seek` operation in the `memmap` also caused some performance bottlenecks. Maybe I'll have time to summarize my findings an other day.

The particular usecase of `ffmpeg` is tricky since it is grabbing a lot of data from `stdout` which isn't a typical file buffer. Specifically, it is often `buffered` but `unseekable`.

On Mon, Nov 5, 2018 at 9:38 AM Marten van Kerkwijk <[hidden email]> wrote:
Hi Mark,

Having an `out` might make sense. With present numpy, if you are really dealing with a file or file-like object, you might consider using `np.memmap` to access the data more directly. If it is something that looks more like a buffer, `np.frombuffer` may be useful (that doesn't copy data, but points the array at the memory that holds the buffer).

All the best,

Marten


On Sun, Nov 4, 2018 at 10:35 PM Mark Harfouche <[hidden email]> wrote:

I was wondering what would your thoughts be on adding an output parameter to np.fromfile?

The advantage would be when interfacing with executables like ffmpeg which are arguably easier to use by calling them as a subprocess compared to a shared library in python.

Having the output parameter in np.fromfile would enable pre-allocation of large arrays that are reused during the computation of new image frames when decoding large video files.

Thoughts are appreciated!

Mark

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion