np.genfromtxt StopIteration Error

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

np.genfromtxt StopIteration Error

Stephen P. Molnar
I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach.

Here is the code:


import os
import glob
import numpy as np

fileList = []
filesList = []

for files in glob.glob("*.log"):
?????? fileName, fileExtension = os.path.splitext(files)
?????? fileList.append(fileName)
?????? filesList.append(files)

print('fileList = ', fileList)
print('filesList = ', filesList)

fname = filesList
print('fname = ', fname)
data = np.genfromtxt(fname, usecols=(1), skip_header=27, skip_footer=1, encoding=None)
print(data)

np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}')
print(data.dG)
I am using the Spyder IDE which has a variable explorer which shows:

filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log']
fileList = ['C-VX3', '18-7', '14-7', '15-7']

so the lists that genfromtxt needs are being generated.

Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied.

I would greatly appreciate some assistance here.

Thanks is advance.
-- 
Stephen P. Molnar, Ph.D.
www.molecular-modeling.net
614.312.7528 (c)
Skype:  smolnar1


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: np.genfromtxt StopIteration Error

Bennet Fauber
I think genfromtxt() wants a filename as the first argument, and you
have to tell it the entries in the file are strings not numerics.

test.py
------------------------------------------
import os
import glob
import numpy as np

fileList = []
filesList = []

for files in glob.glob("*.log"):
    fileName, fileExtension = os.path.splitext(files)
    fileList.append(fileName)
    filesList.append(files)

print('fileList = ', fileList)
print('filesList = ', filesList)

fname = '/tmp/foo.txt'
print('fname = ', fname)
data = np.genfromtxt(fname, dtype=str)
print(data)
------------------------------------------

Contents of /tmp/foo.txt
------------------------------------------
15-7.log
18-7.log
14-7.log
C-VX3.log
------------------------------------------

Sample run

$ python --version
Python 2.7.15+

$ python t.py
('fileList = ', ['15-7', '18-7', '14-7', 'C-VX3'])
('filesList = ', ['15-7.log', '18-7.log', '14-7.log', 'C-VX3.log'])
('fname = ', '/tmp/foo.txt')
['15-7.log' '18-7.log' '14-7.log' 'C-VX3.log']

Is that any help?


On Fri, Oct 11, 2019 at 12:41 PM Stephen P. Molnar
<[hidden email]> wrote:

>
> I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach.
>
> Here is the code:
>
>
> import os
> import glob
> import numpy as np
>
> fileList = []
> filesList = []
>
> for files in glob.glob("*.log"):
> ?????? fileName, fileExtension = os.path.splitext(files)
> ?????? fileList.append(fileName)
> ?????? filesList.append(files)
>
> print('fileList = ', fileList)
> print('filesList = ', filesList)
>
> fname = filesList
> print('fname = ', fname)
> data = np.genfromtxt(fname, usecols=(1), skip_header=27, skip_footer=1, encoding=None)
> print(data)
>
> np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}')
> print(data.dG)
>
> I am using the Spyder IDE which has a variable explorer which shows:
>
> filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log']
> fileList = ['C-VX3', '18-7', '14-7', '15-7']
>
> so the lists that genfromtxt needs are being generated.
>
> Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied.
>
> I would greatly appreciate some assistance here.
>
> Thanks is advance.
>
> --
> Stephen P. Molnar, Ph.D.
> www.molecular-modeling.net
> 614.312.7528 (c)
> Skype:  smolnar1
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: np.genfromtxt StopIteration Error

Stephen P. Molnar
Thanks for the reply.

Keep in mind that i am a Chemist, not an IT person. I used to be a
marginally proficient FORTRAN II user in the ancient past.

I tried running your code. Please see my comments/questing below:

On 10/11/2019 01:12 PM, Bennet Fauber wrote:

> I think genfromtxt() wants a filename as the first argument, and you
> have to tell it the entries in the file are strings not numerics.
>
> test.py
> ------------------------------------------
> import os
> import glob
> import numpy as np
>
> fileList = []
> filesList = []
>
> for files in glob.glob("*.log"):
>      fileName, fileExtension = os.path.splitext(files)
>      fileList.append(fileName)
>      filesList.append(files)
>
> print('fileList = ', fileList)
> print('filesList = ', filesList
>
> fname = '/tmp/foo.txt'
There is no '/temp/foo.txt' Where did it come from in your example?

> print('fname = ', fname)
> data = np.genfromtxt(fname, dtype=str)
> print(data)
> ------------------------------------------
>
> Contents of /tmp/foo.txt
> ------------------------------------------
> 15-7.log
> 18-7.log
> 14-7.log
> C-VX3.log
> ------------------------------------------
>
> Sample run
I'm using python 3.7.3, should this make a difference?

>
> $ python --version
> Python 2.7.15+
>
> $ python t.py
> ('fileList = ', ['15-7', '18-7', '14-7', 'C-VX3'])
> ('filesList = ', ['15-7.log', '18-7.log', '14-7.log', 'C-VX3.log'])
> ('fname = ', '/tmp/foo.txt')
> ['15-7.log' '18-7.log' '14-7.log' 'C-VX3.log']
>
> Is that any help?
if I use data = np.genfromtxt('14-7.log', dtype=str, usecols=(1),
skip_header=27, skip_footer=1, encoding=None) with a specific file name.
in this example 14-7, I get the resutt I desired:

# 14-7
-9.960902669
-8.979504781
-8.942611364
-8.915523010
-8.736508831
-8.663387139
-8.410739711
-8.389146347
-8.296798909
-8.168454106
-8.127990818
-8.127103774
-7.979090739
-7.941872682
-7.900766215
-7.881485228
-7.837826485
-7.815909505
-7.722540286
-7.720346742

so, my question is; why the StopIteration error message in my original
query? Why is the dcrtipt not iterating over the log files?

Sorry to be so dense.


>
> On Fri, Oct 11, 2019 at 12:41 PM Stephen P. Molnar
> <[hidden email]> wrote:
>> I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach.
>>
>> Here is the code:
>>
>>
>> import os
>> import glob
>> import numpy as np
>>
>> fileList = []
>> filesList = []
>>
>> for files in glob.glob("*.log"):
>> ?????? fileName, fileExtension = os.path.splitext(files)
>> ?????? fileList.append(fileName)
>> ?????? filesList.append(files)
>>
>> print('fileList = ', fileList)
>> print('filesList = ', filesList)
>>
>> fname = filesList
>> print('fname = ', fname)
>> data = np.genfromtxt(fname, usecols=(1), skip_header=27, skip_footer=1, encoding=None)
>> print(data)
>>
>> np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}')
>> print(data.dG)
>>
>> I am using the Spyder IDE which has a variable explorer which shows:
>>
>> filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log']
>> fileList = ['C-VX3', '18-7', '14-7', '15-7']
>>
>> so the lists that genfromtxt needs are being generated.
>>
>> Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied.
>>
>> I would greatly appreciate some assistance here.
>>
>> Thanks is advance.
>>
>> --
>> Stephen P. Molnar, Ph.D.
>> www.molecular-modeling.net
>> 614.312.7528 (c)
>> Skype:  smolnar1
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> [hidden email]
>> https://mail.python.org/mailman/listinfo/numpy-discussion
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
>

--
Stephen P. Molnar, Ph.D.
www.molecular-modeling.net
614.312.7528 (c)
Skype:  smolnar1

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: np.genfromtxt StopIteration Error

Andras Deak
Hi Stephen,

Is this not what your original question to this list was about? See
https://mail.python.org/pipermail/numpy-discussion/2019-October/080130.html
and replies.
I still believe that you _can't_ give genfromtxt file names in an
iterable. The iterable input is only inteded to contain the contents
of a single file, probably as if read using `f.readlines()`.
Genfromtxt seems to read data from at most one file with each call.
Regards,

AndrĂ¡s

On Fri, Oct 11, 2019 at 8:41 PM Stephen P. Molnar
<[hidden email]> wrote:

>
> Thanks for the reply.
>
> Keep in mind that i am a Chemist, not an IT person. I used to be a
> marginally proficient FORTRAN II user in the ancient past.
>
> I tried running your code. Please see my comments/questing below:
>
> On 10/11/2019 01:12 PM, Bennet Fauber wrote:
> > I think genfromtxt() wants a filename as the first argument, and you
> > have to tell it the entries in the file are strings not numerics.
> >
> > test.py
> > ------------------------------------------
> > import os
> > import glob
> > import numpy as np
> >
> > fileList = []
> > filesList = []
> >
> > for files in glob.glob("*.log"):
> >      fileName, fileExtension = os.path.splitext(files)
> >      fileList.append(fileName)
> >      filesList.append(files)
> >
> > print('fileList = ', fileList)
> > print('filesList = ', filesList
> >
> > fname = '/tmp/foo.txt'
> There is no '/temp/foo.txt' Where did it come from in your example?
> > print('fname = ', fname)
> > data = np.genfromtxt(fname, dtype=str)
> > print(data)
> > ------------------------------------------
> >
> > Contents of /tmp/foo.txt
> > ------------------------------------------
> > 15-7.log
> > 18-7.log
> > 14-7.log
> > C-VX3.log
> > ------------------------------------------
> >
> > Sample run
> I'm using python 3.7.3, should this make a difference?
> >
> > $ python --version
> > Python 2.7.15+
> >
> > $ python t.py
> > ('fileList = ', ['15-7', '18-7', '14-7', 'C-VX3'])
> > ('filesList = ', ['15-7.log', '18-7.log', '14-7.log', 'C-VX3.log'])
> > ('fname = ', '/tmp/foo.txt')
> > ['15-7.log' '18-7.log' '14-7.log' 'C-VX3.log']
> >
> > Is that any help?
> if I use data = np.genfromtxt('14-7.log', dtype=str, usecols=(1),
> skip_header=27, skip_footer=1, encoding=None) with a specific file name.
> in this example 14-7, I get the resutt I desired:
>
> # 14-7
> -9.960902669
> -8.979504781
> -8.942611364
> -8.915523010
> -8.736508831
> -8.663387139
> -8.410739711
> -8.389146347
> -8.296798909
> -8.168454106
> -8.127990818
> -8.127103774
> -7.979090739
> -7.941872682
> -7.900766215
> -7.881485228
> -7.837826485
> -7.815909505
> -7.722540286
> -7.720346742
>
> so, my question is; why the StopIteration error message in my original
> query? Why is the dcrtipt not iterating over the log files?
>
> Sorry to be so dense.
>
>
> >
> > On Fri, Oct 11, 2019 at 12:41 PM Stephen P. Molnar
> > <[hidden email]> wrote:
> >> I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach.
> >>
> >> Here is the code:
> >>
> >>
> >> import os
> >> import glob
> >> import numpy as np
> >>
> >> fileList = []
> >> filesList = []
> >>
> >> for files in glob.glob("*.log"):
> >> ?????? fileName, fileExtension = os.path.splitext(files)
> >> ?????? fileList.append(fileName)
> >> ?????? filesList.append(files)
> >>
> >> print('fileList = ', fileList)
> >> print('filesList = ', filesList)
> >>
> >> fname = filesList
> >> print('fname = ', fname)
> >> data = np.genfromtxt(fname, usecols=(1), skip_header=27, skip_footer=1, encoding=None)
> >> print(data)
> >>
> >> np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}')
> >> print(data.dG)
> >>
> >> I am using the Spyder IDE which has a variable explorer which shows:
> >>
> >> filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log']
> >> fileList = ['C-VX3', '18-7', '14-7', '15-7']
> >>
> >> so the lists that genfromtxt needs are being generated.
> >>
> >> Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied.
> >>
> >> I would greatly appreciate some assistance here.
> >>
> >> Thanks is advance.
> >>
> >> --
> >> Stephen P. Molnar, Ph.D.
> >> www.molecular-modeling.net
> >> 614.312.7528 (c)
> >> Skype:  smolnar1
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> [hidden email]
> >> https://mail.python.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >
>
> --
> Stephen P. Molnar, Ph.D.
> www.molecular-modeling.net
> 614.312.7528 (c)
> Skype:  smolnar1
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: np.genfromtxt StopIteration Error

Bennet Fauber
In reply to this post by Stephen P. Molnar
What happens if you use this?

for txtfile in filesList:
    data = np.genfromtxt(txtfile, dtype=str, usecols=(1),
        skip_header=27, skip_footer=1, encoding=None)
print(data)

That pulls one file name from filesList, stuffs the name into txtfile,
which is then provided to genfromtxt().

The point is that genfromtxt() wants one thing; you have to give it
one thing.  The above gives it one thing, but runs it for however many
things are in the filesList.

Does that help?

On Fri, Oct 11, 2019 at 2:41 PM Stephen P. Molnar
<[hidden email]> wrote:

>
> Thanks for the reply.
>
> Keep in mind that i am a Chemist, not an IT person. I used to be a
> marginally proficient FORTRAN II user in the ancient past.
>
> I tried running your code. Please see my comments/questing below:
>
> On 10/11/2019 01:12 PM, Bennet Fauber wrote:
> > I think genfromtxt() wants a filename as the first argument, and you
> > have to tell it the entries in the file are strings not numerics.
> >
> > test.py
> > ------------------------------------------
> > import os
> > import glob
> > import numpy as np
> >
> > fileList = []
> > filesList = []
> >
> > for files in glob.glob("*.log"):
> >      fileName, fileExtension = os.path.splitext(files)
> >      fileList.append(fileName)
> >      filesList.append(files)
> >
> > print('fileList = ', fileList)
> > print('filesList = ', filesList
> >
> > fname = '/tmp/foo.txt'
> There is no '/temp/foo.txt' Where did it come from in your example?
> > print('fname = ', fname)
> > data = np.genfromtxt(fname, dtype=str)
> > print(data)
> > ------------------------------------------
> >
> > Contents of /tmp/foo.txt
> > ------------------------------------------
> > 15-7.log
> > 18-7.log
> > 14-7.log
> > C-VX3.log
> > ------------------------------------------
> >
> > Sample run
> I'm using python 3.7.3, should this make a difference?
> >
> > $ python --version
> > Python 2.7.15+
> >
> > $ python t.py
> > ('fileList = ', ['15-7', '18-7', '14-7', 'C-VX3'])
> > ('filesList = ', ['15-7.log', '18-7.log', '14-7.log', 'C-VX3.log'])
> > ('fname = ', '/tmp/foo.txt')
> > ['15-7.log' '18-7.log' '14-7.log' 'C-VX3.log']
> >
> > Is that any help?
> if I use data = np.genfromtxt('14-7.log', dtype=str, usecols=(1),
> skip_header=27, skip_footer=1, encoding=None) with a specific file name.
> in this example 14-7, I get the resutt I desired:
>
> # 14-7
> -9.960902669
> -8.979504781
> -8.942611364
> -8.915523010
> -8.736508831
> -8.663387139
> -8.410739711
> -8.389146347
> -8.296798909
> -8.168454106
> -8.127990818
> -8.127103774
> -7.979090739
> -7.941872682
> -7.900766215
> -7.881485228
> -7.837826485
> -7.815909505
> -7.722540286
> -7.720346742
>
> so, my question is; why the StopIteration error message in my original
> query? Why is the dcrtipt not iterating over the log files?
>
> Sorry to be so dense.
>
>
> >
> > On Fri, Oct 11, 2019 at 12:41 PM Stephen P. Molnar
> > <[hidden email]> wrote:
> >> I have been fighting with the genfromtxt function in numpy for a while now and am trying a slightly different approach.
> >>
> >> Here is the code:
> >>
> >>
> >> import os
> >> import glob
> >> import numpy as np
> >>
> >> fileList = []
> >> filesList = []
> >>
> >> for files in glob.glob("*.log"):
> >> ?????? fileName, fileExtension = os.path.splitext(files)
> >> ?????? fileList.append(fileName)
> >> ?????? filesList.append(files)
> >>
> >> print('fileList = ', fileList)
> >> print('filesList = ', filesList)
> >>
> >> fname = filesList
> >> print('fname = ', fname)
> >> data = np.genfromtxt(fname, usecols=(1), skip_header=27, skip_footer=1, encoding=None)
> >> print(data)
> >>
> >> np.savetxt('fileList.dG', data, fmt='%12.9f', header='${d}')
> >> print(data.dG)
> >>
> >> I am using the Spyder IDE which has a variable explorer which shows:
> >>
> >> filesList = ['C-VX3.log', '18-7.log', '14-7.log', '15-7.log']
> >> fileList = ['C-VX3', '18-7', '14-7', '15-7']
> >>
> >> so the lists that genfromtxt needs are being generated.
> >>
> >> Goggling 'numpy genfromtxt stopiteration error' does not seem to address this problem. At least, I didn't find plaything that I thought applied.
> >>
> >> I would greatly appreciate some assistance here.
> >>
> >> Thanks is advance.
> >>
> >> --
> >> Stephen P. Molnar, Ph.D.
> >> www.molecular-modeling.net
> >> 614.312.7528 (c)
> >> Skype:  smolnar1
> >>
> >> _______________________________________________
> >> NumPy-Discussion mailing list
> >> [hidden email]
> >> https://mail.python.org/mailman/listinfo/numpy-discussion
> > _______________________________________________
> > NumPy-Discussion mailing list
> > [hidden email]
> > https://mail.python.org/mailman/listinfo/numpy-discussion
> >
>
> --
> Stephen P. Molnar, Ph.D.
> www.molecular-modeling.net
> 614.312.7528 (c)
> Skype:  smolnar1
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.python.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion