Problem with libgfortran installed with pip install numpy

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Problem with libgfortran installed with pip install numpy

Dave Grote

Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.

I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.

If I download the numpy source and do python setup.py install, I don't have this problem.

After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used?
   Thanks!
       Dave

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with libgfortran installed with pip install numpy

Charles R Harris


On Wed, Sep 5, 2018 at 5:38 PM David Grote <[hidden email]> wrote:

Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.

I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.

If I download the numpy source and do python setup.py install, I don't have this problem.

After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used?

The library came in with the use of OpenBLAS, I don't think there is a fundamental reason that a newer version of gfortran couldn't be used, but I have little experience with the Mac. Note that we have also given up on 32 bit Python on Mac for library related reasons. Matthew Brett would be the guy to discuss this with.

Chuck 

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with libgfortran installed with pip install numpy

Dave Grote

Hi Matthew - Do you have any comment in this?
   Thanks!
        Dave


On Wed, Sep 5, 2018 at 5:01 PM Charles R Harris <[hidden email]> wrote:


On Wed, Sep 5, 2018 at 5:38 PM David Grote <[hidden email]> wrote:

Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue.

I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen.

If I download the numpy source and do python setup.py install, I don't have this problem.

After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used?

The library came in with the use of OpenBLAS, I don't think there is a fundamental reason that a newer version of gfortran couldn't be used, but I have little experience with the Mac. Note that we have also given up on 32 bit Python on Mac for library related reasons. Matthew Brett would be the guy to discuss this with.

Chuck 
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with libgfortran installed with pip install numpy

Nathaniel Smith
In reply to this post by Dave Grote
On Wed, Sep 5, 2018 at 4:37 PM, David Grote <[hidden email]> wrote:

>
> Hi - I have recently come across this problem. On my mac, I build a Fortran
> code, producing a shared object, that I import into Python along with numpy.
> This had been working fine until recently when I started seeing sag faults
> deep inside the Fortran code, usually in Fortran print statements. I tracked
> this down to a gfortran version issue.
>
> I use the brew installation of Python and gcc (using the most recent
> version, 8.2.0). gcc of course installs a version of libgfortran.dylib.
> Doing a lsof of a running Python, I see that it finds that copy of
> libgfortran, and also a copy that was downloaded with numpy
> (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib).
> Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much
> older. Since my code is importing numpy first, the OS seems be using numpy's
> version of libgfortran to link when importing my code. I know from other
> experience that older versions of libgfortran are not compatible with code
> compiled using a new version of gfortran and so therefore segfaults happen.

Normally on MacOS, it's fine to have multiple versions of the same
library used at the same time, because the linker looks up symbols
using a (source library, symbol name) pair. (This is called the
"two-level namespace".) So it's strange that these two libgfortrans
would interfere with each other. Does gfortran not use the two-level
namespace when linking fortran code?

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Problem with libgfortran installed with pip install numpy

Dave Grote

Yes, thanks, that's something I hadn't looked into. For legacy reasons, my code was being built with the option -flat_namespace. I don't remember the reason why, but many years ago that option was needed for the code to run on the mac. The code is made up of several shared objects that have dependencies on each other and apparently there was a problem getting it all linked together properly without that option. But, I tried it out now without flat_namespace and it seemed to work fine. Whatever the problem was, it seems to been fixed some other way. It works Ok having the pip version of numpy (with its old libgfortran).

I'm still curious about why such an old version of gfortran is still being used.
   Dave

On Fri, Nov 9, 2018 at 3:09 PM Nathaniel Smith <[hidden email]> wrote:
On Wed, Sep 5, 2018 at 4:37 PM, David Grote <[hidden email]> wrote:
>
> Hi - I have recently come across this problem. On my mac, I build a Fortran
> code, producing a shared object, that I import into Python along with numpy.
> This had been working fine until recently when I started seeing sag faults
> deep inside the Fortran code, usually in Fortran print statements. I tracked
> this down to a gfortran version issue.
>
> I use the brew installation of Python and gcc (using the most recent
> version, 8.2.0). gcc of course installs a version of libgfortran.dylib.
> Doing a lsof of a running Python, I see that it finds that copy of
> libgfortran, and also a copy that was downloaded with numpy
> (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib).
> Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much
> older. Since my code is importing numpy first, the OS seems be using numpy's
> version of libgfortran to link when importing my code. I know from other
> experience that older versions of libgfortran are not compatible with code
> compiled using a new version of gfortran and so therefore segfaults happen.

Normally on MacOS, it's fine to have multiple versions of the same
library used at the same time, because the linker looks up symbols
using a (source library, symbol name) pair. (This is called the
"two-level namespace".) So it's strange that these two libgfortrans
would interfere with each other. Does gfortran not use the two-level
namespace when linking fortran code?

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion