Re: failure building numpy using icc

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Glen W. Mabey
Robert,

I do appreciate your response to this question (oh way back when) and am
just now getting back to this problem.

On Sat, Mar 01, 2008 at 02:17:46AM -0600, Robert Kern wrote:

> On Thu, Feb 28, 2008 at 1:21 PM, Glen W. Mabey <[hidden email]> wrote:
> > Hello,
> >
> >  I'm using svn numpy and get the following error upon executing
> >
> >   /usr/local/bin/python2.5 setup.py config --noisy --cc=/opt/intel/cce/10.0.025/bin/icc --compiler=intel --fcompiler=intel build_clib build_ext
> >
> >  I see:
> >
> >  conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc
> >  Traceback (most recent call last):
> >   File "setup.py", line 96, in <module>
> >     setup_package()
> >   File "setup.py", line 89, in setup_package
> >     configuration=configuration )
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/core.py", line 184, in setup
> >     return old_setup(**new_attr)
> >   File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup
> >     dist.run_commands()
> >   File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands
> >     self.run_command(cmd)
> >   File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
> >     cmd_obj.run()
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_ext.py", line 56, in run
> >     self.run_command('build_src')
> >   File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command
> >     self.distribution.run_command(command)
> >   File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
> >     cmd_obj.run()
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 130, in run
> >     self.build_sources()
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 147, in build_sources
> >     self.build_extension_sources(ext)
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 252, in build_extension_sources
> >     sources = self.template_sources(sources, ext)
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/command/build_src.py", line 359, in template_sources
> >     outstr = process_c_file(source)
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 185, in process_file
> >     % (sourcefile, process_str(''.join(lines))))
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 150, in process_str
> >     newstr[sub[0]:sub[1]], sub[4])
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 117, in expand_sub
> >     % (line, template_re.sub(namerepl, substr)))
> >   File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080228_svn/numpy/distutils/conv_template.py", line 113, in namerepl
> >     return names[name][thissub[0]]
> >  KeyError: 'PREFIX'
> >
> >
> >  And I do not see any errors when building the same svn version with gcc (on
> >  a different machine).
> >
> >  I've unsuccessfully tried to follow that backtrace of functions to
> >  figure out exactly what is going on.
> >
> >  Any hints/suggestions?
>
> Off-hand, no, sorry. I'm not sure why the compiler would matter in
> this part of the code, though. Can you try using gcc on the same
> machine?

I did successfully use gcc on the same machine without any problems.
However, I still encounter a similar error with today's svn.

Now, I don't know anything about how code is produced within numpy, but
within pdb I was able to find this string that some sort of pattern
substitution is acting on.  It suspiciously looks like the first function
belongs after the last one ...



static PyObject *
gentype_@name@(PyObject *m1)
{
        PyObject *arr, *ret;

        arr = PyArray_FromScalar(m1, NULL);
        if (arr == NULL) return NULL;
        ret = arr->ob_type->tp_as_number->nb_@name@(arr);
        Py_DECREF(arr);
        return ret;
}
/**end repeat**/

static int
gentype_nonzero_number(PyObject *m1)
{
        PyObject *arr;
        int ret;

        arr = PyArray_FromScalar(m1, NULL);
        if (arr == NULL) return -1;
        ret = arr->ob_type->tp_as_number->nb_nonzero(arr);
        Py_DECREF(arr);
        return ret;
}

static PyObject *
gentype_str(PyObject *self)
{
        PyArrayObject *arr;
        PyObject *ret;

        arr = (PyArrayObject *)PyArray_FromScalar(self, NULL);
        if (arr==NULL) return NULL;
        ret = PyObject_Str((PyObject *)arr);
        Py_DECREF(arr);
        return ret;
}


static PyObject *
gentype_repr(PyObject *self)
{
        PyArrayObject *arr;
        PyObject *ret;

        arr = (PyArrayObject *)PyArray_FromScalar(self, NULL);
        if (arr==NULL) return NULL;
        ret = PyObject_Str((PyObject *)arr);
        Py_DECREF(arr);
        return ret;
}
/**begin repeat

#name=float, double, longdouble#
#NAME=FLOAT, DOUBLE, LONGDOUBLE#
#PREFIX=NPY_,NPY_,NPY_#
*/
static void
format_@name@(char *buf, size_t buflen, @name@ val,
                  unsigned int precision)
{
        char *cp;

        PyOS_snprintf(buf, buflen, "%.*" @PREFIX@@NAME@_FMT, precision, val);
        cp = buf;
        if (*cp == '-')
                cp++;
        for (; *cp != '\0'; cp++) {
                if (!isdigit(Py_CHARMASK(*cp)))
                        break;
        }
        if (*cp == '\0') {
                *cp++ = '.';
                *cp++ = '0';
                *cp++ = '\0';
        }
}



And the command I ran to get to this point was:

  python setup.py config --noisy --cc=/opt/intel/cce/10.0.025/bin/icc --compiler=intel --fcompiler=intel build_clib build_ext

Thank you,
Glen Mabey
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Charles R Harris


On Tue, Apr 29, 2008 at 9:21 AM, Glen W. Mabey <[hidden email]> wrote:
Robert,
<snip>

Could you post the new output? The conv_template routine has changed a bit, you should make sure you have the latest version. Also, it is pure python and the use of icc shouldn't make a difference in substituting the keys in the template to generate the source files. Also try deleting the build directory before the new build.

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Glen W. Mabey
On Tue, Apr 29, 2008 at 10:41:25AM -0500, Charles R Harris wrote:
> Could you post the new output? The conv_template routine has changed a bit, you should make sure you have the latest version. Also, it is pure python and the use of icc shouldn't make a difference in substituting the keys in the template to generate the source files. Also try deleting the build directory before the new build.

Okay, I deleted the build directory, and here is the full output:


executing numpy/core/code_generators/generate_array_api.py
  adding 'build/src.linux-x86_64-2.5/numpy/core/__multiarray_api.h' to sources.
creating build/src.linux-x86_64-2.5/numpy/core/src
conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc
Traceback (most recent call last):
  File "setup.py", line 96, in <module>
    setup_package()
  File "setup.py", line 89, in setup_package
    configuration=configuration )
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/core.py", line 184, in setup
    return old_setup(**new_attr)
  File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands
    self.run_command(cmd)
  File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_ext.py", line 56, in run
    self.run_command('build_src')
  File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command
    self.distribution.run_command(command)
  File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
    cmd_obj.run()
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 130, in run
    self.build_sources()
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 147, in build_sources
    self.build_extension_sources(ext)
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 252, in build_extension_sources
    sources = self.template_sources(sources, ext)
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 359, in template_sources
    outstr = process_c_file(source)
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 252, in process_file
    code = process_str(''.join(lines))
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 222, in process_str
    code.extend(parse_string(astr, global_names, 0, 1))
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 210, in parse_string
    newcode = parse_string(text, newenv, newlevel, newline)
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 216, in parse_string
    code.append(replace_re.sub(replace, astr))
  File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 187, in replace
    raise KeyError, msg
KeyError: "#line 497\n: 'PREFIX'"



Thank you,
Glen Mabey
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Charles R Harris


On Tue, Apr 29, 2008 at 9:50 AM, Glen W. Mabey <[hidden email]> wrote:
On Tue, Apr 29, 2008 at 10:41:25AM -0500, Charles R Harris wrote:
> Could you post the new output? The conv_template routine has changed a bit, you should make sure you have the latest version. Also, it is pure python and the use of icc shouldn't make a difference in substituting the keys in the template to generate the source files. Also try deleting the build directory before the new build.

Okay, I deleted the build directory, and here is the full output:


executing numpy/core/code_generators/generate_array_api.py
 adding 'build/src.linux-x86_64-2.5/numpy/core/__multiarray_api.h' to sources.
creating build/src.linux-x86_64-2.5/numpy/core/src
conv_template:> build/src.linux-x86_64-2.5/numpy/core/src/scalartypes.inc
Traceback (most recent call last):
 File "setup.py", line 96, in <module>
   setup_package()
 File "setup.py", line 89, in setup_package
   configuration=configuration )
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/core.py", line 184, in setup
   return old_setup(**new_attr)
 File "/usr/local/lib/python2.5/distutils/core.py", line 151, in setup
   dist.run_commands()
 File "/usr/local/lib/python2.5/distutils/dist.py", line 974, in run_commands
   self.run_command(cmd)
 File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
   cmd_obj.run()
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_ext.py", line 56, in run
   self.run_command('build_src')
 File "/usr/local/lib/python2.5/distutils/cmd.py", line 333, in run_command
   self.distribution.run_command(command)
 File "/usr/local/lib/python2.5/distutils/dist.py", line 994, in run_command
   cmd_obj.run()
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 130, in run
   self.build_sources()
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 147, in build_sources
   self.build_extension_sources(ext)
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 252, in build_extension_sources
   sources = self.template_sources(sources, ext)
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/command/build_src.py", line 359, in template_sources
   outstr = process_c_file(source)
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 252, in process_file
   code = process_str(''.join(lines))
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 222, in process_str
   code.extend(parse_string(astr, global_names, 0, 1))
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 210, in parse_string
   newcode = parse_string(text, newenv, newlevel, newline)
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 216, in parse_string
   code.append(replace_re.sub(replace, astr))
 File "/home/gmabey/src/DiamondBack/Diamondback/src/numpy-20080429_svn/numpy/distutils/conv_template.py", line 187, in replace
   raise KeyError, msg
KeyError: "#line 497\n: 'PREFIX'"


Hmm, something must be mucking with the sources. You can run conv_template on the pure file from the src directory with

 python ../../distutils/conv_template.py scalartypes.inc.src

However, when called from within python the routine also processes the include files and I suspect that may be the problem. Try going into distutils/conv_template and find the line

include_src_re = re.compile(r"(\n|\A)#include\s*['\"]"
                            r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)


and replace it with

include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]"
                            r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)

Which should knock off any whitespace before #include. This is not a fix, but it might help us find the problem.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Glen W. Mabey
On Tue, Apr 29, 2008 at 11:24:56AM -0500, Charles R Harris wrote:

> Hmm, something must be mucking with the sources. You can run conv_template on the pure file from the src directory with
>
>  python ../../distutils/conv_template.py scalartypes.inc.src
>
> However, when called from within python the routine also processes the include files and I suspect that may be the problem. Try going into distutils/conv_template and find the line
>
> include_src_re = re.compile(r"(\n|\A)#include\s*['\"]"
>                             r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)
>
>
> and replace it with
>
> include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]"
>                             r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)
>
> Which should knock off any whitespace before #include. This is not a fix, but it might help us find the problem.

Humm.  I couldn't see any difference in the behavior.  Is there
something that I should be looking for?

Glen
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Charles R Harris


On Tue, Apr 29, 2008 at 11:19 AM, Glen W. Mabey <[hidden email]> wrote:
On Tue, Apr 29, 2008 at 11:24:56AM -0500, Charles R Harris wrote:
> Hmm, something must be mucking with the sources. You can run conv_template on the pure file from the src directory with
>
>  python ../../distutils/conv_template.py scalartypes.inc.src
>
> However, when called from within python the routine also processes the include files and I suspect that may be the problem. Try going into distutils/conv_template and find the line
>
> include_src_re = re.compile(r"(\n|\A)#include\s*['\"]"
>                             r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)
>
>
> and replace it with
>
> include_src_re = re.compile(r"(\n|\A)\s*#include\s*['\"]"
>                             r"(?P<name>[\w\d./\\]+[.]src)['\"]", re.I)
>
> Which should knock off any whitespace before #include. This is not a fix, but it might help us find the problem.

Humm.  I couldn't see any difference in the behavior.  Is there
something that I should be looking for?

Not really, it was just a shot in the dark. The thing is, if conv_python is being run on the same source each time, it should give the same result. So I suspect that something is changing the sources and that is what I'm trying to track down. One of the build guys should know...

Hmmm..., I notice that there is a line break in line 18 of scalartypes.inc.src that doesn't belong there. Could you fix that?

And the line number of the loop is 497, which is the loop before the one with PREFIX defined. For some reason the parser is missing the  /**end repeat**/ on line 509. Hmmm, try putting a blank line before it.

Just fishing here... Why that should happen with icc and not with gnu is a mystery.

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Glen W. Mabey
On Tue, Apr 29, 2008 at 01:01:55PM -0500, Charles R Harris wrote:
> Not really, it was just a shot in the dark. The thing is, if conv_python is being run on the same source each time, it should give the same result. So I suspect that something is changing the sources and that is what I'm trying to track down. One of the build guys should know...
>
> Hmmm..., I notice that there is a line break in line 18 of scalartypes.inc.src that doesn't belong there. Could you fix that?

Okay, got that one.

> And the line number of the loop is 497, which is the loop before the one with PREFIX defined. For some reason the parser is missing the  /**end repeat**/ on line 509. Hmmm, try putting a blank line before it.

Putting a blank line before it didn't do the trick.  

I have been looking at the python code that should have found the /**end
repeat**/ mark and I can't understand why it didn't find it.  In fact,
watch this great trick in pdb:

  ipdb> text[265:281]
  '/**end repeat**/'
  ipdb> text.find( text[265:281] )
  -1

Isn't that cool?  I can only assume that it is a compiler bug and I will
have to upgrade to a newer version of icc (I'm using 10.0.025, actually
it's cce).

After I do that, I'll post again if I have trouble.

Thanks for your help.

Glen
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: failure building numpy using icc

Glen W. Mabey
On Tue, Apr 29, 2008 at 04:43:09PM -0500, Glen W. Mabey wrote:
> Isn't that cool?  I can only assume that it is a compiler bug and I will
> have to upgrade to a newer version of icc (I'm using 10.0.025, actually
> it's cce).
>
> After I do that, I'll post again if I have trouble.

Just to follow up, compiling python with 10.1.015 resolved the problem.

Glen
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion