New project : Spyke python-to-C compiler

[Rahul Garg]

Note this message has been posted to numpy-discussion and python-dev.
Sorry for the multiple posting but I thought both python devs and
numpy users will be interested. If you believe your list should not  
receive this email, let me know. Also I just wanted to introduce  
myself since I may ask doubts about Python and Numpy internals from  
time to time :)

Hi.

I am a student at Univ of Alberta doing my masters in computing science.
I am writing a Python-to-C compiler as one part of my thesis.
The compiler, named Spyke, will be made available in a couple of weeks
and is geared towards scientific applications and will therefore focus  
mostly on needs of scientific app developers.

What is Spyke?
In many performance critical projects, it is often necessary to
rewrite parts of the application in C. However writing C wrappers can
be time consuming. Spyke offers an alternative approach. You add  
annotations to your  Python code as strings. These strings are  
discarded by the Python
interpreter but these are interpreted as types by Spyke compiler to
convert to C.

Example :

"int -> int"
def f(x): return 2*x

In this case the Spyke compiler will consider the string "int -> int"
as a decalration that the function accepts int as parameter and
returns int. Spyke will then generate a C function and a wrapper  
function. This
idea is directly copied from PLW (Python Language Wrapper) project.
Once Python3k arrives, much of these declarations will be moved to
function annotations and class decorators.

This way you can do all your development and debugging interactively
using the standard Python interpreter. When you need to compile to C,
you just add type annotations to places that you want to convert and
invoke spyke on the annotated module. This is different from Pyrex
because Pyrex does not accept Python code. With Spyke, your code is
100% pure python.

Spyke has basic support for functions and classes. Spyke can do very  
basic type inference for local variables in function bodies. Spyke  
also has
partial support for homogenous lists and dictionaries and fixed length tuples.
One big advantage of Spyke is that it understands at least part of  
numpy. Numpy arrays  are treated as fundamental types and Spyke knows  
what C code to
generate for slicing/indexing of numpy arrays etc. This should help a
lot in scientific applications. Note that Spyke can handle only a
subset of Python. Exceptions, iterators, generators, runtime code
generation of any kind etc is not handled. Nested functions will be  
added soon. I will definitely add some of these missing features based  
on what is actually required for real world Python codes. Currently if  
Spyke does not understand a function, it  just leaves it as Python  
code. Classes can be handled but special
methods are not currently supported. The support of classes is a
little brittle because I am trying to resolve some issues b/w old and
new style of classes.

Where is Spyke?
Spyke will be available as a binary only release in a couple of weeks.
I intend to make it open source after a few months.
Spyke is written in Python and Java and should be platform independant.
I do intend to make the source open in a few months. Right now its
undergoing very rapid development and has negligible amounts of  
documentation so the source code right now is pretty useless to anyone  
else anyway.

I need help:
However I need a bit of help. I am having a couple of problems :
a) I am finding it hard to get pure Python+NumPy testing codes. I need
more codes to test the compiler. Developing a compiler without a  
test-suite is kind of useless. If you have some pure Python codes  
which need better performance, please contact me. I guarantee that  
your codes will not be released to public without your permission but  
might be referenced in academic publications. I can also make the  
compiler available to you hopefully after 10th of April. Its kind of  
unstable currently. I will also need your help in annotating the  
provided testing codes since I probably wont know what your  
application is doing.

b) Libraries which interface with C/C++ : Many codes in SciPy for
instance have mixed language codes. Part of the code is written in
C/C++. Spyke only knows how to annotated Python codes. For C/C++
libraries wrapped into Python modules, Spyke will therefore need to
know at least 2 things :
i) The mapping of a C function name/struct etc  to Python
ii) The type information of the said C function.

There are many many ways that people interact with C code. People
either write wrappers manually, or use autogenerated wrappers using
SWIG or SIP Boost.Python etc., use Pyrex or Cython while some people
use ctypes. I dont have the time or resources to support these
multitude of methods. I considered trying to parse the C code  
implementing wrappers but its "non-trivial" to put it mildly. Parsing  
only SWIG generated code is another possibility but its still hard.  
Another approach that I am seriously considering is to support a  
subset of ctypes (with additional restriction) instead. But my  
question is : Is  ctypes good enough for most of you? Ctypes cannot  
interface with C++  code but its pure Python. However I have not seen  
too many instances of people using ctypes.

c) Strings as type declarations : Do you think I should use decorators
instead at least for function type declarations?

thanks for patiently reading this,
comments and inquiries sought.
rahul
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Travis Oliphant-5]

Rahul Garg wrote:
> Note this message has been posted to numpy-discussion and python-dev.
> Sorry for the multiple posting but I thought both python devs and
> numpy users will be interested. If you believe your list should not  
> receive this email, let me know. Also I just wanted to introduce  
> myself since I may ask doubts about Python and Numpy internals from  
> time to time :)
>  
Hey Rahul,

This is a very interesting project.   I've been interested in something
like this for a while.   I'd love to see more about what you are doing.

Best regards,

-Travis Oliphant

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Travis Oliphant-5]

What will be the licensing of this project?  Do you know yet?

I have a couple of comments because I've been thinking along these lines.

> What is Spyke?
> In many performance critical projects, it is often necessary to
> rewrite parts of the application in C. However writing C wrappers can
> be time consuming. Spyke offers an alternative approach. You add  
> annotations to your  Python code as strings. These strings are  
> discarded by the Python
> interpreter but these are interpreted as types by Spyke compiler to
> convert to C.
>
> Example :
>
> "int -> int"
> def f(x): return 2*x
>
> In this case the Spyke compiler will consider the string "int -> int"
> as a decalration that the function accepts int as parameter and
> returns int. Spyke will then generate a C function and a wrapper  
> function.

What about the use of decorators in this case?

Also, it would be great to be able to create ufuncs (and general numpy
funcs) using this approach.   A decorator would work well here as well.
> Where is Spyke?
> Spyke will be available as a binary only release in a couple of weeks.
> I intend to make it open source after a few months.
>  
I'd like to encourage you to make it available as open source as early
as possible.    I think you are likely to get help in ways you didn't
expect.  People are used to reading code, so even an alpha project can
get help early.     In fact given that you are looking for help.  I
think this may be the best way to get it.

If you need help getting set up in terms of hosting it somewhere, I can
help you do that.
> Spyke is written in Python and Java and should be platform independant.
> I do intend to make the source open in a few months. Right now its
> undergoing very rapid development and has negligible amounts of  
> documentation so the source code right now is pretty useless to anyone  
> else anyway.
>  

> c) Strings as type declarations : Do you think I should use decorators
> instead at least for function type declarations?
>  
I think you should use decorators.   That way you can work towards
having the compiler "embedded" in the decorator and happen seamlessly
without invoking a separte "program" (it just happens when the module is
loaded -- a.l.a weave).

Best regards,

-Travis Oliphant



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Gael Varoquaux]

Hi Rahul,

Nice project. I think you are taking the right direction with type
annotations. I you get this working and reliable, you will be much loved
by the community.

On Sun, Apr 06, 2008 at 11:19:58PM -0500, Travis E. Oliphant wrote:
> > c) Strings as type declarations : Do you think I should use decorators
> > instead at least for function type declarations?

> I think you should use decorators.   That way you can work towards
> having the compiler "embedded" in the decorator and happen seamlessly
> without invoking a separte "program" (it just happens when the module is
> loaded -- a.l.a weave).

+1. This is a very promising route. You can then choose exactly what you
want to compile and what you want to keep pure Python (something similar
to Cython, without the intermediate file).

I would even stick with some decoration after Python3K, say
"@compiled()", so that you keep this compilation on the fly that Travis
is mentionning.

Cheers,

Gaël
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Dag Sverre Seljebotn]

> What is Spyke?
> In many performance critical projects, it is often necessary to
> rewrite parts of the application in C. However writing C wrappers can
> be time consuming. Spyke offers an alternative approach. You add
> annotations to your  Python code as strings. These strings are
> discarded by the Python
> interpreter but these are interpreted as types by Spyke compiler to
> convert to C.

Have you had a look at Cython? http://cython.org.

>From seeing what you write, it looks like we have almost exactly the same
long-term goals; one could almost say that the two pieces of software will
be complete duplicates in functionality. (Cython isn't "there" just yet
though.)

(Though as the saying goes, little duplication is normal (and perhaps
wanted) for open source software.)

Dag Sverre

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Dag Sverre Seljebotn]

> (Though as the saying goes, little duplication is normal (and perhaps
> wanted) for open source software.)

Sorry! I meant "a little", completely reversing the meaning of my sentence.

Dag Sverre

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Michael McLay-2]

On Sun, Apr 6, 2008 at 8:48 PM, Rahul Garg <[hidden email]> wrote:

>  function. This
>  idea is directly copied from PLW (Python Language Wrapper) project.
>  Once Python3k arrives, much of these declarations will be moved to
>  function annotations and class decorators.

Python 3k alpha4 is available. Why not skip the string based version
and go directly to using the function annotation capability of 3.0.
Your project could be a good test case for the new feature of the
language.
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion

[Terry Reedy]

"Rahul Garg" <[hidden email]> wrote in message
news:[hidden email]...
| Note this message has been posted to numpy-discussion and python-dev.
| Sorry for the multiple posting but I thought both python devs and
| numpy users will be interested. If you believe your list should not
| receive this email, let me know. Also I just wanted to introduce
| myself since I may ask doubts about Python and Numpy internals from
| time to time :)

Pydev is for discussion about development of  future versions of Python
(ie, 2.6 and some of 3.0).  I think this would have been better posted on
comp.lang.python (or gmane.comp.python.general or the corresponding mailing
list).  You can get answers about current Python internals there.



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion