Branching 1.1.x and starting 1.2.x development

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Branching 1.1.x and starting 1.2.x development

Jarrod Millman
Hello everyone,

In response to some of the recent discussion (and confusion)
surrounding my plan for the trunk and 1.2 development over the next
month or two, I propose the following:

1.  Stay with the "original" plan until Thursday, May 22nd.  That is,
all commits to the trunk should be extremely trivial, bug-fixes for
issues that specifically arise during from the release candidate.  (I
think that there is no need to revert any changes that have happened
on the trunk up to this point, but let's be more conservative going
forward over the next 2 days.)
2.  On Thursday, May 22nd, I will create a 1.1.x branch off the trunk.
 I will tag 1.1.0 off the branch.  The trunk will become open for
development of the 1.2.x series.

Commits to the 1.1.x branch should only be relatively trivial
bug-fixes.  Ideally, 1.1.1 will be practically identical to 1.1.0 with
only a handful of important bug-fixes.  More specifically, this means
that

1. There will be no new features added to the 1.1.x series.
2.  There will be only very minor documentation fixes to the 1.1.x
series.  Only if the documentation has something so incorrect that it
is considered a bug will it be updated.  In particular, most of
Stefan's work will go into 1.2.x.
3.  There will be only very trivial bug-fixes in the 1.1.x series.  I
would expect no more than, say, 10 bug-fixes in a given
micro/maintenance release.  This is just a rule of thumb; but given
our current development size, I would prefer to see most of our effort
focused on the 1.2.x series.  Given that 1.2.0 will be released by the
end of August, this shouldn't cause any major problems for our users.
Moreover, it will help ensure that if they upgrade from 1.1.x to
1.1.x+1 that there will be almost no chance that their code will
break.

Commits to the trunk (1.2.x) should follow these rules:

1.  Documentation fixes are allowed and strongly encouraged.
2.  Bug-fixes are strongly encouraged.
3.  Do not break backwards compatibility.
4.  New features are permissible.
5.  New tests are highly desirable.
6.  If you add a new feature, it must have tests.
7.  If you fix a bug, it must have tests.

If you want to break a rule, don't.  If you feel you absolutely have
to, please don't--but feel free send an email to the list explain your
problem.

In addition to these rules for 1.2 development, let me remind everyone
that we have all ready agreed that 1.2 will:

1.  Use the nose testing framework.
2.  Require Python 2.4 or greater.  This means we have built-in
decorators, set objects, generators, etc.
3.  Contain some planned changes to median and histogram.

I hope this is more clear than the numerous partial emails I sent out
before.  Again let me apologize for the earlier confusion.  Please let
me know if this is a workable plan for you.  In particular, let me
know it there is some aspect of this that you simply refuse to agree
to in at least principle.  Also at this point let's focus on the
overall picture.  Namely, that (1) the trunk is currently is a
relatively hard freeze and that (2) I will create a 1.1.x branch on
Thursday and open the trunk for 1.2 development.  The other details
should be viewed as explanatory narrative that can be refined later.

Unless there are major objections to this proposal, I will take some
time later this week to make this information available on the wiki.

Thanks,

--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Stéfan van der Walt
2008/5/20 Jarrod Millman <[hidden email]>:
> In response to some of the recent discussion (and confusion)
> surrounding my plan for the trunk and 1.2 development over the next
> month or two, I propose the following:

Thank you for the clarification, Jarrod.  Your plan is sound, I'm on board.

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Pearu Peterson-3
In reply to this post by Jarrod Millman
On Tue, May 20, 2008 12:59 pm, Jarrod Millman wrote:

> Commits to the trunk (1.2.x) should follow these rules:
>
> 1.  Documentation fixes are allowed and strongly encouraged.
> 2.  Bug-fixes are strongly encouraged.
> 3.  Do not break backwards compatibility.
> 4.  New features are permissible.
> 5.  New tests are highly desirable.
> 6.  If you add a new feature, it must have tests.
> 7.  If you fix a bug, it must have tests.
>
> If you want to break a rule, don't.  If you feel you absolutely have
> to, please don't--but feel free send an email to the list explain your
> problem.
...
> In particular, let me know it there is some aspect of this that
> you simply refuse to agree to in at least principle.

Since you asked, I have a problem with the rule 7 when applying
it to packages like numpy.distutils and numpy.f2py, for instance.

Do you realize that there exists bugs/features for which unittests cannot
be written in principle? An example: say, a compiler vendor changes
a flag of the new version of the compiler so that numpy.distutils
is not able to detect the compiler or it uses wrong flags for the
new compiler when compiling sources. Often, the required fix
is trivial to find and apply, also just reading the code one can
easily verify that the patch does not break anything. However, to
write a unittest covering such a change would mean that one needs
to ship also the corresponding compiler to the unittest directory.
This is nonsense, of course. I can find other similar examples
that have needed attention and changes to numpy.distutils and
numpy.f2py in past and I know that few are coming up.

Pearu

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

cdavid
Pearu Peterson wrote:
>
> Since you asked, I have a problem with the rule 7 when applying
> it to packages like numpy.distutils and numpy.f2py, for instance.
>  

Although Jarrod did not mention it, I think everybody agrees that
distutils is not really testable, by its nature and design. But that's
even more a reason to be careful when changing it just before a release.

I don't see why f2py would not be testable by nature, though. A big part
of it is parsing fortran, right ?

> Often, the required fix
> is trivial to find and apply, also just reading the code one can
> easily verify that the patch does not break anything.

IMHO, the most useful aspect of unit tests is not testing that the fix
works, but being sure that if it breaks again, it will be detected
(regression test). This is specially important when refactoring.

David
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Stéfan van der Walt
In reply to this post by Pearu Peterson-3
2008/5/20 Pearu Peterson <[hidden email]>:

>> 7.  If you fix a bug, it must have tests.
>
> Since you asked, I have a problem with the rule 7 when applying
> it to packages like numpy.distutils and numpy.f2py, for instance.
>
> Do you realize that there exists bugs/features for which unittests cannot
> be written in principle? An example: say, a compiler vendor changes
> a flag of the new version of the compiler so that numpy.distutils
> is not able to detect the compiler or it uses wrong flags for the
> new compiler when compiling sources. Often, the required fix
> is trivial to find and apply, also just reading the code one can
> easily verify that the patch does not break anything. However, to
> write a unittest covering such a change would mean that one needs
> to ship also the corresponding compiler to the unittest directory.
> This is nonsense, of course. I can find other similar examples
> that have needed attention and changes to numpy.distutils and
> numpy.f2py in past and I know that few are coming up.

My earlier message, regarding your patch to trunk, was maybe not
clearly phrased.  I didn't want to start a unit-testing discussion,
and was simply trying to say "if we apply patches now, we should make
sure they work".  You did that, so I was happy.  I've been the main
transgressor in applying new changes to trunk; sorry about the
misunderstanding, Jarrod.

As for your comment above: yes, writing unit tests is hard.  As you
mentioned, sometimes changes are trivial, difficult to test and you
can see they work.  If the code was already exercised in the test
suite, I would be less worried about such trivial changes.  Then, at
least, we know that the code is executed every time the test suite
runs, so if a person forgot to close a parenthesis or a string quote,
it would be picked up.

The case I describe above is exceptional (but it does occur much more
frequently in f2py and distutils).  Still, I would not say that those
tests are impossible to write, just that they require thinking outside
the box.  For example, it would be quite feasible to have a set of
small Python scripts that pretend to be compilers, to assist in
asserting that the flags we think we pass are the ones that reach the
compiler.  Similarly, a fake directory tree can be created to help
verify that `site.cfg` is correctly parsed and applied.  You are
right: some factors are out of our control, but we need to test for
every piece of functionality that isn't.

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Jarrod Millman
In reply to this post by Pearu Peterson-3
On Tue, May 20, 2008 at 3:47 AM, Pearu Peterson <[hidden email]> wrote:

> On Tue, May 20, 2008 12:59 pm, Jarrod Millman wrote:
>
>> Commits to the trunk (1.2.x) should follow these rules:
>>
>> 1.  Documentation fixes are allowed and strongly encouraged.
>> 2.  Bug-fixes are strongly encouraged.
>> 3.  Do not break backwards compatibility.
>> 4.  New features are permissible.
>> 5.  New tests are highly desirable.
>> 6.  If you add a new feature, it must have tests.
>> 7.  If you fix a bug, it must have tests.
>>
>> If you want to break a rule, don't.  If you feel you absolutely have
>> to, please don't--but feel free send an email to the list explain your
>> problem.
> ...
>> In particular, let me know it there is some aspect of this that
>> you simply refuse to agree to in at least principle.
>
> Since you asked, I have a problem with the rule 7 when applying
> it to packages like numpy.distutils and numpy.f2py, for instance.

I obviously knew this would be controversial.  Personally, I would
prefer if we boldly state that tests are required.  I know that this
"rule" will be broken occasionally and may not even always make sense.
 We could  change the language to be something like "if you a fix a
bug, there should be tests".  Saying "you must" means that you're
breaking a rule when you don't.  Importantly I am not proposing that
we have some new enforcement mechanism; I am happy to leave this to
Stefan and whoever else wants to join him. (Thanks Stefan--you have
taken on a tough job!)

However, let's not worry too much about this at this point.  Let's get
1.1.0 out.  I think everyone agrees that unit tests are a good idea,
some are more passionate than others.  We need to figure out how best
to increase the number of tests, but we are doing a great job
currently.  NumPy 1.0.4 had around 686 tests and the trunk now has
roughly 996 tests.

For now, let's officially consider rules 6 and 7 to have question
marks at the end of them.  Once 1.1.0 is out and we have started
developing 1.2, we can start this conversation again.

Thanks,

--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris


On Tue, May 20, 2008 at 6:11 AM, Jarrod Millman <[hidden email]> wrote:
On Tue, May 20, 2008 at 3:47 AM, Pearu Peterson <[hidden email]> wrote:
> On Tue, May 20, 2008 12:59 pm, Jarrod Millman wrote:
>
>> Commits to the trunk (1.2.x) should follow these rules:
>>
>> 1.  Documentation fixes are allowed and strongly encouraged.
>> 2.  Bug-fixes are strongly encouraged.
>> 3.  Do not break backwards compatibility.
>> 4.  New features are permissible.
>> 5.  New tests are highly desirable.
>> 6.  If you add a new feature, it must have tests.
>> 7.  If you fix a bug, it must have tests.
>>
>> If you want to break a rule, don't.  If you feel you absolutely have
>> to, please don't--but feel free send an email to the list explain your
>> problem.
> ...
>> In particular, let me know it there is some aspect of this that
>> you simply refuse to agree to in at least principle.
>
> Since you asked, I have a problem with the rule 7 when applying
> it to packages like numpy.distutils and numpy.f2py, for instance.

I obviously knew this would be controversial.  Personally, I would
prefer if we boldly state that tests are required.  I know that this
"rule" will be broken occasionally and may not even always make sense.
 We could  change the language to be something like "if you a fix a
bug, there should be tests".  Saying "you must" means that you're
breaking a rule when you don't.  Importantly I am not proposing that
we have some new enforcement mechanism; I am happy to leave this to
Stefan and whoever else wants to join him. (Thanks Stefan--you have
taken on a tough job!)

However, let's not worry too much about this at this point.  Let's get
1.1.0 out.  I think everyone agrees that unit tests are a good idea,
some are more passionate than others.  We need to figure out how best
to increase the number of tests, but we are doing a great job
currently.  NumPy 1.0.4 had around 686 tests and the trunk now has
roughly 996 tests.

For now, let's officially consider rules 6 and 7 to have question
marks at the end of them.  Once 1.1.0 is out and we have started
developing 1.2, we can start this conversation again.

Two of the buildbots are showing problems, probably installation related, but it would be nice to see all green before the release.

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Jarrod Millman
On Tue, May 20, 2008 at 8:37 AM, Charles R Harris
<[hidden email]> wrote:
> Two of the buildbots are showing problems, probably installation related,
> but it would be nice to see all green before the release.

Absolutely.  Thanks for catching this.

--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris


On Tue, May 20, 2008 at 9:48 AM, Jarrod Millman <[hidden email]> wrote:
On Tue, May 20, 2008 at 8:37 AM, Charles R Harris
<[hidden email]> wrote:
> Two of the buildbots are showing problems, probably installation related,
> but it would be nice to see all green before the release.

Absolutely.  Thanks for catching this.

It would be good if we could find a PPC to add to the buildbot in order to catch endianess problems.
SPARC might also do for this.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Stéfan van der Walt
In reply to this post by Charles R Harris
2008/5/20 Charles R Harris <[hidden email]>:
> Two of the buildbots are showing problems, probably installation related,
> but it would be nice to see all green before the release.

Thanks, fixed in SVN.

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Stéfan van der Walt
In reply to this post by Charles R Harris
2008/5/20 Charles R Harris <[hidden email]>:

>
>
> On Tue, May 20, 2008 at 9:48 AM, Jarrod Millman <[hidden email]>
> wrote:
>>
>> On Tue, May 20, 2008 at 8:37 AM, Charles R Harris
>> <[hidden email]> wrote:
>> > Two of the buildbots are showing problems, probably installation
>> > related,
>> > but it would be nice to see all green before the release.
>>
>> Absolutely.  Thanks for catching this.
>
> It would be good if we could find a PPC to add to the buildbot in order to
> catch endianess problems.
> SPARC might also do for this.

Absolutely!  If anybody has access to such a machine and is willing to
let a build-slave run on it, please let me know and I'll send you the
necessary configuration files.

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Michael Abshoff-2
Stéfan van der Walt wrote:

> 2008/5/20 Charles R Harris <[hidden email]>:
>  
>> On Tue, May 20, 2008 at 9:48 AM, Jarrod Millman <[hidden email]>
>> wrote:
>>    
>>> On Tue, May 20, 2008 at 8:37 AM, Charles R Harris
>>> <[hidden email]> wrote:
>>>      
>>>> Two of the buildbots are showing problems, probably installation
>>>> related,
>>>> but it would be nice to see all green before the release.
>>>>        
>>> Absolutely.  Thanks for catching this.
>>>      
>> It would be good if we could find a PPC to add to the buildbot in order to
>> catch endianess problems.
>> SPARC might also do for this.
>>    
>
> Absolutely!  If anybody has access to such a machine and is willing to
> let a build-slave run on it, please let me know and I'll send you the
> necessary configuration files.
>
>  

Hi Stefan,

I got access to Solaris 9/Sparc and Solaris 10/Sparc and am certainly
willing to help out.

> Regards
> Stéfan
>  

Cheers,

Michael

> _______________________________________________
> Numpy-discussion mailing list
> [hidden email]
> http://projects.scipy.org/mailman/listinfo/numpy-discussion
>
>  

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris
In reply to this post by Stéfan van der Walt


On Tue, May 20, 2008 at 10:30 AM, Stéfan van der Walt <[hidden email]> wrote:
2008/5/20 Charles R Harris <[hidden email]>:
>
>
> On Tue, May 20, 2008 at 9:48 AM, Jarrod Millman <[hidden email]>
> wrote:
>>
>> On Tue, May 20, 2008 at 8:37 AM, Charles R Harris
>> <[hidden email]> wrote:
>> > Two of the buildbots are showing problems, probably installation
>> > related,
>> > but it would be nice to see all green before the release.
>>
>> Absolutely.  Thanks for catching this.
>
> It would be good if we could find a PPC to add to the buildbot in order to
> catch endianess problems.
> SPARC might also do for this.

Absolutely!  If anybody has access to such a machine and is willing to
let a build-slave run on it, please let me know and I'll send you the
necessary configuration files.

The current Mac machine seems to have a configuration problem. If anyone out there has an SGI machine we could use that too. I heart the buildbot.

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Stéfan van der Walt
2008/5/20 Charles R Harris <[hidden email]>:
> The current Mac machine seems to have a configuration problem. If anyone out
> there has an SGI machine we could use that too. I heart the buildbot.

I mailed Barry, he'll reset it for us.  I'm glad the buildbot makes
you happy, Charles :)

Regards
Stéfan
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Chris Barker - NOAA Federal
In reply to this post by Charles R Harris
Charles R Harris wrote:
>     Absolutely!  If anybody has access to such a machine and is willing to
>     let a build-slave run on it, please let me know and I'll send you the
>     necessary configuration files.
>
> The current Mac machine seems to have a configuration problem.

I've got a PPC Mac that I may be able to use -- any particular ports
need to be open -- we've got pretty draconian firewall rules.

-Chris


--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris
Can we close ticket 605, Incorrect Behavior of Numpy Histogram?
http://projects.scipy.org/scipy/numpy/ticket/605

Chuck

_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Jarrod Millman
On Tue, May 20, 2008 at 12:10 PM, Charles R Harris
<[hidden email]> wrote:
> Can we close ticket 605, Incorrect Behavior of Numpy Histogram?
> http://projects.scipy.org/scipy/numpy/ticket/605

Yes, but we need to create a new ticket for 1.2 detailing the planned
changes.  If no one else gets to it, I will do so tonight.

--
Jarrod Millman
Computational Infrastructure for Research Labs
10 Giannini Hall, UC Berkeley
phone: 510.643.4014
http://cirl.berkeley.edu/
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris


On Tue, May 20, 2008 at 1:16 PM, Jarrod Millman <[hidden email]> wrote:
On Tue, May 20, 2008 at 12:10 PM, Charles R Harris
<[hidden email]> wrote:
> Can we close ticket 605, Incorrect Behavior of Numpy Histogram?
> http://projects.scipy.org/scipy/numpy/ticket/605

Yes, but we need to create a new ticket for 1.2 detailing the planned
changes.  If no one else gets to it, I will do so tonight.

How about ticket 770? Did Roberts endianess fixes cover that?

Chuck



_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Robert Kern-2
On Tue, May 20, 2008 at 2:26 PM, Charles R Harris
<[hidden email]> wrote:
> How about ticket 770? Did Roberts endianess fixes cover that?

Yes.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless
enigma that is made terrible by our own mad attempt to interpret it as
though it had an underlying truth."
 -- Umberto Eco
_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Branching 1.1.x and starting 1.2.x development

Charles R Harris
Curious bug on Stefan's Ubuntu build client:

ImportError: /usr/lib/atlas/libblas.so.3gf: undefined symbol: _gfortran_st_write_done
make[1]: *** [test] Error 1
Anyone know what that is about?

Chuck


_______________________________________________
Numpy-discussion mailing list
[hidden email]
http://projects.scipy.org/mailman/listinfo/numpy-discussion
12