Numpy 1.9 release date

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Numpy 1.9 release date

Charles R Harris
Hi All,

The question has come up as to how much effort we should spend backporting fixes to 1.8.x. An alternative would be to tag 1.9.0 early next year, aiming for a release around April. I think there is almost enough in 1.9-devel to justify a release. There is Sebastian's index work, Julian's continuing work on speedups, the removal of oldnumeric and numarray support, and various other deprecations and cleanups that add up to a significant number of changes. I've tended to think of 1.9 as a cleanup and consolidation release and think that the main thing missing at this point is fixing the datetime problems.

Thoughts?

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Numpy 1.9 release date

ralfgommers



On Fri, Nov 8, 2013 at 8:22 PM, Charles R Harris <[hidden email]> wrote:
Hi All,

The question has come up as to how much effort we should spend backporting fixes to 1.8.x. An alternative would be to tag 1.9.0 early next year, aiming for a release around April. I think there is almost enough in 1.9-devel to justify a release. There is Sebastian's index work, Julian's continuing work on speedups, the removal of oldnumeric and numarray support, and various other deprecations and cleanups that add up to a significant number of changes. I've tended to think of 1.9 as a cleanup and consolidation release

Makes sense.
 
and think that the main thing missing at this point is fixing the datetime problems.

Is anyone planning to work on this? If yes, you need a rough estimate of when this is ready to go. If no, it needs to be decided if this is critical for the release. From the previous discussion I tend to think so. If it's critical but no one does it, why plan a release.......

A suggestion for backporting strategy: do not backport things that have just been merged. Because (a) doing it PR by PR gives a lot of overhead, and (b) if the commit causes issues that have to be fixed or reverted, you have to fix things twice. Instead, just keep a list of backport candidates in a github issue, then do it all at once when it's clear that a bugfix release is needed.

Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Numpy 1.9 release date

dhirschfeld
Ralf Gommers <ralf.gommers <at> gmail.com> writes:

>

> On Fri, Nov 8, 2013 at 8:22 PM, Charles R Harris <charlesr.harris <at>
gmail.com> wrote:
>
>
> and think that the main thing missing at this point is fixing the datetime
problems.
>
>
> Is anyone planning to work on this? If yes, you need a rough estimate of
when this is ready to go. If no, it needs to be decided if this is critical
for the release. From the previous discussion I tend to think so. If it's
critical but no one does it, why plan a release.......
>
>
> Ralf
>

Just want to pipe up here as to the criticality of datetime bug.

Below is a minimal example from some data analysis code I found in our
company that was giving incorrect results (fortunately it was caught by
thorough testing):

In [110]: records = [
     ...:  ('2014-03-29 23:00:00', '2014-03-29 23:00:00'),
     ...:  ('2014-03-30 00:00:00', '2014-03-30 00:00:00'),
     ...:  ('2014-03-30 01:00:00', '2014-03-30 01:00:00'),
     ...:  ('2014-03-30 02:00:00', '2014-03-30 02:00:00'),
     ...:  ('2014-03-30 03:00:00', '2014-03-30 03:00:00'),
     ...:  ('2014-10-25 23:00:00', '2014-10-25 23:00:00'),
     ...:  ('2014-10-26 00:00:00', '2014-10-26 00:00:00'),
     ...:  ('2014-10-26 01:00:00', '2014-10-26 01:00:00'),
     ...:  ('2014-10-26 02:00:00', '2014-10-26 02:00:00'),
     ...:  ('2014-10-26 03:00:00', '2014-10-26 03:00:00')]
     ...:
     ...:
     ...: data = np.asarray(records, dtype=[('date obj', 'M8[h]'), ('str
repr', object)])
     ...: df = pd.DataFrame(data)

In [111]: df
Out[111]:
             date obj             str repr
0 2014-03-29 23:00:00  2014-03-29 23:00:00
1 2014-03-30 00:00:00  2014-03-30 00:00:00
2 2014-03-30 00:00:00  2014-03-30 01:00:00
3 2014-03-30 01:00:00  2014-03-30 02:00:00
4 2014-03-30 02:00:00  2014-03-30 03:00:00
5 2014-10-25 22:00:00  2014-10-25 23:00:00
6 2014-10-25 23:00:00  2014-10-26 00:00:00
7 2014-10-26 01:00:00  2014-10-26 01:00:00
8 2014-10-26 02:00:00  2014-10-26 02:00:00
9 2014-10-26 03:00:00  2014-10-26 03:00:00


Note the local timezone adjusted `date obj` including the duplicate value at
the clock-change in March and the missing value at the clock-change in
October. As you can imagine this could very easily lead to incorrect
analysis.

If running this exact same code in the (Eastern) US you'd see the following
results:
             date obj             str repr
0 2014-03-30 03:00:00  2014-03-29 23:00:00
1 2014-03-30 04:00:00  2014-03-30 00:00:00
2 2014-03-30 05:00:00  2014-03-30 01:00:00
3 2014-03-30 06:00:00  2014-03-30 02:00:00
4 2014-03-30 07:00:00  2014-03-30 03:00:00
5 2014-10-26 03:00:00  2014-10-25 23:00:00
6 2014-10-26 04:00:00  2014-10-26 00:00:00
7 2014-10-26 05:00:00  2014-10-26 01:00:00
8 2014-10-26 06:00:00  2014-10-26 02:00:00
9 2014-10-26 07:00:00  2014-10-26 03:00:00


Unfortunately I don't have the skills to meaningfully contribute in this
area but it is a very real problem for users of numpy, many of whom are not
active on the mailing list.

HTH,
Dave


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Numpy 1.9 release date

Stéfan van der Walt
In reply to this post by Charles R Harris

On 9 Nov 2013 03:22, "Charles R Harris" <[hidden email]> wrote:
>
> that the main thing missing at this point is fixing the datetime problems.

What needs to be done, and what is the plan forward? Is there perhaps an issue one can follow?

Thanks
Stéfan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Numpy 1.9 release date

Chris Barker - NOAA Federal
On Sun, Nov 10, 2013 at 7:27 PM, Stéfan van der Walt <[hidden email]> wrote:

> that the main thing missing at this point is fixing the datetime problems.

What needs to be done, and what is the plan forward?


I'm not sure that's quite been decided, but my take:

1) remove the existing time zone handling -- it simply isn't useful often, and does cause a pain in the &%^& often.
  - as far as I know, the only point of debate to the simple not-time-zone aware datetimes is whether that means "UTC" or "Local" or "Not Known" -- these are pretty subtle distinctions  and I think really only have an impact when you try to parse an iso string with a timezone attached.

2) _maybe_ do something smarter -- though this takes a lot more work and discussion as to what that should be.

I think they key points are captured here:


There is an issue:


but there is no detail there.

There are a number of other issues that come up in discussion:

* More precision with lap-seconds, etc.

*  Allowing an epoch that can change -- this is really crucial if you want picoseconds and friends to be remotely useful.

But these are orthogonal issues AFIIC, except that maybe one we open it up it makes sense to do it at once...

-Chris









 

Is there perhaps an issue one can follow?

Thanks
Stéfan


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion




--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

[hidden email]

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
http://mail.scipy.org/mailman/listinfo/numpy-discussion