Long term plans for dropping Python 2.7

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Long term plans for dropping Python 2.7

Charles R Harris
Hi All,

It may be early to discuss dropping support for Python 2.7, but there is a disturbance in the force that suggests that it might be worth looking forward to the year 2020 when Python itself will drop support for 2.7. There is also a website, http://www.python3statement.org, where several projects in the scientific python stack have pledged to be Python 2.7 free by that date.  Given that, a preliminary discussion of the subject might be interesting, if only to gather information of where the community currently stands.

Chuck

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Nathaniel Smith
On Fri, Apr 14, 2017 at 5:19 PM, Charles R Harris
<[hidden email]> wrote:

> Hi All,
>
> It may be early to discuss dropping support for Python 2.7, but there is a
> disturbance in the force that suggests that it might be worth looking
> forward to the year 2020 when Python itself will drop support for 2.7. There
> is also a website, http://www.python3statement.org, where several projects
> in the scientific python stack have pledged to be Python 2.7 free by that
> date.  Given that, a preliminary discussion of the subject might be
> interesting, if only to gather information of where the community currently
> stands.

One reasonable position would that numpy releases that happen while
2.7 is supported upstream will also support 2.7, and releases after
that won't.

From numpy's perspective, I feel like the most important reason to
continue supporting 2.7 is our ability to convince people to keep
upgrading. (Not the only reason, but the most important.) What I mean
is: if we dropped 2.7 support tomorrow then it wouldn't actually make
numpy unavailable on python 2.7; it would just mean that lots of users
stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
end of the world – numpy is mature software and 1.12 works pretty
well. The big problem IMO would be if this then meant that lots of
downstream projects felt that they had to continue supporting 1.12
going forward, which makes it very difficult for us to effectively
ship new features or even bug fixes – I mean, we can ship them, but
no-one will use them. And if a downstream project finds a bug in numpy
and can't upgrade numpy, then the tendency is to work around it
instead of reporting it upstream. I think this is the main thing we
want to avoid.

This kind of means that we're at the mercy of downstream projects,
though – if scipy/pandas/etc. decide they want to support 2.7 until
2022, it might be in our best interest to do the same. But there's a
collective action problem here: we want to keep supporting 2.7 so long
as they do, but at the same time they may feel they need to keep
supporting 2.7 as long as we do. And all of us would prefer to drop
2.7 support sooner rather than later, but we might all get stuck
because we're waiting for someone else to move first.

So my suggestion would be that numpy make some official announcement
that our plan is to drop support for python 2 immediately after
cpython upstream does. If worst comes to worst we can always decide to
extend it at the time... but if we make the announcement now, then
it's less likely that we'll need to :-).

Another interesting project to look at here is django, since they
occupy a similar place in the ecosystem (e.g. last I checked numpy and
django are the two most-imported python packages on github):
https://www.djangoproject.com/weblog/2015/jun/25/roadmap/
Their approach isn't directly applicable, because unlike us they have
a strict time-based release schedule, defined support period for each
release, and a distinction between regular and long-term support
releases, where regular releases act sort of like
pre-releases-on-steroids for the next LTS release. But basically what
they settled on is philosophically similar to what I'm suggesting:
they don't want an LTS to be supporting 2.7 beyond when cpython is
supporting it. Then on top of that they don't want to support 2.7 in
the regular releases leading up to that LTS either, so the net effect
is that their last release with 2.7 support came out last week, and it
will be supported until 2020 :-). And another useful precedent I think
is that they announced this two years ago, back in 2015; if we make an
announcement now, we'll be be giving a similar amount of warning.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

ralfgommers


On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 5:19 PM, Charles R Harris
<[hidden email]> wrote:
> Hi All,
>
> It may be early to discuss dropping support for Python 2.7, but there is a
> disturbance in the force that suggests that it might be worth looking
> forward to the year 2020 when Python itself will drop support for 2.7. There
> is also a website, http://www.python3statement.org, where several projects
> in the scientific python stack have pledged to be Python 2.7 free by that
> date.  Given that, a preliminary discussion of the subject might be
> interesting, if only to gather information of where the community currently
> stands.

One reasonable position would that numpy releases that happen while
2.7 is supported upstream will also support 2.7, and releases after
that won't.

From numpy's perspective, I feel like the most important reason to
continue supporting 2.7 is our ability to convince people to keep
upgrading. (Not the only reason, but the most important.) What I mean
is: if we dropped 2.7 support tomorrow then it wouldn't actually make
numpy unavailable on python 2.7; it would just mean that lots of users
stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
end of the world – numpy is mature software and 1.12 works pretty
well. The big problem IMO would be if this then meant that lots of
downstream projects felt that they had to continue supporting 1.12
going forward, which makes it very difficult for us to effectively
ship new features or even bug fixes – I mean, we can ship them, but
no-one will use them. And if a downstream project finds a bug in numpy
and can't upgrade numpy, then the tendency is to work around it
instead of reporting it upstream. I think this is the main thing we
want to avoid.

+1
 

This kind of means that we're at the mercy of downstream projects,
though – if scipy/pandas/etc. decide they want to support 2.7 until
2022, it might be in our best interest to do the same. But there's a
collective action problem here: we want to keep supporting 2.7 so long
as they do, but at the same time they may feel they need to keep
supporting 2.7 as long as we do. And all of us would prefer to drop
2.7 support sooner rather than later, but we might all get stuck
because we're waiting for someone else to move first.

I don't quite agree about being stuck. These kind of upgrades should and usually do go top of stack to bottom. Something like Jupyter which is mostly an end user tool goes first (they announced 2020 quite a while ago), domain specific packages go at a similar time, then scipy & co, and only after that numpy. Cython will be even later I'm sure - it still supports Python 2.6.
 

So my suggestion would be that numpy make some official announcement
that our plan is to drop support for python 2 immediately after
cpython upstream does.

Not quite sure CPython schedule is relevant - important bug fixes haven't been making it into 2.7 for a very long time now, so the only change is the rare security patch.
 
If worst comes to worst we can always decide to
extend it at the time... but if we make the announcement now, then
it's less likely that we'll need to :-).

I'd be in favor of putting out a schedule in coordination with scipy/pandas/etc, but it probably should look more like
- 2020: what's on http://www.python3statement.org/ now
- 2021: scipy / pandas / scikit-learn / etc.
- 2022: numpy
 
Ralf


Another interesting project to look at here is django, since they
occupy a similar place in the ecosystem (e.g. last I checked numpy and
django are the two most-imported python packages on github):
https://www.djangoproject.com/weblog/2015/jun/25/roadmap/
Their approach isn't directly applicable, because unlike us they have
a strict time-based release schedule, defined support period for each
release, and a distinction between regular and long-term support
releases, where regular releases act sort of like
pre-releases-on-steroids for the next LTS release. But basically what
they settled on is philosophically similar to what I'm suggesting:
they don't want an LTS to be supporting 2.7 beyond when cpython is
supporting it. Then on top of that they don't want to support 2.7 in
the regular releases leading up to that LTS either, so the net effect
is that their last release with 2.7 support came out last week, and it
will be supported until 2020 :-). And another useful precedent I think
is that they announced this two years ago, back in 2015; if we make an
announcement now, we'll be be giving a similar amount of warning.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Nathaniel Smith
On Fri, Apr 14, 2017 at 10:47 PM, Ralf Gommers <[hidden email]> wrote:
>
>
> On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith <[hidden email]> wrote:
[...]

>> From numpy's perspective, I feel like the most important reason to
>> continue supporting 2.7 is our ability to convince people to keep
>> upgrading. (Not the only reason, but the most important.) What I mean
>> is: if we dropped 2.7 support tomorrow then it wouldn't actually make
>> numpy unavailable on python 2.7; it would just mean that lots of users
>> stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
>> end of the world – numpy is mature software and 1.12 works pretty
>> well. The big problem IMO would be if this then meant that lots of
>> downstream projects felt that they had to continue supporting 1.12
>> going forward, which makes it very difficult for us to effectively
>> ship new features or even bug fixes – I mean, we can ship them, but
>> no-one will use them. And if a downstream project finds a bug in numpy
>> and can't upgrade numpy, then the tendency is to work around it
>> instead of reporting it upstream. I think this is the main thing we
>> want to avoid.
>
>
> +1
>
>>
>>
>> This kind of means that we're at the mercy of downstream projects,
>> though – if scipy/pandas/etc. decide they want to support 2.7 until
>> 2022, it might be in our best interest to do the same. But there's a
>> collective action problem here: we want to keep supporting 2.7 so long
>> as they do, but at the same time they may feel they need to keep
>> supporting 2.7 as long as we do. And all of us would prefer to drop
>> 2.7 support sooner rather than later, but we might all get stuck
>>
>> because we're waiting for someone else to move first.
>
>
> I don't quite agree about being stuck. These kind of upgrades should and
> usually do go top of stack to bottom. Something like Jupyter which is mostly
> an end user tool goes first (they announced 2020 quite a while ago), domain
> specific packages go at a similar time, then scipy & co, and only after that
> numpy. Cython will be even later I'm sure - it still supports Python 2.6.

To make sure we're on the same page about what "2020" means here: the
latest release of IPython is 5.0, which came out in July last year.
This is the last release that supports py2; they dropped support for
py2 in master months ago, and 6.0 (whose schedule has been slipping,
but I think should be out Any Time Now?) won't support py2. Their plan
is to keep backporting bug fixes to 5.x until the end of 2017; after
that the core team won't support py2 at all. And they've also
announced that if volunteers want to step up to maintain 5.x after
that, then they're willing to keep accepting pull requests until July
2019.

Refs:
  https://blog.jupyter.org/2016/07/08/ipython-5-0-released/
  https://github.com/jupyter/roadmap/blob/master/accepted/migration-to-python-3-only.md

I suspect that in practice that "end of 2017" date will the
end-of-support date for most intents and purposes. And for numpy with
its vaguely defined support periods, I think it makes most sense to
talk in terms of release dates; so if we want to compare
apples-to-apples, my suggestion is that numpy drops py2 support in
2020 and in that sense ipython dropped py2 support in july last year.

>>
>> So my suggestion would be that numpy make some official announcement
>> that our plan is to drop support for python 2 immediately after
>> cpython upstream does.
>
>
> Not quite sure CPython schedule is relevant - important bug fixes haven't
> been making it into 2.7 for a very long time now, so the only change is the
> rare security patch.

Huh? 2.7 gets tons of changes: https://github.com/python/cpython/commits/2.7
Officially CPython has 2 modes for releases: "regular support" and
"security fixes only". 2.7 is special – it get regular support, and
then on top of that it also has a special exception to allow certain
kinds of major changes, like the ssl module backports.

If you know of important bug fixes that they're missing then I think
they'd like to know :-).

Anyway, the reason the CPython schedule is relevant is that once they
drop support, it *will* stop getting security patches, so it will
become increasingly impossible to use safely.

>>
>> If worst comes to worst we can always decide to
>> extend it at the time... but if we make the announcement now, then
>> it's less likely that we'll need to :-).
>
>
> I'd be in favor of putting out a schedule in coordination with
> scipy/pandas/etc, but it probably should look more like
> - 2020: what's on http://www.python3statement.org/ now
> - 2021: scipy / pandas / scikit-learn / etc.

Um... pandas is already on python3statement.org right now :-)

> - 2022: numpy

Honestly I don't see why we should plan to support python 2 a day
longer than our major downstream dependencies. That was the point of
my first paragraph: for us the main benefit to supporting 2 is to
avoid forcing our downstream dependencies to pin an old numpy. What's
that extra year get us if they've already moved on?

The other odd thing about this schedule is that you're suggesting that
the organizing principle should be that the stack switches from
top-of-stack to bottom... but then you left out the bottom of the
stack! :-)

- 2020: python

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Julian Taylor-3
In reply to this post by Charles R Harris
On 15.04.2017 02:19, Charles R Harris wrote:

> Hi All,
>
> It may be early to discuss dropping support for Python 2.7, but there is
> a disturbance in the force that suggests that it might be worth looking
> forward to the year 2020 when Python itself will drop support for 2.7.
> There is also a website, http://www.python3statement.org
> <http://www.python3statement.org/>, where several projects in the
> scientific python stack have pledged to be Python 2.7 free by that
> date.  Given that, a preliminary discussion of the subject might be
> interesting, if only to gather information of where the community
> currently stands.
>
> Chuck
>
>

I am very against planning to drop it.
Numpy is the lowest part of the scipy stack so it is not our decision to
do so and we don't gain that much by doing so.
Lets discuss this in 3 years or when the distributions kick out
python2.7 (which won't happen before ~2022). There is no point doing so now.
Also PyPy does not plan on dropping 2.7 by that time.

Also before we even consider this we need to fix our python3 support.
This means getting the IO functions
(https://github.com/numpy/numpy/pull/4208) in order and adding a string
type that people are less reluctant to use than the 4 byte unicode we
currently offer.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

CJ Carey
What do we think about the trade-offs of having a shared 2.7/3.x codebase going forward?

As Python3 adds more nontrivial features, keeping compatibility with 2.7 becomes more burdensome.

Will there be a separate py2-numpy branch/repo at some point before ending support?


On Apr 15, 2017 4:48 AM, "Julian Taylor" <[hidden email]> wrote:
On 15.04.2017 02:19, Charles R Harris wrote:
> Hi All,
>
> It may be early to discuss dropping support for Python 2.7, but there is
> a disturbance in the force that suggests that it might be worth looking
> forward to the year 2020 when Python itself will drop support for 2.7.
> There is also a website, http://www.python3statement.org
> <http://www.python3statement.org/>, where several projects in the
> scientific python stack have pledged to be Python 2.7 free by that
> date.  Given that, a preliminary discussion of the subject might be
> interesting, if only to gather information of where the community
> currently stands.
>
> Chuck
>
>

I am very against planning to drop it.
Numpy is the lowest part of the scipy stack so it is not our decision to
do so and we don't gain that much by doing so.
Lets discuss this in 3 years or when the distributions kick out
python2.7 (which won't happen before ~2022). There is no point doing so now.
Also PyPy does not plan on dropping 2.7 by that time.

Also before we even consider this we need to fix our python3 support.
This means getting the IO functions
(https://github.com/numpy/numpy/pull/4208) in order and adding a string
type that people are less reluctant to use than the 4 byte unicode we
currently offer.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Marten van Kerkwijk
Hi All,

I think Nathaniel had a good summary. My own 2¢ are mostly about the
burden of supporting python2. I have only recently attempted to make
changes in the C codebase of numpy and one of the reasons I found this
more than a little daunting is the complex web of include files. In
this respect, the python3/2 split is certainly not the biggest
hindrance, but it was also not particularly helpful for understanding
to have "translations" of python2 macros to python3 equivalents in
npy_3kcompat.h: for newcomers, it would seem helpful if they could
read the Python3 C-API and be able to understand what is going on.

Of course, the above also proves Julian's point: for strings in
particular, numpy still has a bit to go to be fully python3-ized.

Finally, as for pypy: they just made a huge effort to become
compatible with python3; is their plan really to stick with python2
much beyond 2020?

All the best,

Marten
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Julian Taylor-3
On 15.04.2017 16:17, Marten van Kerkwijk wrote:

> Hi All,
>
> I think Nathaniel had a good summary. My own 2¢ are mostly about the
> burden of supporting python2. I have only recently attempted to make
> changes in the C codebase of numpy and one of the reasons I found this
> more than a little daunting is the complex web of include files. In
> this respect, the python3/2 split is certainly not the biggest
> hindrance, but it was also not particularly helpful for understanding
> to have "translations" of python2 macros to python3 equivalents in
> npy_3kcompat.h: for newcomers, it would seem helpful if they could
> read the Python3 C-API and be able to understand what is going on.
>
> Of course, the above also proves Julian's point: for strings in
> particular, numpy still has a bit to go to be fully python3-ized.
>
> Finally, as for pypy: they just made a huge effort to become
> compatible with python3; is their plan really to stick with python2
> much beyond 2020?
>

http://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2

According to that Python2 support will be available as long as PyPy
itself exists.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Julian Taylor-3
On 15.04.2017 16:30, Julian Taylor wrote:

> On 15.04.2017 16:17, Marten van Kerkwijk wrote:
>> Hi All,
>>
>> I think Nathaniel had a good summary. My own 2¢ are mostly about the
>> burden of supporting python2. I have only recently attempted to make
>> changes in the C codebase of numpy and one of the reasons I found this
>> more than a little daunting is the complex web of include files. In
>> this respect, the python3/2 split is certainly not the biggest
>> hindrance, but it was also not particularly helpful for understanding
>> to have "translations" of python2 macros to python3 equivalents in
>> npy_3kcompat.h: for newcomers, it would seem helpful if they could
>> read the Python3 C-API and be able to understand what is going on.
>>
>> Of course, the above also proves Julian's point: for strings in
>> particular, numpy still has a bit to go to be fully python3-ized.
>>
>> Finally, as for pypy: they just made a huge effort to become
>> compatible with python3; is their plan really to stick with python2
>> much beyond 2020?
>>
>
> http://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2
>
> According to that Python2 support will be available as long as PyPy
> itself exists.
>

Of course they don't support the stdlib itself, so this doesn't actually
mean much depending on how the much community will care about fixing
security issues in the python2 stdlib.
But at least there might be a place where patches can get accepted and
released.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Charles R Harris
In reply to this post by ralfgommers


On Fri, Apr 14, 2017 at 11:47 PM, Ralf Gommers <[hidden email]> wrote:


On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 5:19 PM, Charles R Harris
<[hidden email]> wrote:
> Hi All,
>
> It may be early to discuss dropping support for Python 2.7, but there is a
> disturbance in the force that suggests that it might be worth looking
> forward to the year 2020 when Python itself will drop support for 2.7. There
> is also a website, http://www.python3statement.org, where several projects
> in the scientific python stack have pledged to be Python 2.7 free by that
> date.  Given that, a preliminary discussion of the subject might be
> interesting, if only to gather information of where the community currently
> stands.

One reasonable position would that numpy releases that happen while
2.7 is supported upstream will also support 2.7, and releases after
that won't.

From numpy's perspective, I feel like the most important reason to
continue supporting 2.7 is our ability to convince people to keep
upgrading. (Not the only reason, but the most important.) What I mean
is: if we dropped 2.7 support tomorrow then it wouldn't actually make
numpy unavailable on python 2.7; it would just mean that lots of users
stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
end of the world – numpy is mature software and 1.12 works pretty
well. The big problem IMO would be if this then meant that lots of
downstream projects felt that they had to continue supporting 1.12
going forward, which makes it very difficult for us to effectively
ship new features or even bug fixes – I mean, we can ship them, but
no-one will use them. And if a downstream project finds a bug in numpy
and can't upgrade numpy, then the tendency is to work around it
instead of reporting it upstream. I think this is the main thing we
want to avoid.

+1
 

This kind of means that we're at the mercy of downstream projects,
though – if scipy/pandas/etc. decide they want to support 2.7 until
2022, it might be in our best interest to do the same. But there's a
collective action problem here: we want to keep supporting 2.7 so long
as they do, but at the same time they may feel they need to keep
supporting 2.7 as long as we do. And all of us would prefer to drop
2.7 support sooner rather than later, but we might all get stuck
because we're waiting for someone else to move first.

I don't quite agree about being stuck. These kind of upgrades should and usually do go top of stack to bottom. Something like Jupyter which is mostly an end user tool goes first (they announced 2020 quite a while ago), domain specific packages go at a similar time, then scipy & co, and only after that numpy. Cython will be even later I'm sure - it still supports Python 2.6.
 

So my suggestion would be that numpy make some official announcement
that our plan is to drop support for python 2 immediately after
cpython upstream does.

Not quite sure CPython schedule is relevant - important bug fixes haven't been making it into 2.7 for a very long time now, so the only change is the rare security patch.
 
If worst comes to worst we can always decide to
extend it at the time... but if we make the announcement now, then
it's less likely that we'll need to :-).

I'd be in favor of putting out a schedule in coordination with scipy/pandas/etc, but it probably should look more like
- 2020: what's on http://www.python3statement.org/ now
- 2021: scipy / pandas / scikit-learn / etc.
- 2022: numpy
 

I think things will move faster than one might think. In any case, we are probably about 5 releases away from 2020. As Nathaniel points out, numpy is mature and 1.12 is pretty good already, so hopefully 1.17 would be even better. I think dropping Python 2.7 support at that point would not cause much in the way of problems as 1.17 should be good for a number of years after that and would be easily installed from PyPI.

A bigger driver long term might be uptake by distros, although the impact of that might be harder to estimate. I suspect it will affect developers more than end users, who will more likely be using Anaconda, Canopy, or similar to manage their development environment.

Another thing to consider is that future developers will likely have less and less experience with Python 2.7 as teaching and classroom use moves to 3.

Whatever we decide, I think Nathaniel's point about making an early announcement is a good one, as is Julian's comment about bringing Numpy into full support of Python 3. We need to put together a plan with at least a tentative schedule that will help get downstream projects thinking about their own plans and engender more feedback.

It might be useful to have a BOF(s) at SciPy 2017 where the issue can be discussed with a broader range of people.

<snip>

Chuck






_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

ralfgommers
In reply to this post by Nathaniel Smith


On Sat, Apr 15, 2017 at 7:02 PM, Nathaniel Smith <[hidden email]> wrote:
On Fri, Apr 14, 2017 at 10:47 PM, Ralf Gommers <[hidden email]> wrote:
>
>
> On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith <[hidden email]> wrote:
[...]
>> From numpy's perspective, I feel like the most important reason to
>> continue supporting 2.7 is our ability to convince people to keep
>> upgrading. (Not the only reason, but the most important.) What I mean
>> is: if we dropped 2.7 support tomorrow then it wouldn't actually make
>> numpy unavailable on python 2.7; it would just mean that lots of users
>> stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
>> end of the world – numpy is mature software and 1.12 works pretty
>> well. The big problem IMO would be if this then meant that lots of
>> downstream projects felt that they had to continue supporting 1.12
>> going forward, which makes it very difficult for us to effectively
>> ship new features or even bug fixes – I mean, we can ship them, but
>> no-one will use them. And if a downstream project finds a bug in numpy
>> and can't upgrade numpy, then the tendency is to work around it
>> instead of reporting it upstream. I think this is the main thing we
>> want to avoid.
>
>
> +1
>
>>
>>
>> This kind of means that we're at the mercy of downstream projects,
>> though – if scipy/pandas/etc. decide they want to support 2.7 until
>> 2022, it might be in our best interest to do the same. But there's a
>> collective action problem here: we want to keep supporting 2.7 so long
>> as they do, but at the same time they may feel they need to keep
>> supporting 2.7 as long as we do. And all of us would prefer to drop
>> 2.7 support sooner rather than later, but we might all get stuck
>>
>> because we're waiting for someone else to move first.
>
>
> I don't quite agree about being stuck. These kind of upgrades should and
> usually do go top of stack to bottom. Something like Jupyter which is mostly
> an end user tool goes first (they announced 2020 quite a while ago), domain
> specific packages go at a similar time, then scipy & co, and only after that
> numpy. Cython will be even later I'm sure - it still supports Python 2.6.

To make sure we're on the same page about what "2020" means here: the
latest release of IPython is 5.0, which came out in July last year.
This is the last release that supports py2; they dropped support for
py2 in master months ago, and 6.0 (whose schedule has been slipping,
but I think should be out Any Time Now?) won't support py2. Their plan
is to keep backporting bug fixes to 5.x until the end of 2017; after
that the core team won't support py2 at all. And they've also
announced that if volunteers want to step up to maintain 5.x after
that, then they're willing to keep accepting pull requests until July
2019.

Refs:
  https://blog.jupyter.org/2016/07/08/ipython-5-0-released/
  https://github.com/jupyter/roadmap/blob/master/accepted/migration-to-python-3-only.md

I suspect that in practice that "end of 2017" date will the
end-of-support date for most intents and purposes. And for numpy with
its vaguely defined support periods, I think it makes most sense to
talk in terms of release dates;

agreed, release dates makes sense. we don't want to be doing some kind of LTS scheme.
 
so if we want to compare
apples-to-apples, my suggestion is that numpy drops py2 support in
2020 and in that sense ipython dropped py2 support in july last year.

>>
>> So my suggestion would be that numpy make some official announcement
>> that our plan is to drop support for python 2 immediately after
>> cpython upstream does.
>
>
> Not quite sure CPython schedule is relevant - important bug fixes haven't
> been making it into 2.7 for a very long time now, so the only change is the
> rare security patch.

Huh? 2.7 gets tons of changes: https://github.com/python/cpython/commits/2.7

You're right. My experience is ending up on bugs.python.org when debugging and the answer to "can this be backported to 2.7" usually being no - but it looks like my experience is skewed by distutils, which is not exactly well maintained.
 
Officially CPython has 2 modes for releases: "regular support" and
"security fixes only". 2.7 is special – it get regular support, and
then on top of that it also has a special exception to allow certain
kinds of major changes, like the ssl module backports. 

If you know of important bug fixes that they're missing then I think
they'd like to know :-). 

Anyway, the reason the CPython schedule is relevant is that once they
drop support, it *will* stop getting security patches, so it will
become increasingly impossible to use safely.

For web stuff yes, but not all that relevant for scientific work.
 

>>
>> If worst comes to worst we can always decide to
>> extend it at the time... but if we make the announcement now, then
>> it's less likely that we'll need to :-).
>
>
> I'd be in favor of putting out a schedule in coordination with
> scipy/pandas/etc, but it probably should look more like
> - 2020: what's on http://www.python3statement.org/ now
> - 2021: scipy / pandas / scikit-learn / etc.

Um... pandas is already on python3statement.org right now :-)

> - 2022: numpy

Honestly I don't see why we should plan to support python 2 a day
longer than our major downstream dependencies. That was the point of
my first paragraph: for us the main benefit to supporting 2 is to
avoid forcing our downstream dependencies to pin an old numpy. What's
that extra year get us if they've already moved on?

The other odd thing about this schedule is that you're suggesting that
the organizing principle should be that the stack switches from
top-of-stack to bottom... but then you left out the bottom of the
stack! :-)

I don't think of Python as part of the stack, because it's not upgradeable for most users (except for with conda). It's more like having a base platform (OS + compilers + Python version) on which you install a scientific stack which has numpy as its lowest level component.

Ralf



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|

Re: Long term plans for dropping Python 2.7

Antoine Pitrou-2
In reply to this post by Nathaniel Smith
On Fri, 14 Apr 2017 22:19:42 -0700
Nathaniel Smith <[hidden email]> wrote:

>
> From numpy's perspective, I feel like the most important reason to
> continue supporting 2.7 is our ability to convince people to keep
> upgrading. (Not the only reason, but the most important.) What I mean
> is: if we dropped 2.7 support tomorrow then it wouldn't actually make
> numpy unavailable on python 2.7; it would just mean that lots of users
> stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the
> end of the world – numpy is mature software and 1.12 works pretty
> well. The big problem IMO would be if this then meant that lots of
> downstream projects felt that they had to continue supporting 1.12
> going forward, which makes it very difficult for us to effectively
> ship new features or even bug fixes – I mean, we can ship them, but
> no-one will use them.

Everyone using Python 3, which is a large and growing number of
people, will be able to use the new features.

I think the model you've outlined above -- a kind of "LTS" Numpy
version that supports 2.7 (with some amount of maintenance going on,
at least to fix important bugs), and later feature releases being
3.x-only, is the right way forward.  It will lighten maintenance of
later versions, allow the Numpy codebase to use modern Python idioms
and stdlib features, and will leave 2.x maintenance to people who
really care about it.


You may already have heard of it, but Django 1.11, which was just
released, is the last feature release to support Python 2.  Further
feature releases of Django will only support Python 3.
https://docs.djangoproject.com/en/1.11/releases/1.11/

Regards

Antoine.
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.python.org/mailman/listinfo/numpy-discussion