Move scipy.org docs to Github?

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Move scipy.org docs to Github?

Matthew Brett
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

ralfgommers


On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].

Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.

Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Ilhan Polat
In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github

On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers <[hidden email]> wrote:


On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].

Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.

Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Ilhan Polat
By the way, this is not bad at all in the absence of the actual documentation http://devdocs.io/numpy~1.12/

On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat <[hidden email]> wrote:
In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github

On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers <[hidden email]> wrote:


On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].

Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.

Ralf


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion




_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Nathaniel Smith
In reply to this post by ralfgommers
On Mar 15, 2017 02:47, "Ralf Gommers" <[hidden email]> wrote:


On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].

Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.

No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course...

-n

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Bryan Van de Ven-2
In reply to this post by Ilhan Polat
Doesn't help all the CI builds that are failing because they utilize intersphinx to link to the official docs, unfortunately.

Bryan

> On Mar 15, 2017, at 05:59, Ilhan Polat <[hidden email]> wrote:
>
> By the way, this is not bad at all in the absence of the actual documentation http://devdocs.io/numpy~1.12/
>
> On Wed, Mar 15, 2017 at 11:55 AM, Ilhan Polat <[hidden email]> wrote:
> In the meantime maybe it's a good idea to keep one of the issues open so that people can see that this is an open issue? As we close them they disappear from the issues tab on Github
>
> On Wed, Mar 15, 2017 at 10:47 AM, Ralf Gommers <[hidden email]> wrote:
>
>
> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
> Hi,
>
> The scipy.org site is down at the moment, and has been for more than 36 hours:
>
> https://github.com/numpy/numpy/issues/8779#issue-213781439
>
> This has happened before:
>
> https://github.com/scipy/scipy.org/issues/187#issue-186426408
>
> I think it was down for about 24 hours that time.
>
> From the number of people opening issues or commenting on the
> scipy.org website this time, it seems to be causing quite a bit of
> disruption.
>
> It seems to me that we would have a much better chances of avoiding
> significant down-time, if we switched to hosting the docs on github
> pages.
>
> What do y'all think?
>
> Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].
>
> Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.
>
> Ralf
>
> [1] https://help.github.com/articles/what-is-github-pages/#usage-limits
>
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Bryan Van de Ven-2
In reply to this post by Nathaniel Smith
NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting.

Bryan


> On Mar 15, 2017, at 07:24, Nathaniel Smith <[hidden email]> wrote:
>
> On Mar 15, 2017 02:47, "Ralf Gommers" <[hidden email]> wrote:
>
>
> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
> Hi,
>
> The scipy.org site is down at the moment, and has been for more than 36 hours:
>
> https://github.com/numpy/numpy/issues/8779#issue-213781439
>
> This has happened before:
>
> https://github.com/scipy/scipy.org/issues/187#issue-186426408
>
> I think it was down for about 24 hours that time.
>
> From the number of people opening issues or commenting on the
> scipy.org website this time, it seems to be causing quite a bit of
> disruption.
>
> It seems to me that we would have a much better chances of avoiding
> significant down-time, if we switched to hosting the docs on github
> pages.
>
> What do y'all think?
>
> Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].
>
> Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.
>
> No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course...
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Benjamin Root
How is it that scipy.org's bandwidth usage is that much greater than matplotlib.org's? We are quite image-heavy, but we haven't hit any bandwidth limits that I am aware of.

Ben Root

On Wed, Mar 15, 2017 at 9:16 AM, Bryan Van de ven <[hidden email]> wrote:
NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting.

Bryan


> On Mar 15, 2017, at 07:24, Nathaniel Smith <[hidden email]> wrote:
>
> On Mar 15, 2017 02:47, "Ralf Gommers" <[hidden email]> wrote:
>
>
> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
> Hi,
>
> The scipy.org site is down at the moment, and has been for more than 36 hours:
>
> https://github.com/numpy/numpy/issues/8779#issue-213781439
>
> This has happened before:
>
> https://github.com/scipy/scipy.org/issues/187#issue-186426408
>
> I think it was down for about 24 hours that time.
>
> From the number of people opening issues or commenting on the
> scipy.org website this time, it seems to be causing quite a bit of
> disruption.
>
> It seems to me that we would have a much better chances of avoiding
> significant down-time, if we switched to hosting the docs on github
> pages.
>
> What do y'all think?
>
> Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].
>
> Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.
>
> No matter where we go, we can likely reduce the endpoint bandwidth requirements substantially by putting something like cloudflare's free tier in front. That doesn't help for the actual disk size though of course...
>
> -n
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Matthew Brett
In reply to this post by ralfgommers
Hi,

On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers <[hidden email]> wrote:

>
>
> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]>
> wrote:
>>
>> Hi,
>>
>> The scipy.org site is down at the moment, and has been for more than 36
>> hours:
>>
>> https://github.com/numpy/numpy/issues/8779#issue-213781439
>>
>> This has happened before:
>>
>> https://github.com/scipy/scipy.org/issues/187#issue-186426408
>>
>> I think it was down for about 24 hours that time.
>>
>> From the number of people opening issues or commenting on the
>> scipy.org website this time, it seems to be causing quite a bit of
>> disruption.
>>
>> It seems to me that we would have a much better chances of avoiding
>> significant down-time, if we switched to hosting the docs on github
>> pages.
>>
>> What do y'all think?
>
>
> Once the site is back up we should look at migrating to a better (hosted)
> infrastructure. I suspect that Github Pages won't work, we'll exceed or be
> close to exceeding both the 1 GB site size limit and the 100 GB/month
> bandwidth limit [1].
>
> Rough bandwidth estimate (using page size from
> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per
> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that
> pdf docs, which are ~20 MB in size: if only a small fraction of visitors
> download those, we'll be at >100 GB.

Maybe we could host the PDF docs somewhere else?   I wonder if Github
would consider allowing us to go a bit over if necessary?

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Daπid
What about readthedocs? I haven't seen any explicit limit in traffic.

On 15 March 2017 at 16:33, Matthew Brett <[hidden email]> wrote:

> Hi,
>
> On Wed, Mar 15, 2017 at 2:47 AM, Ralf Gommers <[hidden email]> wrote:
>>
>>
>> On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>>
>>> The scipy.org site is down at the moment, and has been for more than 36
>>> hours:
>>>
>>> https://github.com/numpy/numpy/issues/8779#issue-213781439
>>>
>>> This has happened before:
>>>
>>> https://github.com/scipy/scipy.org/issues/187#issue-186426408
>>>
>>> I think it was down for about 24 hours that time.
>>>
>>> From the number of people opening issues or commenting on the
>>> scipy.org website this time, it seems to be causing quite a bit of
>>> disruption.
>>>
>>> It seems to me that we would have a much better chances of avoiding
>>> significant down-time, if we switched to hosting the docs on github
>>> pages.
>>>
>>> What do y'all think?
>>
>>
>> Once the site is back up we should look at migrating to a better (hosted)
>> infrastructure. I suspect that Github Pages won't work, we'll exceed or be
>> close to exceeding both the 1 GB site size limit and the 100 GB/month
>> bandwidth limit [1].
>>
>> Rough bandwidth estimate (using page size from
>> http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per
>> month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that
>> pdf docs, which are ~20 MB in size: if only a small fraction of visitors
>> download those, we'll be at >100 GB.
>
> Maybe we could host the PDF docs somewhere else?   I wonder if Github
> would consider allowing us to go a bit over if necessary?
>
> Cheers,
>
> Matthew
> _______________________________________________
> NumPy-Discussion mailing list
> [hidden email]
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Marten van Kerkwijk
Astropy uses readthedocs quite happily (auto-updates on merges to master too).
-- Marten
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Pauli Virtanen-3
Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti:
> Astropy uses readthedocs quite happily (auto-updates on merges to master
> too).

AFAIK, scipy cannot be built on readthedocs.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Nathaniel Smith
On Mar 15, 2017 12:28 PM, "Pauli Virtanen" <[hidden email]> wrote:
Wed, 15 Mar 2017 12:11:09 -0400, Marten van Kerkwijk kirjoitti:
> Astropy uses readthedocs quite happily (auto-updates on merges to master
> too).

AFAIK, scipy cannot be built on readthedocs.

Another issue is that switching to rtd would (I think?) force us into their URL structure and break all incoming links, which would be really bad.

-n


_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Nathaniel Smith
In reply to this post by Bryan Van de Ven-2
On Wed, Mar 15, 2017 at 6:16 AM, Bryan Van de ven <[hidden email]> wrote:
> NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting.

Enthought already provides hosting and operations support (thanks!) –
the problem is that it doesn't make sense to have a full-time ops
person just for numpy, but if we're taking a tiny slice of someone's
time then occasional outages are going to happen.

The advantage of something like github pages is that it's big enough
that it *does* have dedicated ops support.

As long as we can fit under the 1 gig size limit then GH pages seems
like the best option so far... it's reliable, widely understood, and
all of the limits besides the 1 gig size are soft limits where they
say they'll work with us to figure things out.

-n

--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Todd
In reply to this post by ralfgommers


On Mar 15, 2017 05:47, "Ralf Gommers" <[hidden email]> wrote:


On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett <[hidden email]> wrote:
Hi,

The scipy.org site is down at the moment, and has been for more than 36 hours:

https://github.com/numpy/numpy/issues/8779#issue-213781439

This has happened before:

https://github.com/scipy/scipy.org/issues/187#issue-186426408

I think it was down for about 24 hours that time.

From the number of people opening issues or commenting on the
scipy.org website this time, it seems to be causing quite a bit of
disruption.

It seems to me that we would have a much better chances of avoiding
significant down-time, if we switched to hosting the docs on github
pages.

What do y'all think?

Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1].

Rough bandwidth estimate (using page size from http://scipy.github.io/devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB.

Ralf


Would it be possible to include the PDF and zipped HTML as sources in GitHub releases and (if desired) link to those on the documentation page?  That way only the individual HTML pages would need to be stored in GitHub pages, reducing both the size and bandwidth.



_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Pauli Virtanen-3
In reply to this post by Nathaniel Smith
Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti:
[clip]
> As long as we can fit under the 1 gig size limit then GH pages seems
> like the best option so far... it's reliable, widely understood, and all
> of the limits besides the 1 gig size are soft limits where they say
> they'll work with us to figure things out.

The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done
if a limited number of releases are kept, and the rest and the auxiliary
files are put as release downloads.

Otherwise there's probably no problem as you can stick a CDN in front if
it's too heavy otherwise.

Sounds sensible? Certainly it's the lowest-effort approach, and would
simplify management of S3/etc origin site access permissions.

        Pauli

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Matthew Brett
Hi,

On Wed, Mar 15, 2017 at 4:20 PM, Pauli Virtanen <[hidden email]> wrote:

> Wed, 15 Mar 2017 14:56:52 -0700, Nathaniel Smith kirjoitti:
> [clip]
>> As long as we can fit under the 1 gig size limit then GH pages seems
>> like the best option so far... it's reliable, widely understood, and all
>> of the limits besides the 1 gig size are soft limits where they say
>> they'll work with us to figure things out.
>
> The Scipy html docs weigh 60M apiece and numpy is 35M, so it can be done
> if a limited number of releases are kept, and the rest and the auxiliary
> files are put as release downloads.
>
> Otherwise there's probably no problem as you can stick a CDN in front if
> it's too heavy otherwise.
>
> Sounds sensible? Certainly it's the lowest-effort approach, and would
> simplify management of S3/etc origin site access permissions.

Sounds very sensible to me,

Cheers,

Matthew
_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Didrik Pinte-5
In reply to this post by Nathaniel Smith


On 15 March 2017 at 22:56, Nathaniel Smith <[hidden email]> wrote:
On Wed, Mar 15, 2017 at 6:16 AM, Bryan Van de ven <[hidden email]> wrote:
> NumPy is a NumFocus fiscally sponsored project, perhaps they can help with the costs of different/better hosting.

Enthought already provides hosting and operations support (thanks!) –
the problem is that it doesn't make sense to have a full-time ops
person just for numpy, but if we're taking a tiny slice of someone's
time then occasional outages are going to happen.

The key issue is this specific case is that the entire system was totally undocumented. If it had been, the outage would have lasted much less than an hour!

The advantage of something like github pages is that it's big enough
that it *does* have dedicated ops support.

Agreed. One issue is that we are working with a lot of legacy. Github will more than likely be a great solution to host static web pages but the evaluation for the shift needs to get into all the funky legacy redirects/rewrites we have in place, etc. This is probably not a real issue for docs.scipy.org but would be for other services.
 
As long as we can fit under the 1 gig size limit then GH pages seems
like the best option so far... it's reliable, widely understood, and
all of the limits besides the 1 gig size are soft limits where they
say they'll work with us to figure things out.

Another option would be to just host the content under S3 with Cloudfront. It will also be pretty simple as a setup, scale nicely and won't have much restrictions on sizing.

-- Didrik

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Pauli Virtanen-3
Thu, 16 Mar 2017 08:15:08 +0100, Didrik Pinte kirjoitti:
>> The advantage of something like github pages is that it's big enough
>> that it *does* have dedicated ops support.
>
> Agreed. One issue is that we are working with a lot of legacy. Github
> will more than likely be a great solution to host static web pages but
> the evaluation for the shift needs to get into all the funky legacy
> redirects/rewrites we have in place, etc. This is probably not a real
> issue for docs.scipy.org but would be for other services.

IIRC, there's not that many of them, so in principle it could be possible
to cobble them with <meta> redirects.

>> As long as we can fit under the 1 gig size limit then GH pages seems
>> like the best option so far... it's reliable, widely understood, and
>> all of the limits besides the 1 gig size are soft limits where they say
>> they'll work with us to figure things out.
>
> Another option would be to just host the content under S3 with
> Cloudfront.
> It will also be pretty simple as a setup, scale nicely and won't have
> much restrictions on sizing.

Some minor-ish disadvantages of this are that it brings a new set of
credentials to manage, it will be somewhat less transparent, and the
tooling will be less familiar to people (eg release managers) who have to
deal with it.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Move scipy.org docs to Github?

Robert McGibbon
I have always put my docs on Amazon S3 (examples: http://mdtraj.org/1.8.0/, .http://msmbuilder.org/3.7.0/) For static webpages, you can't beat the cost, and there's a lot of tooling in the wild for uploading pages to S3.

It might be an option to consider.

-Robert

On Thu, Mar 16, 2017 at 5:08 PM, Pauli Virtanen <[hidden email]> wrote:
Thu, 16 Mar 2017 08:15:08 +0100, Didrik Pinte kirjoitti:
>> The advantage of something like github pages is that it's big enough
>> that it *does* have dedicated ops support.
>
> Agreed. One issue is that we are working with a lot of legacy. Github
> will more than likely be a great solution to host static web pages but
> the evaluation for the shift needs to get into all the funky legacy
> redirects/rewrites we have in place, etc. This is probably not a real
> issue for docs.scipy.org but would be for other services.

IIRC, there's not that many of them, so in principle it could be possible
to cobble them with <meta> redirects.

>> As long as we can fit under the 1 gig size limit then GH pages seems
>> like the best option so far... it's reliable, widely understood, and
>> all of the limits besides the 1 gig size are soft limits where they say
>> they'll work with us to figure things out.
>
> Another option would be to just host the content under S3 with
> Cloudfront.
> It will also be pretty simple as a setup, scale nicely and won't have
> much restrictions on sizing.

Some minor-ish disadvantages of this are that it brings a new set of
credentials to manage, it will be somewhat less transparent, and the
tooling will be less familiar to people (eg release managers) who have to
deal with it.

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion



--
-Robert

_______________________________________________
NumPy-Discussion mailing list
[hidden email]
https://mail.scipy.org/mailman/listinfo/numpy-discussion
12
Loading...