Profile

Cover photo
The SageMathCloud
309 followers|80,634 views
AboutPostsPhotos

Stream

The SageMathCloud

Shared publicly  - 
 
status update from +William Stein 

Hi SageMathCloud Users,

A week ago, I moved all of SageMathCloud to Google Compute Engine, and rolled out a storage backend for SMC that relied on btrfs send/receive. The resulting week was one of the most painful weeks of my life. At UW, SMC ran on 19 computers, each having 1TB of SSD, 96GB (or more) of RAM, and 16 cores.    Squishing all of this into something affordable on GCE meant shrinking resources by an order of magnitude and this required rewriting or rethinking most of the backend to be more efficient. Also, there are over 150,000 projects, so just dealing with all that data is pretty daunting. Efficiency is important because Google isn't giving me a penny anymore to help with this (they gave us $60K in credits for last year), and SageMath, Inc. is paying for it using borrowed money in the hopes that there will be enough paying customers soon.

Unfortunately, it turns out that btrfs streaming isn't as robust as
I had hoped, especially under heavy load, and  would regularly crash
the OS, and have other performance problems.  Just in case, I had SMC store incremental rsync backups to Google nearline storage regularly, so if you find that there are file you made during the last week that are missing now, they are likely very easy for me to restore.

Over the last week, I wrote a new much more robust and efficient storage backend that doesn't use btrfs streaming, and switched everything over to it last night.  This new storage system works well so far, and I intend to stick with it.

There are some loose ends remaining.  For example, the snapshots you can browse in a project are made  periodically across all projects
(not just yours), so don't currently represent when you used your project, though that will change.  They are really just symbolic links into the /snapshots directory, which includes snapshots across all
projects.  Right now the display only shows less than a day of
snapshots, but in fact there are much more -- I just need to write some more code to properly present them.

Reducing other resources (e.g., number of web servers) also highlighted other issues, which I've tracked down and fixed.

Despite running on way less cores and much less RAM, right now SMC feels much, much faster than it did before.  The Google network is extremely fast, the Haswell Intel processors they provide have 45MB caches so are much much faster in arithmetic benchmarks I've tried, and the local disk (where your project files sit when you're using
them), is a fast PCIe SSD.   I've also been fixing a lot of little issues over the last 3 weeks that could lead to things feeling slow intermittently or connections being dropped (mostly recently fixing a
couple issues this afternoon).
4
1
Andrew Malcolmson's profile photoSage Mathematical Software System's profile photo
 
Thanks for slogging through all that.
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
I gave another talk about SageMathCloud to UW Applied/Computational mathematics students today -- https://youtu.be/q0jO7IfuZGQ
1 comment on original post
4
1
Flavio Coelho's profile photo
 
I really appreciated the talk. Thanks. I shared i it with my colleagues in hope they see the light. ;-)
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
New SageMathCloud Sage worksheet button bar.
View original post
1
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
SageMathCloud worksheets now support a new raw_input function, like the one in Python.  Of course this is way souped up compared to the raw_input that comes with Python (or with IPython Notebook).  I also added clear() and delete_last_output() functions to be control the output messages (and rewrote how they get rendered to be more powerful).  Restart your project server and refresh your browser to try it -- it even behaves sensibly with multiple copies of the same worksheet open. Here's a demo worksheet you can copy:  https://cloud.sagemath.com/projects/4a5f0542-5873-4eed-a85c-a18c706e8bcd/files/support/2015-03-23-194232-raw_input.sagews
View original post
3
1
Angel Blasco's profile photo
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
SageMath has been accepted for +Google Summer of Code 2015.

Please spread the word about this by sharing this posting with prospective students! We are looking forward to receiving your applications ;-)
SageMath is an open-source mathematical software system. It combines numerous open-source software libraries under one umbrella and contains many unique and novice algorithms built on top of those libraries. Besides a command-line and programming-library interface, its primary user interface is ...
View original post
1
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
Inspired by +Theron Hitchman, I introduced my linear algebra class to +The SageMathCloud tonight.  We'll see what they make of it. 
1 comment on original post
1
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
I put some work into making the backend node.js/nginx web servers behind https://SageMathCloud.com more efficient.  Here's top output on one (named 10.4.2.3 at [1]) running on Google Compute Engine in Europe with over 150 simultaneous connections.   This VM is also running haproxy, nginx, and stunnel, and it's barely loaded.   In November/December things were dying...

[1] See https://cloud.sagemath.com/stats for number of connections.
View original post
1
Add a comment...
In their circles
34 people
Have them in circles
309 people
John Hammersley's profile photo
Jaime Arango's profile photo
Olivier Guillon's profile photo
Mingzhang Yang's profile photo
Karen Gunderson's profile photo
Selahattin Murat Sirin's profile photo
Joel Bondurant's profile photo
Jasper Groenewegen's profile photo
Jyun-Ao Lin's profile photo

The SageMathCloud

Shared publicly  - 
 
SageMathCloud is now running 100% in Google Compute Engine.

If you have a project you care about, please try opening it and
looking at it.   I have to delete all data at UW in the near future,
and it's possible something didn't get properly transferred.

I completely rewrote much of the backend to make things much more
efficient regarding use of resources, so that hosting entirely on GCE
would be feasible cost-wise (it cost thousands a month, which I'm
paying for myself in the hopes people will become customers later).
With such a dramatic rewrite, there could be some major problems, and
also with a major transfer of data things could have gone wrong.

With this rewrite a lot of things under the hood are different.
6
2
Евгений Гринес's profile photoFlavio Coelho's profile photoSage Mathematical Software System's profile photoTheron Hitchman's profile photo
4 comments
 
It's working for me!
Add a comment...

The SageMathCloud

Shared publicly  - 
 
 
SageMathCloud -- link your account to Google/Facebook/Github, so you can login with those instead.   Also, people can just make an account by clicking one of those buttons.  Hopefully -- I just launched this...
9 comments on original post
1
Add a comment...

The SageMathCloud

Shared publicly  - 
4
2
Rodrigo Miranda's profile photoFlavio Coelho's profile photo
Add a comment...
People
In their circles
34 people
Have them in circles
309 people
John Hammersley's profile photo
Jaime Arango's profile photo
Olivier Guillon's profile photo
Mingzhang Yang's profile photo
Karen Gunderson's profile photo
Selahattin Murat Sirin's profile photo
Joel Bondurant's profile photo
Jasper Groenewegen's profile photo
Jyun-Ao Lin's profile photo
Contact Information
Contact info
Email
Story
Tagline
SageMathCloud is a distributed scalable online mathematical software environment.
Introduction
SageMathCloud is an online service, where you can develop and run mathematical and scientific computations in a web-browser. You have access to a full-fledged linux environment, that builds on top of the best open-source solutions, e.g. Python & many Python libraries like NumPy, SciPy, IPython, ..., the mathematical software system Sage, statistical software tool R, and much more. Additionally, all files allow collaborative editing and the Sage and IPython worksheets add convenience. Your data is periodically snapshotted and backed up.