Shared publicly  - 
 
TL;DR: New pricing postponed to Nov 1, instance hour discount extended to Dec 1, Python 2.7 expected by Dec 1, your bill is likely to go up but less than you fear, plus an analysis of why App Engine is still a great deal.

I’m the Engineering Director at Google responsible for App Engine. Nice to meet you!

We’re excited about coming out of preview and becoming a fully supported Google product. Besides new features, a 99.95% SLA, new Terms of Service, paid support, and monthly invoicing, we’re also changing the pricing model. We rolled out “side-by-side” billing last week to all App Engine developers, and sent an email with accompanying information, showing the predicted effects of the new pricing on all applications.

This has created a bit of consternation. I’d like to take the opportunity to provide some commentary on two topics: the timeline, and the price increase per se.

First topic, the timeline.

Many developers feel that they are being given too little time to make adjustments. We announced the new prices in May, and we thought the side-by-side billing would be just the next phase; instead, for many (arguably most) developers the side-by-side billing is just the start of their adjustment.

It’s clear we were wrong: expecting developers to figure out their future costs from information in the admin console was simply too obtuse. We made a classic error: we’re too familiar with our own product.

So I apologize: we should have realized this and put out a version of side-by-side billing much sooner. But this is easy enough to fix: we’ll just give you more time. So instead of turning on the billing in the second half of September, we are giving developers more time to fine tune their application by moving that date to November 1st.

Another aspect of the timeline that has caused concern is the uncertainty about the availability of Python 2.7, which will bring concurrent request support to our Python developers (it’s already in place for Java). We added a 50% instance price discount to compensate for the delay, and said we would remove that discount on November 20th. We’ve decided that we’ll extend that discount to December 1st, by which time we expect to have Python 2.7 available.

Second topic, the pricing increase per se.

The vision for Google App Engine is to provide a development environment for cloud applications to run on Google’s infrastructure. In particular, if you build on App Engine, it should be:

* free to get started and easy to use
* simple to make it scalable
* trivial to maintain it

App Engine is a classical case of platform computing trade-offs. What would you like? High scalability, ease of use, low maintenance, high reliability, security? Pick any two and somebody can make it super cheap. But if you want three or four, or all five, things become a little more dicey. With App Engine we are targeting doing all five at the same time, because we think that’s what cloud application developers ultimately want, and we think that’s what the future of cloud computing entails. And we’ve decided to package that offering into a free tier and two paid tiers:

* The generous free tier that App Engine has been offering has remained unique in the industry. We will continue to offer a free tier: a small web application with low traffic should not cost anything to run on App Engine. The new free quota levels are lower than before, so many pre-existing free applications will require tuning, however the principle will remain: App Engine is the only major platform to offer a free tier that’s not time-limited. If you have a small app that you can’t get under the free quotas, post in our forums and we’ll try to help. (But please try to tune it first.)
* The paid tier will charge for resource usage, with a minimum $9/month to enable the 99.95% SLA. And that SLA does not carve-out time for maintenance windows: App Engine can be upgraded without planned downtime for your app. The pricing in the paid tier is structured so that we believe that the vast majority of applications will find their total cost of ownership (TCO) to be lower than the competition. If you find that this is not the case for you, then feel free to share your calculations with us. The second paid tier, premier accounts, adds operational support and monthly invoicing.

Our goal is to make the best possible cloud application platform for developers. The feedback we’ve received over the years is that people want things like great reliability, more features, quicker bug fixes, fewer restrictions, Delivering these things requires App Engine to become a sustainable product for Google. To be clear, we’re not in the business of selling cycles. The vast majority of our costs are in the talented engineers that develop, maintain, operate, and support the overall App Engine service. And they’re not just any engineers, they’re some of the most talented and dedicated individuals I’ve had the honor to work with. They care passionately about the platform and the developer experience. And that’s where we want to invest.

But even if we look at just the cycles - then no, not all cycles are created equal. This is one of the reasons we want to change our resource concept from “CPU” to “Instance”. App Engine instance hours are fully managed, fully provisioned, run in the context of a set of fully-maintained services, and there are no hidden costs. Just the consumer cost of electricity for running a single server in your home will cost you more than running most apps on App Engine.

And our instances are fully redundant, and we take care of switching between redundant data centers for you. We have over 100% capacity provisioning: we can lose not just one but more than one data centers and still run the entire workload, without applications being impacted. And we have full provisioning for spikes: in the week following the Japanese earthquake, our traffic to Japan doubled. Japan is our second largest country in terms of App Engine traffic after the US, so this amounted to adding capacity for a whole 100M population country in a just a few days. App Engine is so well provisioned that we didn’t need to add more capacity or intervene in any way.

App Engine instances run on Google’s own infrastructure in our own data centers, with the same security and monitoring as services like Gmail and Docs: Google employs a large team of security experts. And our extremely talented reliability engineers are on pagers 24/7 across global time zones: when subsystems have problems, we’re on the case, so you don’t have to be. The high replication datastore (HRD) that we rolled out in January has had no outages since launch.

That said, the new App Engine prices are higher. In fact I expect many large applications after optimizing will end up paying 2-5x more than before. Many small applications will no longer fit into the free quota without optimization or performance tradeoffs. And many applications that only had to pay a little bit above free quota now have to pay more.

But I believe that for the vast majority of applications, a reasonable total cost analysis will find that App Engine is a great deal. And it’s only going to get better. We have a ton of cool improvements in the pipeline.

Thank you for your attention, and feel free to email me directly at psm@google.com. And if you come to Thirsty Bear in San Francisco tonight, I’ll buy you a beer.
93
105
Peter Magnusson's profile photoPatrick Chanezon's profile photoRahul Sood's profile photoGuido van Rossum's profile photo
20 comments
 
So you're extending the instance discount to end the same day that you think maybe you'll have the concurrent enabled python 2.7 available? Wow, big thanks. I have all of one day to figure out how to refactor my app to use concurrency before you end the single-threaded sucks discount? I might not be so irritated if ya know I could actually start working on the transition now. But to have the trusted tester program not open to every one makes it a bit insulting that you're not extending the instance discount to some time (like 30 days) AFTER the new python runtime is available.
 
Thanks +Peter Magnusson for this. The community was waiting for a detailed answer to what was going on. I understand you guys needed the pricing switch, and therefore you prefer letting people "optimize if they can't afford it" and lose the ones that simply have no time to optimize (startups). I still feel like the kind of optimization you guys are requiring in order to let people's apps scale is quite complex.

If you're a startup, and your daily audience is a couple hundred people, you probably already have about 4 instances running, even though technically a single one could just be enough. All of this is controlled by the scheduler, which is still unknown to developers, and the scheduler being the single most important factor that will most likely affect your bill, you probably want to know exactly why it spawned a new instance, rather then using the one that was already there. This doesn't really bring a lot of confidence in me (and probably other developers as well), if the way my bill is going to be shaped will depend on how the scheduler algorithm works.

So at the end of the day your startup, which could easily go by with a single instance, is spawning instances all over the place, and so you're forced to break the precious development cycle of your app and concentrate on performance and optimization - I call this premature optimization, and in the startup world it will most likely kill your app.

I'm sure the App Engine team has the best engineers in the world, but it seems like you guys are thinking too much about the engineering part rather than concentrating on what the costumers really want. The costumers don't want to go and optimize their code, a codebase that was already highly optimized in the beginning to work with App Engine's APIs, they want to use a high-level API so that optimization is taken care by that layer.

If this is not possible, oh well, I cannot afford dealing with an uncertain scheduler.
 
I have a simple blog application (http://glaforge.appspot.com), with not much trafic, but enough of regular hits from random guys, feed readers, and google search bots to keep my blog instance always on. Despite having toggled the "sliders" (max instances / idle time), my small blog will cost me quite a bit. I was really expecting a small blog website wouldn't be not free to run. I'm really disapointed.
 
+Emlyn O'Regan, not all developers have the patience you have when they have an angry boss down their neck pressuring them to meet a deadline. Plus, it's not like you'll be able to apply the experiences you've earned with 'optimizing App Engine' in other environments, it's a closed platform. Sure you'll still earn experience when doing any kind of optimization/coding, but I like to optimize my apps to make them better or faster, to optimize them just to make them cheaper is a waste of time. Hopefully a higher-level API or framework that can take care of all this optimization for you can come out as a community project, that would be a good step forward.
 
I think the new prices are fair, and after some optimizations my app seems to be performing acceptably money-wise.

So I think that what you guys did is ok, understandable even. But HOW you did it, well, sucks. I've had to plow through reams of tl;dr discussions in the issuetracker and in the groups to get some decent idea of what's going on. Facts have come out when average users have done extensive investigative and deductive footwork and then had their conclusions confirmed by Google reps - the efforts of Emlyn come to mind, and I am indebted to him and others like him. But why was this necessary? Why was this left to the community?

The old model was easily understandable. The new one is opaque, and relies on a barely-documented black-box system. You know that "articles" section in the docs? You know about all those wonderful deep-dives at IO? Why is there not one of those on the scheduler? Had you guys published all the information that's surfaced in the group in a transparent organized and exhaustive fashion, back in May, perhaps even released some case-studies with before-and-after code and performance profiles, you wouldn't have all this hand-wringing and speculation right now.

"We’re too familiar with our own product." is a bit of a rookie mistake - you guys are google, and I'm the rookie, can we keep that straight? =)
 
Heh I just assumed that talk was a rehash of Brett Slatkin's older talk and overlooked it. My mistake, thanks for pointing it out.
 
+Mike Fayer The scheduler is no more obtuse now than before, if anything we've increased visibility and control over it. Why is "CPU hour" less opaque than "instance hour"? "All" that we've really done is raise the prices on resource consumption inside the system. I understand people disliking price changes, but I don't understand why you think opacity has changed. But yes your point that we could have done this better is well taken. And no, I disagree that it's a rookie mistake, pros make it all the time. :-)

+Sasha Hart Not sure I understand your point. I'm saying precisely that what we are selling is not a commodity.

+Luca Matteis It's exactly with looming deadlines from your boss that GAE shines. If you think optimizing to make them cheaper is a waste of time, then don't optimize. I don't understand. You either have an ROI in the tuning, in which case do it; or you don't, in which case don't.
 
Sure the scheduler didn't get more opaque, we just used to not care about how opaque it was (or, not care that much, it was CPU we cared about.) And please don't be facetious, no one's saying that the concept of "instance-hour" is difficult, but the concept of "when/why is an instance spun up/killed" is anything but simple or transparent.

http://groups.google.com/group/google-appengine/browse_thread/thread/cee4570298469cf9
This is more helpful and lucid than anything in the docs. And coming to the understanding that's reached in that thread took some serious detective work, while cycles (even for some virtual made-up CPU) are completely and immediately understandable.

There may be a lot of reflexive moaning and groaning on the forums, but there's also plenty of schematic-on-the-table pencil-in-mouth muttering-and-coffee banging of heads as people try to figure out how the damn contraption actually works. If we were just complaining about a price hike and everything was crystal-clear, you'd only have the former, not the latter.
 
Yes, I signed up the day that was posted. I'm still waiting. It's really not cool to discontinue the discount before everyone has had some reasonable time to work with the new runtime, not just the closed group.
 
We'll try hard to get it out as soon as possible.
 
Hi - you're welcome :)

Surely saying one thing and having people hear something else is a defining experience for geeks, but in this case I'm just overjoyed to find that that most of my feelings of (too strong?) betrayal in the new pricing announcements was due to collective mis-speaking/mis-hearing - in a year or two I may look back and see this as an affirmative indicator of the fact that we get to speak to techies not PR/Sales people :)

Between this post, the detailed addressing of specific issues by +Jon McAlister, and the other post/announcements from +Greg D'Alesandre and +Ikai Lan, I'm feeling more than a little foolish for my post that ended "so long and thanks for all the fish", and more than happy that I kept watching the forums whilst exploring other options (I've now put "other options" back to the same position in my personal risk register as it occupied before any out-of-preview annoucements).

I know there are more than these 3 people involved (publicly and behind the scenes), but being the public interface is not an easy role at times like this, and I'm sure they can pass my thanks on to the rest of the team.
 
Thanks Peter, this explanation is quite well done and I think the move forward plans make a lot of sense :)
 
It's amusing to see everybody upset with the price increases, but to see so few people announcing that they're moving their projects to appscale on EC2 because app engine is still a very cheap option. +Sasha Hart
 
An excerpt from an email i received from one of the many Enterprise ISVs building cloud apps & evaluating GAE - "Our developers are convinced GAE is the platform to go forward after reading the blog" . Thanks +Peter Magnusson
 
+Steve Anderson by now everybody who requested Py2.7 TT access for starting working on concurrent requests should have gotten it by now; let me know if we've missed anybody.
 
That's good to hear Peter. I'm glad everyone will have at least 30 days to refactor their apps before the instance discount goes away. Hopefully we'll have a reasonable amount of time to develop and test with the SDK (once that's working).
 
+salim madjd, you're not very bright. Go get appscale and install it on AWS if you're that desperate to migrate away. Have fun managing that yourself.
Add a comment...