We experienced some brief availability issues this morning. One of our search nodes went down and we didn't handle that failure gracefully at all.

I've put some measures in place that were tested during this outage and feel confident in saying we should be able to handle this type of a failure much more gracefully in the future.

All systems are back and operating normally.

8:12 AM MST: Amazon reported this issue fixed at 5:52 AM PDT. Our own internal reports validate this, you can see this here: https://www.evernote.com/l/AJSDysKBRptO1ZnW2swDkAwTPCdQSMWrEL8B/image.png

4:24 AM MST: Amazon S3 is currently experiencing an outage that is affecting our image service. At this time, this is affecting approximately 30% of the origin requests from our CDN.

I'll update this post when I have more information to share. You can check the AWS status page for updates as well under "Amazon Simple Storage Service".


I only get empty responses to collection search calls like:

the result is {"page":1,"results":[],"total_pages":1,"total_results":1}

same goes for all other calls

CloudFlare (our CDN provider) experienced some DNS issues early this morning starting on March 4, 23:58 UTC and ending around March 5, 00:50 UTC. This affected some of our requests.

We haven't seen any more reports of issues since it was reported fixed.

2:08 PM MST: We are currently experiencing an issue with our image service. You will see sporadic 504 gateway time-out errors. We are working on this diligently and will post an update when we have one.

3:46 PM MST: Most (although not necessarily all) of the gateway time out errors should be resolved. We are still monitoring this service and working to ensure it returns to 100% back to normal.

4:25 PM MST: Our reporting tools show that the image service has returned back to normal operation.

3:39 PM MST: We're currently experiencing an issue with our Redis instances and it is affecting the general availability of the API. I will update this post with information as soon as I have more information to share.

3:50 PM MST: We've decided to revert the infrastructure upgrade we were attempting to make. Things should get back to normal shortly.

4:20 PM MST: We're back. The cache server upgrade was cancelled and reverted for the moment. We need to take a look at what went wrong so when we move ahead with this, we're confident it can be completed without any downtime. Our apologies.

1:30 PM MST (Sept. 26): We completed the change in our Redis cluster without interruption, the way it was supposed to go yesterday. All systems are green.

Last night we were affected by a AWS DNS resolution error in our availability zone in the US-EAST-1 region.

Amazon posted these status updates:

9:30 PM PDT We are investigating connectivity issues affecting some instances in a single Availability Zone in the US-EAST-1 Region.
9:34 PM PDT We can confirm that we are experiencing DNS resolution issues within a single Availability Zone in the US-EAST-1 Region. We are working to resolve the issue.
10:12 PM PDT Between 9:10 PM and 10:00 PM PDT we experienced DNS resolution issues within a single Availability Zone in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.

API stats were the primary victim of this outage and you might not see very reliable statistics from the past 12 hours.

AWS restored normal functionality and we took the necessary actions to make sure your usage is being tracked again.

There are no other known issues at this time.

Search movie with search_type=ngram returns nothing:
We have been working well with movie search api (/3/search/movie) for last month. While calling the API, we specify the parameter search_type=ngram. Suddenly now, it returns nothing no matter what term we query.

We are scheduled to launch this feature to the public in less than 12 hours. Any insights would be greatly appreciated!!!

Has this address for images been deprecated.

Ex. http://cf2.imgobject.com/t/p/original/7UEJEpstHI7ZnlzfAt7YbmCQuwS.jpg
Wait while more posts are being loaded