« Kathy Sierra | Main | Creating Alternate Realities: Jane McGonigal »
AWS and Your Data Center: ETech 2007
Werner Vogels, Amazon’s CTO, is talking about their Web services—specifically the outsourced data center products (S3, EC2, and SQS) that I’ve written about before and that were the subject of an IT Conversations interview I did with Doug Kaye and Jeff Barr.
Werner begins by making a case that (a) scaling is critical to Web businesses and (b) scaling, economically, is really hard. I was just twittering with Phil Burns last night about servers. He just took delivery of four for TagJungle. He’s got a lot of work ahead of him setting them up. When TagJungle grows again, Phil has to do it all over again. Werner’s making a case that small businesses shouldn’t have that headache.
EC2 relies an Amazon Machine Images, virtualized machine images that you can create yourself (based on Xen) or simply download as a virtual appliance (see my earlier discussion of virtual appliances).
Someone (missed the name) from RightScale demoed using AWS for transcoding music on TuneCore. He shows how servers can be deployed and scaled based on demand.
Doug Kaye was invited up next to talk about the GigaVox system (GigaVox Audio Lite) that is the subject of my discussion with him and Jeff Barr. Doug is in the top 1% of AWS users in terms of the complexity and sophistication of his application. Doug says that the best part is that “no one’s wearing pagers.”
There’s a very small upfront investment and the bill grows as you grow. There are no sudden capital spikes.
Werner recommends From Push to Pull- Emerging Models for Mobilizing Resources, a paper by John Hagel and John Seely Brown. I’ll have to read it. Here’s the abstract:
Not really an abstract: In the past decade, we have seen early signs of a new model for mobilizing resources. Rather than “push”, this new approach focuses on “pull” — creating platforms that help people to mobilize appropriate resources when the need arises. In lean manufacturing, early elements of a pull model began to emerge from Toyota in the early 1950’s. As we will discuss below, however, lean manufacturing represents a hybrid between push and pull models — it still contains significant elements of push models.
Posted by windley on March 27, 2007 11:20 AM
Comment from BillyG at March 27, 2007 12:26 PM
"Werner Vogels, Amazon’s CTO, is talking about their Web services..." where is his article?
Comment from Gary at March 27, 2007 1:16 PM
Hi there. I am the CTO of TuneCore and we employed ELC/RightScale to migrate us to an AWS based system to be able to scale very quickly. It's not without a few bumps here and there but it's certainly cut our server costs dramatically and helped eliminate a lot of pain on the administration side. We currently have a mixed physical/virtual infrastructure but we might go all virtual in the next few months. I am really excited about the potential of virtual servers and storage.
Email me if you have any questions! :)
Comment from Phil801 at March 27, 2007 9:58 PM
Phil, just a point of clarity on our conversation and this post - we are using S3 and EC2 to store documents and scale web servers. The servers we bought are to host a MySQL cluster (you can't do it on EC2 yet) and 2 of them are to be windows servers. My backend processing engines are all in .Net while the front end and MySQL servers are all on Linux. We tried using Mono with our .Net engines but it isn't mature enough to handle what we're doing with it. You're correct that we'll soon have to add more servers, but only for things we can't do yet with EC2. I talked with Jeff quite a bit about which of our services we can run on EC2/S3, we tinkered with the idea of using windows emulators but decided to stick with a pure windows system for our engines. There's also the issue of our engines run on about 80 threads (this fluctuates) and we download about 7.5 terrabytes of data a month. We're using S3 to host millions of XML files that our system generates.
Comment from Indus Khaitan at March 30, 2007 1:01 PM
Great Post.
Amazon EC2 is doing a great job of hosting and allowing applications to scale within minutes if not hours. This is true utility computing.
However, EC2 and S3 need to have a stronger out of the box connection (read, S3 as a mounted filesystem, without using tools).
The other main issue is Mysql on EC2. S3 is not suited for storing data files because of several resons like:
a) S3 has a limitation on size of objects
b) S3 is not a block access, it is accessed via REST calls
c) Clustering, replication, etc. could add to complexity for single server instances and then the size of local disk is 160GB.
d) Using S3 wrapper tools may not work because of delays in read/writes.
All in all, it is a great start, I'm sure, Jeff Barr and his team would figure out the MySQL part.
Leave a comment
I encourage you to leave a comment below. Your email address will not be displayed on Technometria, but allows me to communicate with you directly. Your email address won't be displayed, but will be used to compute a MicroID for your comment.