Recent Blog Posts:
Category Mobile - Web - Media
Date Monday, Feb 19, 2007 2:09:18 PM
Amazon's Web Services Offer Upscaling and Redundancy
Over the last month, I've turned my development focus away from new features, and have targeted performance and scaling issues as my number one priority.

I've been rewriting core parts of the application that generates ArtistServer, simplifying and optimizing it through object oriented design methods, and shaving 10-1,500ms from pages on the site. While that may seem minimal, when you multiply it by 1,000's of users making requests, those milliseconds add up!

Scaling, is being able to support a growing population of users, data and media, while sustaining an acceptable performance for it's users. Doing this usually requires some initial investment in hardware, so you stay ahead of the curve. Thanks to Amazon Web Services, you and I can now ride the growth curve in real time.

Amazon offers several Web Services which could greatly change your business by saving you money, allowing you to grow on less initial investment, and potentially open your business up to new opportunities. In my case, I'm beginning to see how I'll be able to continue growing my business, minimize the risks, improve quality, and save money while freeing my time up for creating and exploring new ideas.

Amazon Web Services Home

Are you aware of these services?

  • Elastic Compute Cloud (Amazon EC2) - is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale computing easier for developers.

    Just as Amazon Simple Storage Service (Amazon S3) enables storage in the cloud, Amazon EC2 enables "compute" in the cloud. Amazon EC2's simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon's proven computing environment. Amazon EC2 reduces the time required to obtain and boot new server instances to minutes, allowing you to quickly scale capacity, both up and down, as your computing requirements change. Amazon EC2 changes the economics of computing by allowing you to pay only for capacity that you actually use.

  • Mechanical Turk - provides a web services API for computers to integrate Artificial Artificial Intelligence directly into their processing by making requests of humans. Developers use the Amazon Mechanical Turk web services API to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications. To the application, the transaction looks very much like any remote procedure call - the application sends the request, and the service returns the results. In reality, a network of humans fuels this Artificial Artificial Intelligence by coming to the web site, searching for and completing tasks, and receiving payment for their work.

  • Simple Storage Service (S3) - S3 is storage for the Internet. It is designed to make web-scale computing easier for developers.

    Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

  • Simple Queue Service (Amazon SQS) - offers a reliable, highly scalable hosted queue for storing messages as they travel between computers. By using Amazon SQS, developers can simply move data between distributed application components performing different tasks, without losing messages or requiring each component to be always available.

    Amazon SQS works by exposing Amazon's web-scale messaging infrastructure as a web service. Any computer on the Internet can add or read messages without any installed software or special firewall configurations. Components of applications using Amazon SQS can run independently, and do not need to be on the same network, developed with the same technologies, or running at the same time.

I suggest spending some time at the Amazon Web Services site and read up on each of their services, then brainstorm on how you could use these services in your business or site.

I'm currently working on an integration with Amazon's S3 service, and once that's complete, I'll most likely look at integrating EC2 and SQS into my application. Below, is a diagram showing how a developer could add Amazon S3 (Simple Storage Solution) to their Web application and gain not only a storage solution that scales, but also data redundancy, allowing you to toggle between local or remote versions of files. The diagram is meant to give you a general idea as to how you could integrate S3, there's definitely room for variations and additional optimization. For example, instead of the using a local page for your queue, you could use Amazon's SQS, or your own queueing server with a 'ticket system' for managing tasks.

The diagram has two sections, the first shows files being uploaded to your site/service. The second section shows the serving of those files. The main idea here, is that files are stored on your server, then copied out to your S3 account. By integrating a variable for tracking the status of the copy out on S3, you can serve either the local or remote copy. Since S3 can scale more than your server, you would use the S3 copy as the primary version, and your local copy as the backup. If for some reason S3 goes offline, you can user an application level variable to force your application to use the local copies. The 'code' you see in the diagram is 'pseudo code' and is only for demonstration purposes.

diagram of S3 integration

 

If image is not displayed, you may access it here: http://photo.artistserver.com/1/8/D94D29F7-10DC-3A80-B070C1CF742F7D63_O.jpg

BTW - If this kind of thing interests you, I suggest that you pick up Cal Henderson's book "Building Scalable Web Sites" published by O'Reilly.



Tags          

ADD TO:
Add to BlinkBlink
Add to Del.icio.usDel.icio.us
Add to DiggDigg
Add to FurlFurl
Add to GoogleGoogle
Add to SimpySimpy
Add to SpurlSpurl
Add to Y! MyWebY! MyWeb

Permalink:

Category Mobile - Web - Media
Date Friday, Jan 19, 2007 12:08:51 PM
MySpace and Handling Millions of Users

Yesterday, I read a very interesting article which covers how MySpace scaled itself to handle millions of users and billions of pageviews. This article probably won't be very interesting unless you work with the Web either through building sites, or managing a network.

Read the article online: http://www.baselinemag.com/article2/0,1540,2082921,00.asp

Here's a link to the print version - which is easier to read: http://www.baselinemag.com/print_article2/0,1217,a=198614,00.asp

They certainly made some bad choices along the way, but I can't even imagine the level of stress their engineers must have been under while dealing with the daily growth problems. It must have been insane.

Some will point out that if they used LAMP, MySpace would have less problems, errors, and could scale better... I'm not so sure about that though. When you consider the numbers of requests coming in, the constant publishing of content on the site, and the ever growing userbase, it's nearly an impossible situation.

You'll see that they've finally settled on Microsoft products and technology to stay online - almost fully ditching ColdFusion. Does this mean ColdFusion can't scale? No, not at all. As you'll see in the article, they took a look at every function and worked to optimize the code on the last rewrite. If they did this originally with the ColdFusion design, I'm sure they would have been able to keep scaling. Plus, the state of their application at the time when they selected to move to .NET sounded like spaghetti code.

No matter what language you develop in, the article's infomative and intersting.


Tags    

ADD TO:
Add to BlinkBlink
Add to Del.icio.usDel.icio.us
Add to DiggDigg
Add to FurlFurl
Add to GoogleGoogle
Add to SimpySimpy
Add to SpurlSpurl
Add to Y! MyWebY! MyWeb

Permalink:

Gideon Marken
Web Technologist & Electronic Artist

My status - click to chat with me.


Add Me to your Linked In Network



About


Blog Categories
 »  ArtistServer
 »  Sonic Wallpaper News
 »  Mobile - Web - Media
 »  SocialNetwork.in

April 2014
SUN MON TUE WED THU FRI SAT
   1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30    
< Previous Month  -  Next Month >
Click on dates, or view all

Twitter Tweets


My Music - Widget from ArtistServer:


What I've Been Listening To:
Listening chart from Last.fm


Free Ringtones From ArtistServer:


My Blog Tag Cloud


Kiss My RSS Get Firefox San Diego Bloggers