Mobile - Web - Media
Thursday, Jan 25, 2007 2:07:43 PM
Optimizing and Scaling Web Apps After reading about MySpace dealt with their scaling up to support millions of users, I decided it was a good idea to review my application's architechture, and identify what areas could scale independently, and what areas need optimization.
I first outlined the application that generates ArtistServer into seven parts - there's opportunity for optimization everywhere, so by looking at the applicaiton in different ways, I feel you can better identify those optimization points.
I then outlined the application into 25 Application Zones:
- Application Zones - the site divided into zones of functionality
- Sub-Applications - areas of the application which can run on separate servers and scale independently from the main site
- Flat File Publshing/Caching - writing of content and charts to files on the server to cut down on database access
- Site Data Methods - listing the interaction with the database by object and method
- Views - listing of all reusable views in the application, like 'song' and 'member'
- Widgets (future) - listing of planned widgets
- API (future) - outline of API plans
For each zone, I wrote what I felt was wrong, what could be done better, and what could run as a sub-application on it's own server. When I use the term sub-application, what I mean, is to take the zone or area that has specific functionality, and modify it so that it can run either on the same server as the main site, or run on a separate server where it can have it's own resources and potentially scale on it's own.
- Artist/Member sites
- Music/Ringtone/Area/Genre pages
- Serving Mp3s/Uploading mp3s
- My Account Admin
- Artist Member Files
- Photos Area
- Serving/Resizing Photos
- Site Skinning
- About/Info pages
- Error system/pages
- Site Admin
For example, the following are already functioning as sub-applications:
After my critical analysis, I found there were three zones I should consider modifying, turning them into sub-applications. They were: stats/tracking, artist/member's files, and RSS. As an example, I'll explain what is being done to the stats and tracking in the application.
- Database Server - database is hosted on it's own server
- Forums - hosted on a separate server using forums.artistserver.com
- Photos - photo uploads, resizing, and serving is done on a separate server using photos.artistserver.com
- MP3 Streams/Downloads- all mp3s are served from media.artistserver.com, which runs on the same server as the main site, but is ready to support a move to it's own server, or even a streaming provider
Stats and Tracking
The application tracks data on artist/member sites, but only those who've upgraded, and for all artists, we track all streams and downloads to their music. Actually, we filter out repeat streams in a 24hr period by the same IP to keep people from gaming the charts.
All the calls
to the stats tables were something that could happen asynchronously. There wasn't a need for the processes that were being tracked to have to wait for the tracking process to complete before completing themselves. For example, you have an artist page, and starting at the top, you would have a query to the database to return that person's record, and another query that tracks the access to the page, another to return their songs, etc. In this case, each query is happening in sequence, which means one has to complete before the other can execute.
Since the capturing of the stats data and writing to the database isn't going to be displayed on the page nor get used in any other portion of the code, it's a prime candidate for becoming an asynchronous process.
After I wrote a new stats object, I set it up on a separate server from the main application server (so my asynchronous stats calls don't compete for resources on the main server), I replaced my stats script on the artist and member pages with a call to the AsyncHTTP object by Compound Theory
(a ColdFusion CFC that uses Java), ran some tests, then took it live. Now when the page executes, and the AsyncHTTP object calls my stats server, the page continues to process without waiting, shaving off several milliseconds from every request to an artist or member page.
The same thing will be done with the mp3 tracking on the site, which as you can imagine, should make the process of requesting an mp3 for streaming and/or download faster for everyone.
After that, I'll break the stats tables out into their own database, so the main database can be
free to handle the application, and the stats database can grow independently. Initially, I'll still run the database on the same database server, but the point is, that at any time, that's a viable scaling option - the stats could easily be setup on it's own database server or server cluster.
Sounds like fun eh? :)
Objectize, Optimize, Modularize, Asynchronize
Create and extend objects.
Optimize at all levels.
Asynchoronize background processes.
After completing changes to the three zones I identified as potential sub-applicaitons, I'll go back over my outline and analysis, and start working on addressing the issues I've listed. While I'm sure there we will still have growing pains as we scale up, I feel the timeinvested now not only delays that experience, but also prepares us for that day. Another plus from this process, is that more of the application is getting documented, something that rarely gets done with Web projects.
- ADD TO:
Mobile - Web - Media
Friday, Jan 19, 2007 12:08:51 PM
MySpace and Handling Millions of Users
Yesterday, I read a very interesting article which covers how MySpace scaled itself to handle millions of users and billions of pageviews. This
article probably won't be very interesting unless you work with the Web
either through building sites, or managing a network.
Read the article online: http://www.baselinemag.com/article2/0,1540,2082921,00.asp
certainly made some bad choices along the way, but I can't even imagine
the level of stress their engineers must have been under while dealing
with the daily growth problems. It must have been insane.
point out that if they used LAMP, MySpace would have less problems,
errors, and could scale better... I'm not so sure about that though.
When you consider the numbers of requests coming in, the constant
publishing of content on the site, and the ever growing userbase, it's
nearly an impossible situation.
that they've finally settled on Microsoft products and technology to
stay online - almost fully ditching ColdFusion. Does this mean
ColdFusion can't scale? No, not at all. As you'll see in the article,
they took a look at every function and worked to optimize the code on
the last rewrite. If they did this originally with the ColdFusion
design, I'm sure they would have been able to keep scaling. Plus, the
state of their application at the time when they selected to move to
.NET sounded like spaghetti code.
No matter what language you develop in, the article's infomative and intersting.
- ADD TO:
Mobile - Web - Media
Monday, Jan 15, 2007 4:41:20 PM
Made With Adobe Product Buttons
While finishing up the SocialNetwork.in site for it's beta launch, I thought it would be good to state in the about section that the site was built with ColdFusion, and that the widget will be built using Flex. But after searching around the Web, I didn't find a good set of "Made With" buttons to put on the site. A made with button usually states what technology was used to develop whatever it is you are looking at.
I tend to use Adobe products, and promote them too... why? I'm not sure? Personally, I usually have to buy their software used off of ebay, as it's too expensive when you buy direct. Actually, yes, I do know why I promote them, it's because I can develop very fast with their products, and I can usually rely on them.
Since I didn't find any buttons to use, I made my own, and carried on to create a whole set for most of Adobe's products. Here's a ZIP file containing all the buttons - free to grab and use!
- ADD TO:
Mobile - Web - Media
Monday, Jan 15, 2007 2:33:25 PM
SocialNetwork.in a Social Aggregator and Directory Launches
As you can see, I haven't been blogging much over the last month, and today, I'm announcing why!
I'm launching a new site today at:
It's a cross between a social aggregator, and a social directory, with
ratings and reviews.
While the site doesn't aggregate content from social sites, it does help
people aggregate links to their social profiles, links to friends
profiles, or simply to just link to their favorite social media sites.
Both profiles on the site, and any links a member adds to their
SocialNetwork.in profile can be made private, allowing people to
aggregate their online profiles in privacy.
The site also provides a means for members to rate and review social
sites and services. Ratings are based on a 1-10 scale, with the site
displaying averages. Clicking on any 'line chart' icon next to any site,
will open a space where you'll be able to compare the selected site with
up to 5 others by generating Alexa comparison charts (once www.compete.com
offers a chart widget, I'll add it as an option or replace Alexa's
charts if works well).
Sites and services are also organized by category, in addition to having
their own site profile. On the site profile pages, visitors can get a
better view of the service by reading over reviews, clicking through
Alexa traffic charts, and viewing data from both del.icio.us and
I hope to have a widget available by the end of February, which will
list your public profile links, and another update soon after, where
I'll add support for RSS feeds per each site on SocialNetwork.in - and
possibly for members too.
There are other players in this space, offering deeper integration with
social sites, mainly focusing on content and friend aggregation
(Mugshot.org, ProfileLinker.com, Spokeo.com). I may offer deeper
integration with key sites once the service grows and users request
these features, but for now, I'm looking to provide profile URL/link
support for all social sites. Currently, the site has support for 97
social sites, with more on the way.
For those who are monitoring the social network/media market, I hope
that this site becomes a tool that makes their job easier. This was the
idea behind the comparing options on the site, in addition to providing
members with a rate and review tool.
The primary criteria for which sites get added to SocialNetwork.in, is
that they need to provide a profile site, and a Username, or UserID.
Although, I've also added support for sevices like Netflix, which offer
RSS feeds to your data. I plan on adding a means for displaying RSS
feeds on your SocialNetwork.in profile at a later date.
On the tech side of things:
- Developed in ColdFusion MX7, MSSQL Server
- Built in 35 days by 1 developer
- Data/Services integrated: del.icio.us, Alexa,
Visit the site - take it for a spin - sign up - and if you have some feedback, let me know!
- ADD TO:
Mobile - Web - Media
Sunday, Oct 22, 2006 6:32:59 PM
Social Publishing or Hyper Recording I was reading a post on Fred Wilson's blog, AVC called 'Hyper Recording.' Where he writes:
QUOTE: The other night at the Red Hot Chili Peppers show I noticed a bunch of cameramen with super high end video cameras right in front of the stage recording the show. And then I noticed thousands of people with cell phones and digital cameras doing the same thing.What caught my attention, was that I had similar thoughts about this in a small club last year:
And a thought popped into my mind. The super high quality video that was being shot either by the band or with the permission of the band may never see the light of day. But the video being shot by the audience was going to be up on Flickr, YouTube, Google, etc within hours.
As I watched people capturing video and photos, sending them off to friends, I though..."It's too bad the band isn't able to participate in this exchange of media... that it's not aggregated so that they could benefit from it."At first, I referred to the idea as "Fan Blogging," but it has evolved to more than just a tool for bands. I now see it as "Social Publishing."
I made two blog posts back in March of this year where I explain the concept in more detail
Here's a Summary: To provide the means for an audience and/or participants of an event or location gathering to self document using photos and video from mobile devices. In addition, this system provides a greater opportunity for follow up sales and marketing through the Website which displays the user generated content. Lastly, depending on the settings, a contact list can also be generated which helps one grow their network.
Later in Fred's post, he refers to a friend in the music business who stated the following:
QUOTE: artists will never allow recordings of their live shows to be released without their permissionHow about "Grateful Dead" and "Phish" - two bands who allowed fans to record nearly everything. Both went on to be top grossing bands, still selling CDs, even live CDs and live digital downloads. Take a look at: http://www.livephish.com/ if more bands provided a live digital download service like this, they'd be able to generate more revenue, give the fans what they want, and provide a hub where they can sell dry goods, like shirts, etc.
A live performance is filled with opportunities to communicate with an audience, and usually, the only thing I see is people handing out flyers at the door trying to get you to visit some artist you aren't interested in. If fans published photos and video to a site, a good percentage would visit the site to see their photos/video. Some would forward them onto others, and some might even buy something on the band's site.
This Social Publishing or, Hyper Recording as Fred calls it, will be made available as a service next year at http://www.ArtistServer.com/
- ADD TO: