Vishal Shah - #V's Blog

Web Scale, Online-Offline Architecture Pattern/Template for Tiny to Large Scale Products

Reading about and practicing myself many many architectural styles & pattern, I think I have something rather interesting to share.

I have identified a very standard architecture pattern, most online-offline systems can use (an online-offline system is a word I made up :) that supports both online/realtime & offline processing).

Its not new or anything revolutionary. It just works. digg follows most of that standard/canned architecture - app server - caching layers - storage - messaging/queues - async/offline hadoop processing.

Such architecture supports everything from small to large use cases and products. It supports processing large amount of data via hadoop while quick data queries uses the cache and datastore. App server is where the “online” business logic is. Offline business logic is in hadoop.

Its so generic & balanced that you can scale individual layers/components without affecting rest of the system. Also it fits in our SOA, where apps on the top are nothing but service end-points with no UI, or they can be java/node/php… apps with a UI.

I am not proposing, this is one stack that everybody should use, but its sort of an architecture pattern that works for many problems. You can skip components as you wish and add them later. For ex, for a mobile app/service we are currently building, we have no need for offline processing & a dedicated messaging layer - hence the hadoop & messaging layer goes away! We can add in the future if we want to - to add lots of stats, processing, etc..

So, you can scale it back to just the app server, caching and data store if you need, which is not that exciting or revealing, but you know you can scale it to ridiculous extent, by adding the async messaging & offline MapReduce/Hadoop processing layer. And bingo!!

Here’s the digg diagram. I chose not to build my own diagram because I am lazy. Digg’s diagram has most of it. Don’t get too carried away with their arrows. The important thing to note are the components and layers and the purpose they serve.

— Vishal