Apache Storm

Apache Storm is an one of the best opensource, real-time processing and distributed computation system available. It comes with best fault tolerant and stateless in nature, increases the performance.

A few days ago, I had to design a high volumes write based fan-out architecture. For people who are new to this school of thought, I will explain it very simply. Write based fan-out architecture tries to use all business logic during write time. The idea is to keep views for each user and each use case ready, so that when someone wants to read the data , they don't have to apply complex logic. Read then becomes simple and most often you can guarantee constant read time. Twitter's architecture is a write based fan-out system. Without getting into the details of the requirements , here is the 60k foot overview. High volumes of writes Almost constant read time required Had to be fault tolerant and scalable on commodity hardware Free text search and social graph traversal was also required. Real time analytics The architecture we designed involved three databases. MongoDB for storing the incoming data. Redis for storing the datasets for each user designed for that user. ElasticSearch for storing the text which required free text or partial text searches. For each incoming dataset, there was business logic which decided which datasets to populate in Redis(based on social graph connections) and what to extract and store in ElasticSearch for free text searches. Sounds simple enough! Given this, I decided to use Apache Kafka as the message broker for its speed and reliability and then use Storm for processing the data and implementing the write based fan-out architecture. The devil is in the details. And that is what I intend to share here. Before using Kafka and Storm there are a few things that you should understand about each.