Amazon Simple Queue Service (Amazon SQS) makes it easy and inexpensive to enhance the scalability and reliability of your cloud application. In this session, we demonstrate design patterns for using Amazon SQS in conjunction with Amazon Simple Storage Service (Amazon S3), Amazon DynamoDB, Amazon Elastic MapReduce, Amazon Relational Database Service, and Amazon Redshift. Shazam will share their experience of combining Amazon SQS with Amazon DynamoDB to support a Super Bowl advertising campaign.
This is a technical architect's case study of how Loggly has employed the latest social-media-scale technologies as the backbone ingestion processing for our multi-tenant, geo-distributed, and real-time log management system. This presentation describes design details of how we built a second-generation system fully leveraging AWS services including Amazon Route 53 DNS with heartbeat and latency-based routing, multi-region VPCs, Elastic Load Balancing, Amazon Relational Database Service, and a number of pro-active and re-active approaches to scaling computational and indexing capacity.
The talk includes lessons learned in our first generation release, validated by thousands of customers; speed bumps and the mistakes we made along the way; various data models and architectures previously considered; and success at scale: speeds, feeds, and an unmeltable log processing engine.