USE CASES OF AWS SQS IN INDUSTRY

Amazon SQS provides several advantages over building your own software for managing message queues or using commercial or open-source message queuing systems that require significant up-front time for development and configuration.

These alternatives require ongoing hardware maintenance and system administration resources. The complexity of configuring and managing these systems is compounded by the need for redundant storage of messages that ensures messages are not lost if hardware fails.

In contrast, Amazon SQS requires no administrative overhead and little configuration. Amazon SQS works on a massive scale, processing billions of messages per day. You can scale the amount of traffic you send to Amazon SQS up or down without any configuration. Amazon SQS also provides extremely high message durability, giving you and your stakeholders added confidence.

BENEFITS OF USING SQS

For Serverless developers, using SQS generally provides a wealth of benefits, which you can read about below.

Scalability

Your SQS queues scale to the volume of messages you’re writing and reading. You don’t need to scale the queues; all the scaling and performance-at-scale aspects are taken care of by AWS.

Pay for what you use

When using SQS, you only get charged for the messages you read and write (see the details in the Pricing section). There aren’t any recurring or base fees.

Ease of setup

Since SQS is a managed service, so you don’t need to set up any infrastructure to start using SQS. You can simply use the API to read and write messages, or use the SQS <-> Lambda integration.

Options for Standard and FIFO queues

When creating an SQS queue, you can choose between a standard queue and a FIFO queue out of the box. Both of these queue types can be useful for different purposes.

Automatic deduplication for FIFO queues

Deduplication is important when using queues, and for FIFO queues SQS will do the work to remove any duplicate messages for you. This makes FIFO queues on SQS suitable for tasks where it’s critical to have each task done exactly once.

A separate queue for unprocessed messages

This feature of SQS is useful for debugging. All messages that couldn’t be processed are sent into a “dead-letter” queue where you can inspect them. This queue has all the usual integrations enabled, so you can subscribe to it using an AWS Lambda event, for example, to send a notification when an item can’t be processed.

What is SQS used for?

The most common ways to use SQS, and of course other messaging systems, in cloud applications are:

Decoupling microservices.

In a microservice architecture, messages represent one of the easiest ways to set up communication between different parts of the system. If your microservices run in AWS, and especially if those are Serverless services, SQS is a great choice for that aspect of the communication.

Sending tasks between different parts of your system.

You don’t have to be running a microservices-oriented application to take advantage of SQS. You can also use it in any kind of application that needs to communicate tasks to other systems.

Distributing workloads from a central node to worker nodes.

You can frequently find messaging systems in the flows of distributed large workloads like map-reduce operations. For these kinds of operations, it’s essential to be able to maintain a queue of all the tasks that need to be processed, efficiently distribute the tasks between the machines or functions doing the work, and guarantee that every part of the work is only done once.

Scheduling batch jobs.

SQS is a great option for scheduling batch jobs for two reasons. First, it maintains a durable queue of all the scheduled jobs, which means you don’t need to keep track of the job status — you can rely on SQS to pass the jobs through and to handle any retries, should an execution fail and your batch system returns the message to the queue. Second, it integrates with AWS Lambda; if you’re using AWS Lambda to process the batch jobs, SQS automatically launches your Lambda functions once the data is available for them to process.

Some important concepts associated with SQS are-

  1. SQS Visibility Timeout-

A period of time that prevent other consumers from processing the message is called SQS visibility timeout

2. Deadletter Queue:

Dead-letter queue helps in handling message failure as it isolates the failure messages such that we can determine why the processing didn’t succeed

INDUSTRIAL USE CASE

redBus Case Study

RedBus is an Indian online bus ticketing platform providing ticket booking facility through its website, iOS and Android mobile apps. Headquartered in Bengaluru, India, it connects bus travellers with a network of over 2500 bus operators, across India, countries in South East Asia and Latin America. The company also offers software, on a Software as a Service (SaaS) basis, which gives bus operators the option of handling their own ticketing and managing their own inventories. To date, the company says they have sold over 30 million bus tickets and has more than 1750 bus operators using the software to manage their operations.

Challenges Faced :

  • The company previously ran its operations from a traditional data center by purchasing and renting its systems and infrastructure.
  • In addition to the expense, several logistical problems evolved from this arrangement.
  • The biggest problem was that the infrastructure could not effectively handle processing fluctuations, which had a negative impact on productivity.
  • Additionally, the procurement of servers or upgrading the server configuration was an extremely time-consuming endeavor.
  • RedBus receives complaints from customers mainly regarding Refunds, Cancellations, Operator queries etc… The process of classifying these emails and redirecting them to the respective customer service agents/queue becomes a humongous task. This also increases during peak days (especially holidays and weekends).

Why RedBus went for Amazon Web Services?

After testing the AWS solution on a small application for several months, the travel agency determined that it was very workable and convenient. Although redBus was quite enthusiastic about the on-demand instances and variety of instance types, several other features cemented the company’s decision to migrate completely to AWS. These features included the ability to easily manage access to servers through security groups, the easy-to-use, self-service management console, the concept of Elastic IPs, and superior support.

The company has incorporated many of the AWS products into its solution, including Amazon Elastic Compute Cloud (Amazon EC2), Elastic Load Balancing, Amazon Relational Database Service (Amazon RDS), Amazon Simple Storage Service (Amazon S3), Amazon Elastic Block Store (Amazon EBS), and Amazon CloudWatch. Charan Padmaraju, Chief Technology Officer believes that “with features like Elastic Load Balancing and multiple availability zones, AWS provides the required infrastructure to build for redundancy and auto-failover. When you incorporate these in your system/application design, you can achieve high reliability and scale.”

The Benefits :

1. Reduced costs by up to 40%

Since migrating to AWS, redBus has seen measurable improvements in the bottom line. Padmaraju says,

“By scaling up and down dynamically based on the load, we maintain performance as well as minimize cost. With the time savings that the IT and development staffs obtain from the AWS solution, AWS gives us an overall cost benefit of about 30–40%.”

2. Reduced website latency by 4x

Padmaraju adds,

“By hosting at [the AWS Asia Pacific (Singapore) region], redBus.in gained significantly in terms of website performance by way of reduced latency (about 4x). This is a great advantage when the customers are from India.”

3. Able to instantly replicate test environments, which in turn reduces time to market

Of the many excellent characteristics of AWS, perhaps the most significant to redBus is the ability to “instantly replicate the whole setup on demand for testing by creating and destroying instances on demand for experimentation, thereby reducing the time to market.” Less time to market translates to increased profitability and success.

The travel agency anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) for monitoring, alerts, and intercommunication.

“Amazon SQS is an especially good solution for enabling messaging between external applications and our applications,” says Padmaraju.

Since joining forces with AWS, redBus has gained the freedom to experiment on new solutions and applications at minimal cost, increased the efficiency of its operations, and improved its profitability.

Results :

With the time savings that the IT and development staffs obtain from the AWS solution, AWS gives us an overall cost benefit of about 30–40%.

-Charan Padmaraju
Chief Technology Officer, redBus

INDIVIDUAL USE CASE

This part of the article provides an overview of how a developer used AWS Lambda and Amazon SQS to download 194,204 files (i.e. CSV format) to Amazon S3 at a speed 350X faster than on his local machine!

The Problem

The developer tried to build a web application to download historical Canadian weather data. As part of the application structure, heneeded to download data from a third-party server for 8,324 independent weather stations and store the data in a database (Amazon S3).

It seems easy, right? Just create some code, execute, and voila you have the data. WRONG.

Each station has multiple years of weather data and the third-party server limits downloads to 1-year increments. In short, downloading the dataset for all 8,324 stations would mean downloading 194,204 separate files (i.e. one file for each year of data at a station), combining data for matching stations, and pushing data to Amazon S3.

The issue with downloading the data is not the file size, the entire dataset is only 4.4 GB, but instead the download speed and network latency. On my local machine, it takes approximately 1.7 seconds to download 1-year of data and push the data to Amazon S3. This means using my local machine to download all 194,204 files and push to Amazon S3 would take 3.9 days, assuming there are no network interruptions or failed downloads. 😲

WE CAN DO BETTER!

The Solution

It’s obvious that downloading the data on a local machine is impractical and not robust, especially with such easy access to cloud computing. Ultimately, he decided to use AWS Lambda to execute his Python code and Amazon SQS message queue to trigger Lambda events and communicate download tasks.