Choose Cloud Brokers Wisely
Demystifying Azure's Event Grid, Event Hub, Storage Queue, and Service Bus
Hi all! Have you noticed how many options exist on azure for messaging and event routing? It can be quite overwhelming and I’m putting an end to this confusion once and for all...
But First…
Last weeks’ challenge
What if we deploy a dedicated virtual machine and use that instead? Could we migrate a collection of Azure functions into 1 VM? What are the tradeoffs there?
There are some benefits
Stable Cost - $70-$130 for keeping a basic VM running 24/7. But will the basic tier handle your compute requirements?
Full Control - scheduled tasks, scripts, web services, any language you want, etc.
But you lose a lot
Because you don’t have access to standard bindings, you’ll spend more time coding to integrate and enable communication between azure services (or other).
Lack of isolation or ability to independently scale.
You’d have to deal with configuring and managing your host machine (IIS, certificates, etc.)
If I had to migrate a collection of azure functions, I would turn them into an Azure App Service instead.
Are you on the Market for a Broker?
Message brokers help coordinate communication between multiple clients or services. They enable several benefits such as load distribution, storage for later, delivery guarantees, decoupling, and much more! The following services serve this general function, but are designed for different use cases.
Introducing the Contenders
Before we step into our metaphor of the week, let’s quickly introduce azure services involved in this function:
Azure Event Grid allows you to route discrete events.
Azure Event Hub is a managed instance of Kafka which is meant for big data ingestion and continuous events.
Azure Service Bus enables consistent communication between microservices.
Azure Queue Storage uses your storage account to create a backlog of asynchronous communication
Our Robot Orders a Smoothie
After a full day of work our robot gets hungry (This is the year 2050 where they eat more than just electricity!).
Our robot (microservice A) is sending a message with a description of the request and an expectation of work from the staff on the other side (microservice B) of the connection.
He’s not just broadcasting to the world “I’m hungry” and sitting around (that would be more like Event Grid), but he is being very specific about what he expects from the other party, won’t take no for an answer, and will keep the line open until the transaction is completed.
Think of their phone line and interaction as Azure Service Bus. The cellphone carrier and infrastructure can be scaled and reconfigured as demand requires, there is an expectation that their time critical voice streams maintain the same order.
Practical Uses
Your backend systems communicate directly but spikes in traffic are affecting performance. Consider Service Bus to increase throughput and decompose your services.
Your service deals with financial transactions and they need to be distributed among a collection of services while maintaining order of events.
You need to synchronize events coming from different sources
Load balancing of events - you want to scale the amount of instances consuming the requests
The Staff Gets to Work
The order shows up in the pending orders screen and the staff gets to work!
Each employee was watching (or subscribed) the pending orders screen. However the actions they may take as a consequence of the new item might be completely different.
Think of this as Azure Event Grid.
One robot might wash the blenders, another cuts the fruits, manager oversees the operation. The key is Azure Event Grid deals with lightweight discrete events.
The subscribers are not waiting for a collection of events to appear in a specific order, and the producer doesn’t typically care what is done with the alert.
Practical Uses
Whenever a document is created, wake up an azure function to open the document and remove confidential information.
Notify an external service endpoint of events occurring within our azure tenant.
Notify all members of a security group that a file in a storage account was deleted
When an image is uploaded to blob storage, trigger a serverless workflow through an Azure Logic App
Use Azure Event Grid’s publisher subscriber model to enable communication between mobile or IoT clients using MQTT. One good example of this is Spotify’s mobile and desktop apps syncing their play/pause state.
Smoothie Shops Need Water
As the staff prepares the order, they rely on the water distribution system which delivers millions of water droplets to everyone in the area.
We can imagine this staff uses way more water throughout the day and are therefore charged more than the customers who are calling them. But how does the county know that?
Their Water Use is Analyzed
The city’s got metering systems in place to monitor and analyze usage data so they know how much to charge. This kind of time series analysis is what consumers of Azure Event Hub are interested in.
It’s All About Volume
Ingesting and monitoring a continuous flow of big data. Whether it is sensor data, or logs - the consumers of the data typically care about a series of events rather and want to analyze a time series.
Practical Uses
You manage an Internet of Things solution that requires consolidating and analyzing millions of sensor data points coming from many devices.
You’re current data ingestion system is struggling and want to leverage partitioning features, for example: partition by geography.
Ingest logs from all devices in your ecosystem for big data analysis through Azure Stream Analytics
Order is Ready!
Finally the staff completes the smoothie; however, the customer has not yet arrived!
What should they do? Store it in a Queue ( Azure Storage Queue !?) until they are here.
Practical Uses
Tasks generated by one service are not time-critical and can be handled asynchronously
Having a backup location to route messages or requests while a system is down.
Conclusion
Message brokering services on Azure have many similarities, but here are the key differences:
Event Hub handles continuous data streams.
Event Grid is meant for for discrete, light-weight status updates.
Service Bus is best suited for real-time microservice coordination.
Queue Storage supports Asynchronous Backlogs.
Resources
Comparison of Messaging Services
AZ-204 Develop Event Based Solutions
AZ-204 Develop Message Based Solutions
Challenge of the Week
I can easily write my own code to implement producer-consumer or publisher-subscribe models between my client and server applications, why should I add more complexity and cost to my system by using any of these services?
Do you agree with this claim? If you don’t, explain why you’re right?