After attending the Microsoft Ignite conference last week in Chicago, what became readily apparent to me was the vast number of products geared towards this new onset of “streaming data”. I saw various implementations of Internet of Things (“IoT”), and how sensors are amassing large amounts of data as companies grow. Sensors can be used to measure any type of data: humidity, temperature, vibration, light intensity, wind force, salinity, and much more. Third party consumer devices for smart phones such as fit bands can constantly send heart rate, sleep patterns, and steps taken to any end points that are set up.
In the IoT world, this is the concept of Telemetry. According to TechTarget, Telemetry is defined as
The wireless transmission and reception of measured quantities for the purpose of remotely monitoring environmental conditions or equipment parameters
In a traditional BI environment, you would want to ensure you capture this data, so that it can be loaded into your BI reporting warehouses or marts for analysis. But what if you want to generate real-time actionable insights? Microsoft has released several products on the Azure Cloud that address these types of use cases. Event Hubs, Stream Analytics, and Power BI can be leveraged to handle large influxes of IoT messages and stream them to data stores and live dashboards. By shifting the heavy lifting of your application to these modular, scalable components, your architecture will be less fault tolerant and allow you to focus on what you want to collect and its analysis. A perfect example of this is its use behind Microsoft’s How Old? Demo at Build, which was able to scale to millions of request as it went viral and not crashing due to the load.
The first part of this architecture is the Event Hub. Event Hubs enable you to ingest a large number of events or messages with low latency. It stores them for a defined retention period so they can be taken in from other parts of your architecture. This defined endpoint gives you a reliable way to process data-streams of any quantity and focus more on the processing of the data rather than ensuring your application will capture the data-stream.
After these events are captured, you want a scalable way to route these events to your various downstream applications in a near real-time way. Stream Analytics allows you to create queries that execute on streaming data as it becomes available. Think about it in terms of a moving Group By statement on any of the information you are capturing in a sliding window of time. With Stream Analytics, you can create SQL-like statements that routes to several different destinations (e.g. Azure Blobs, Database Tables, Power BI Dashboard Datasets). As the amount of messages increases in a given period of time, you can dynamically scale out the amount of processing power needed to consume the events to ensure no data loss. In a real world scenario, you can forward your data stream to Power BI dashboards for real-time reporting, Azure Database Tables to keep retention of your data, and another process to track anomalies that trigger notifications to end users.
Power BI is a dashboard/reporting tool that has made many strides in its latest iteration. In my opinion, one of the cool new features is the RESTful APIs developed that can interact with the underlying datasets displayed on reports. There are API actions that can add rows to an existing dataset that reports are built on top of, which gets reflected live on a given dashboard. Anyone can make API calls to the exposed Azure endpoint, and it will reflect in the dashboards. Stream Analytics allows you to hook directly into this mechanism pretty easily. As the stream of data is constantly flowing to Power BI, it will reflect on the charts as someone is looking at it. This will give you the ability to pair historical data vs real-time or create live notifications on a dashboard from a Spout on Storm.
By letting Microsoft Azure do the heavy lifting in your architecture, you can focus on what really matters. Keep an eye out for a future blog post showing an internal demo incorporating these cool new technologies!