Search results
Sep 13, 2023 · Data ingestion is the process of collecting, importing, and transferring raw data into a system or database where it can be stored, processed, and analyzed. This data can originate from disparate sources, including sensors, web applications, databases, logs, and external data providers.
- Scenario details
- Considerations
- Deploy this scenario
- Contributors
- Next steps
- Related resources
Organizations often deploy varied and large-scale technologies to solve business problems. These systems, and end-user devices, generate large sets of telemetry data.
This architecture is based on a use case for the media industry. Media streaming for live and video-on-demand playback requires near real-time identification of and response to application problems. To support this real-time scenario, organizations need to collect a massive telemetry set, which requires scalable architecture. After the data is collected, other types of analysis, like AI and anomaly detection, are needed to efficiently identify problems across so large a data set.
When large-scale technologies are deployed, the system and end-user devices that interact with them generate massive sets of telemetry data. In traditional scenarios, this data is analyzed via a data warehouse system to generate insights that can be used to support management decisions. This approach might work in some scenarios, but it's not responsive enough for streaming media use cases. To solve this problem, real-time insights are required for the telemetry data that's generated from monitoring servers, networks, and the end-user devices that interact with them. Monitoring systems that catch failures and errors are common, but to catch them in near real-time is difficult. That's the focus of this architecture.
In a live streaming or video-on-demand setting, telemetry data is generated from systems and heterogeneous clients (mobile, desktop, and TV). The solution involves taking raw data and associating context with the data points, for example, dimensions like geography, end-user operating system, content ID, and CDN provider. The raw telemetry is collected, transformed, and saved in Data Explorer for analysis. You can then use AI to make sense of the data and automate the manual processes of observation and alerting. You can use systems like Grafana and Metrics Advisor to read data from Data Explorer to show interactive dashboards and trigger alerts.
Reliability
Reliability ensures your application can meet the commitments you make to your customers. For more information, see Overview of the reliability pillar. Business-critical applications need to keep running even during disruptive events like Azure region or CDN outages. There are two primary strategies and one hybrid strategy for building redundancy into your system: •Active/active. Duplicate code and functions are running. Either system can take over during a failure. •Active/standby. Only one node is active/primary. The other one is ready to take over in case the primary node goes down. •Mixed. Some components/services are in the active/active configuration, and some are in active/standby.
Cost optimization
Cost optimization is about reducing unnecessary expenses and improving operational efficiencies. For more information, see Overview of the cost optimization pillar. The cost of this architecture depends on the number of ingress telemetry events, your storage of raw telemetry in Blob Storage and Data Explorer, an hourly cost for Azure Managed Grafana, and a static cost for the number of time-series charts in Metrics Advisor. You can use the Azure pricing calculator to estimate your hourly or monthly costs.
Performance efficiency
Performance efficiency is the ability of your workload to scale to meet the demands placed on it by users in an efficient manner. For more information, see Performance efficiency pillar overview. Depending on the scale and frequency of incoming requests, the function app might be a bottleneck, for two main reasons: •Cold start. Cold start is a consequence of serverless executions. It refers to the scheduling and setup time that's required to spin up an environment before the function first starts running. At most, the required time is a few seconds. •Frequency of requests. Say you have 1,000 HTTP requests but only a single-threaded server to handle them. You won't be able to service all 1,000 HTTP requests concurrently. To serve these requests in a timely manner, you need to deploy more servers. That is, you need to scale horizontally. We recommend that you use Premium or Dedicated SKUs to: •Eliminate cold start. •Handle requirements for concurrent requests by scaling the number of servicing virtual machines up or down.
For information about deploying this scenario, see real-time-monitoring-and-observability-for-media on GitHub. This code sample includes the necessary infrastructure-as-code (IaC) to bootstrap development and Azure functions to ingest and transform the data from HTTP and blob endpoints.
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal authors:
•John Hauppa | Senior Technical Program Manager
•Uffaz Nathaniel | Principal Software Engineer
Other contributors:
•Mick Alberts | Technical Writer
Nov 7, 2023 · Follow. 5 min read. ·. Nov 7, 2023. Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. It represents the...
Jul 19, 2023 · What is Data Ingestion? Data Ingestion is the process of obtaining, importing, and processing data for later use or storage in a database. This can be achieved manually, or automatically using a combination of software and hardware tools designed specifically for this task.
Sep 7, 2022 · Azure Event Hubs acts like a “front door” for an event pipeline, often called an event ingestor. An event ingestor is a component or service that sits between event publishers and...
Introduction. Data ingestion is the process of transporting data from one or more sources to a target site for further processing and analysis. This data can originate from a range of sources, including data lakes, IoT devices, on-premises databases, and SaaS apps, and end up in different target environments, such as cloud data warehouses or ...
Sep 7, 2016 · The event ingestor is designed to sit between producers of events and consumers of events. At the beginning of the pipeline, the Event Hub collects the data. Then once it is collected, an...
