Azure Essentials: Data analytics



welcome to our essentials in the next few minutes we'll explore the services in Azure to help you analyze your data across both structured and unstructured data and she can help you whether you typically work structured tabular data or if you're looking to reason over large or complex unstructured big data coming from devices services and applications that require a more sophisticated level of processing and scale beyond traditional data warehousing Asia has a comprehensive set of services to ingest store and analyze data of almost all types and scales spanning table file streaming and other data types the azure platform provides tools across the data analytics lifecycle this allows you to ingest data into Azure use robust services for batch ingestion or real-time ingestion so that you can capture events as they're being generated from your devices and services store structured or unstructured data globally at virtually unlimited scale train and prepare your data in data stores to derive insights and create predictive and prescriptive models on your data using machine learning and deep learning techniques furthermore you can extend these capabilities to real-time processing of streaming or log data can even leverage artificial intelligence or AI with machine learning and cognitive services for automated machine analysis and finally you can serve and publish this analyze data to an operational or analytical store to help with visualizing as part of reports and dashboards your apps can also leverage this data directly and securely while meeting your performance needs so let's walk through these services in a bit more detail the first step in data analysis is connecting disparate data sets from multiple sources and ingesting them into Azure your data might originate in your data center and cloud services or spam both now for batch ingestion of your data as your data factory is the primary service that you'll want to use this is an ingestion orchestration and scheduling service and it determines what happens when certain events occur on which engines to use to analyze and optimally process your data it allows you to create sophisticated data pipelines right from ingestion of the data through to the processing storing and then making it available for your end-users an apt to tap into there are other data movement capabilities on Azure too if you've got a massive one-time upload you may want to use the azure import/export service manage the bulk loading of large datasets into Azure blob storage and Azure files by shipping drives to an agile datacenter if your structured data the azure data migration service migrates data from your on-premises structured databases directly into Asia maintaining the same relational structures leveraged by your current apps as your also has engines for ingesting real-time data streams now these engines are capable of ingesting data at a very fast pace and catering to your processing needs down the line as your event hubs enables large-scale telemetry and event ingestion with durable buffering and low latency from millions of devices and events as your iot hubs our device to cloud telemetry data service to track and understand the state of your devices and assets and if you've got custom operations to perform and you want to scale out your ingestion engines with custom logic Roger also supports the open source Apache Kafka in HD insight as a managed high throughput lower latency service for real-time data and of course you can use the azure CLI command line interface to programmatically target and ingest multiple data formats into Azure if you're a developer api's can be called using the azure software development kit or SDK to bring in your data now all the tools and services I just described can bring data into Azure and as you plan for how you ingest data they'll also plan for where and how the date will be stored in Azure as your blob storage can store massive data sets irrespective of their structure or the lack of it and keep it ready for analysis including video images scientific data sets and more and as a managed service you don't need to worry about the knobs and dials it just takes care of itself now if you've got particularly demanding analytical throughput requirements or you have huge file sizes needs to be optimized for analysis you want a specialized big data store as your data Lake store can serve that purpose lets you analyze all your data both structured and unstructured with a very high throughput generally desired by analytics engines it can store trillions of files and a single file can be larger than one petabyte size now for operational and transactional data in structured or relational form you can use Azure sequel DB this works like sequel server but as an azure service so you don't need to worry about managing or scaling your hosts infrastructure of course you can keep existing database apps in hosted Windows or Linux based machines for analytical data that's been aggregated over the years as your sequel data warehouse provides an elastic petabyte scale service which lets you dynamically scale your data either on-premises or in Asia now for no sequel capabilities if you're bringing in data that schema agnostic as your cosmos DB is a turnkey globally distributed no sequel DB service that allows you to use key value graph document data together with multiple consistency levels to cater to your app requirements whatever the need Roger has an optimal store for you interestingly all these stores integrate seamlessly to the analytics engines a sources of data with your data now stored in Asia there are many analytics options for training and preparing your data spanning from super scalable and involved approaches to data engineering through to automated machine analytics on Cerberus infrastructure I'll start with some of the open source analytics capabilities as your data bricks is an optimized Apache spark based analytics cluster service offering the best of SPARC with collaborative notebooks and enterprise features it integrates with Azure Active Directory and we also give you the native connectors to bring in other Azure data services I should later Brix is your hub of spark based analytics whether it's batch streaming or machine learning also we've got HD insight a managed cluster service for a variety of open source big data analytics workloads helps you clean curates process and transform your data in addition to scaling your machine learning workloads using HD insight you can create scale out clusters for Hadoop spark hive HBase store and Microsoft our server without the need to monitor and administer the underlying infrastructure for scale out compute engines similar to traditional sequel infrastructure data Lake analytics lets you develop and run large-scale parallel data transformation and processing programs in you sequel over petabytes of data from your data Lake you can even leverage the familiarity and extensibility of you sequel to scale your machine learning models from our or Python to work against massive amounts of data most importantly it's a serverless environment so you request and leverage compute resources on a per query basis and don't have to worry about maintaining large clusters which makes scaling and parallel execution easy I sure also has engines for processing real-time data streams now to analyze data logged in real-time from devices sensors or more either stream analytics offers a powerful event processing engine together with event hubs allows you to ingest millions of events and find patterns detect anomalies power dashboards or automate event-driven actions in real-time with the simplicity and familiarity of a sequel like language to process real-time streams Asha hdinsight and Azure data brakes also allows you to leverage streaming capabilities within the scale-out processing engines like structured streaming a spark for more advanced analytics Azure machine learning and Microsoft machine learning server provides you the infrastructure and tools to analyze data create high quality data models train and orchestrate machine learning to build intelligent apps and services addition to these tools scale out cluster technologies like agitator bricks also allows scalable machine learning with spark Annelle and deep learning libraries beyond this we've also built a number of first level AI services called cognitive services providing pre-built intelligent services for vision speech text understanding and interpreting finally once you've been able to analyze and derive insights from this data you'd want to serve this enriched data to your users within actually the best destination for all of this analyzed data is actual sequel data warehouse where you can now combine new insights with historical trends and drive a targeted conversation by maintaining one version of data for your organization as a sequel data warehouse not only supports seamless connectivity to the analytics tools and services it also integrates well with business intelligence tools for example as your analysis services and power bi which provide powerful options to find and share through the data insights if the analyzed data contains insights valuable to your end consumers these can be populated into the operational stores like as a sequel DB and Azure cosmos DB so the web and app experiences be augmented by those insights you can even pipe data directly to your apps with Azure platform tools for developers including visual studio as your machine learning workbench or custom service apps and services using Azure functions with a chef you can ensure that data is consumed only by intended users and groups securely authenticate by erasure Active Directory while network performance SLA privacy requirements are met using our accessory you can even hold the keys to your data once it's in the cloud without your key management services so that was an overview of the key services in Azure that comprised the data analytics lifecycle if you're interested in data visualization and machine learning these topics are covered in more detail in separate overviews on Azure essentials of course we're constantly adding new topics on essentials so please keep checking back for more and you can continue your learning with a hands-on learning series of a link show thanks for watching

2 Comments

  1. sudeesh varier said:

    Good video

    June 26, 2019
    Reply
  2. Harriet Shearsmith said:

    Loved watching this, thanks for the fab video lovely x

    June 26, 2019
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *