For example, create the primary Azure Databricks workspace in East US2. Provision multiple Azure Databricks workspaces in separate Azure regions. To create your own regional disaster recovery topology, follow these requirements: The compute (Azure Databricks) and storage sources must be in the same region so that jobs don’t experience high latency. It is ephemeral in nature, meaning that while your data is still available in Azure Storage, the compute (Azure Databricks cluster) can be terminated to avoid paying for compute when you don’t need it. Azure Databricks is the compute for the Big Data pipeline. In the preceding architecture description, there are a number of components used for a Big Data pipeline with Azure Databricks: Azure Storage, Azure Database, and other data sources.
Controlplane backup how to#
How to create a regional disaster recovery topology A key benefit is that both compute (Azure Databricks) and storage can be scaled independently of each other. One of the advantages of this architecture is that users can connect Azure Databricks to any storage resource in their account. All metadata, such as scheduled jobs, is stored in an Azure Database, and the database backups are automatically geo-replicated for into paired regions where it's implemented. Any management operation, such as create cluster, will be initiated from the control plane. The Databricks control plane manages and monitors the Databricks workspace environment. Once the cluster is created, you can run jobs via notebooks, REST APIs, or ODBC/JDBC endpoints, by attaching them to a specific cluster.
The persisted data is available in your storage account.
Controlplane backup driver#
Within the workspace, you create Databricks clusters by providing the worker and driver VM type and Databricks runtime version. The virtual network provides perimeter level security to the Databricks workspace, and is protected via network security group. This appliance is deployed in an Azure Virtual Network with a Network Security Group and an Azure Storage account, available in your subscription. When you create an Azure Databricks workspace from the Azure portal, a managed application is deployed as an Azure resource in your subscription, in the chosen Azure region (for example, West US). This article describes a disaster recovery architecture useful for Azure Databricks clusters, and the steps to accomplish that design.
0 kommentar(er)
