Securely Manage Secrets in Azure Databricks Using Databricks-Backed

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. Designed with the founders of Apache Spark, Databricks is integrated with Azure to provide one-click setup, streamlined workflows, and an interactive workspace that enables collaboration between data scientists, data engineers, and business analysts."

Azure Databricks is enabling companies to accelerate big data analytics and artificial intelligence (AI) solutions. Above all it is providing a fast, easy and collaborative Apache Spark–based analytics service. Now companies can leverage a variety of data sources to drive actionable business insights.

Integrating with a variety of data sources also means management of credentials for connecting to different data sources.

To manage credentials Azure Databricks offers Secret Management. Secret Management allows users to share credentials in a secure mechanism. Currently Azure Databricks offers two types of Secret Scopes:

• Azure Key Vault-backed: To reference secrets stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault.
• Databricks-backed: A Databricks-backed scope is stored in (backed by) an Azure Databricks database. You create a Databricks-backed secret scope using the Databricks CLI (version 0.7.1 and above).

In this tip we will learn about creating Databricks-backed secret scopes. Azure Key Vault-backed secrets are in Preview. Above all, Azure Key Vault-backed currently are only supported via the Azure Databricks UI and not through the Databricks CLI.

To learn more about Databricks-backed please check out my article here.