Databricks

An overview of the Databricks integration with Secoda

Databricks Metadata Extracted

Getting Started with Databricks

There are three steps to get started using Databricks with Secoda:

1. Create an access token

2. Connect Databricks to Secoda

3. Whitelist Secoda IP Address

Note: You can run Databricks on Secoda using AWS or GCP.

Create an access token

In your Databricks console go to the User Settings and generate a new access token. Save the value to be used to connect Databricks to Secoda in the second step.

To have query history and popularity you must provide admin privileges to the token.

Connect Databricks to Secoda

Go to https://app.secoda.co/integrations/new and select the Databricks integration.

Enter in the following credentials:

• Host: This is the URL of your Databricks workspace, i.e, dbc-dc31b5a2-597d.cloud.databricks.com
• Databricks Workspace Id: The numerical id of your workspace, located in the url of your Databricks instance, after the "/?o=". https://<instance_id>.cloud.databricks.com/?o=<workspace_id>.\

• Access Token: The access token you generated in the first step

• Warehouse ID (Recommended) or Cluster ID: This is the resource what SQL queries will run on. For the optimal experience, use a Databricks serverless SQL warehouse.

To ingest table and column level lineage using Databricks Unity Catalog, a Warehouse ID must be specified.

After entering in the information into Secoda, click "Test Connection". After the connection is successful your can Submit and run the initial extraction.

Whitelist Secoda IP Address

If your Databricks instance is behind a firewall, you'll have to whitelist Secoda's IP address to allow for metadata extractions.