Secoda Docs
Get Started
  • Getting Started with Secoda
    • Secoda as an Admin
      • Deployment options
      • Sign in options
      • Settings
      • Connect your data
        • Define Service Accounts
        • Choose which schemas to extract
      • Customize the workspace
      • Populate Questions with FAQs
      • Invite your teammates
        • Joining & Navigating between Multiple Workspaces
      • Onboard new users
        • Onboarding email templates
        • Onboarding Homepage template
        • Training session guide
      • User engagement and adoption
        • Tips & Tricks to share with new users
    • Secoda as an Editor
    • Secoda as a Viewer
      • Introduction guide
      • Requesting changes in Secoda
  • Best practices
    • Setting up your workspace
    • Integrating Secoda into existing workflows
    • Documentation best practices
    • Glossary best practices
    • Data governance
    • Data quality
    • Clean up your data
    • Tool migrations using Secoda
    • Slack <> Questions workflow
    • Defining resources workflow
    • Streamline data access: Private and public teams workflow
    • Exposing Secoda to external clients
  • Resource Management
    • Editing Properties
      • AI Description Editor
      • Bulk Editing
      • Propagation
      • Templates
    • Resource Sidesheet
    • Assigning Owners
    • Custom Properties
    • Tags
      • Custom Tags
      • PII Identifier
      • Verified Identifier
    • Import and Export Resources
    • Related Resources
  • User Management
    • Roles
    • Teams
    • Groups
  • Integrations
    • Integration Settings
    • Data Warehouses
      • BigQuery
        • BigQuery Metadata Extracted
      • Databricks
        • Databricks Metadata Extracted
      • Redshift
        • Redshift Metadata Extracted
      • Snowflake
        • Snowflake Metadata Extracted
        • Snowflake Costs
        • Snowflake Native App
      • Apache Hive
        • Apache Hive Metadata Extracted
      • Azure Synapse
        • Azure Synapse Metadata Extracted
      • MotherDuck
        • MotherDuck Metadata Extracted
      • ClickHouse
        • ClickHouse Metadata Extracted
    • Databases
      • Druid
        • Druid Metadata Extracted
      • MySQL
        • MySQL Metadata Extracted
      • Microsoft SQL Server
        • Page
        • Microsoft SQL Server Metadata Extracted
      • Oracle
        • Oracle Metadata Extracted
      • Salesforce
        • Salesforce Metadata Extracted
      • Postgres
        • Postgres Metadata Extracted
      • MongoDB
        • MongoDB Metadata Extracted
      • Azure Cosmos DB
        • Azure Cosmos DB Metadata Extracted
      • SingleStore
        • SingleStore Metadata Extracted
      • DynamoDB
        • DynamoDB Metadata Extracted
    • Data Visualization Tools
      • Amplitude
        • Amplitude Metadata Extracted
      • Looker
        • Looker Metadata Extracted
      • Looker Studio
        • Looker Studio Metadata Extracted
      • Metabase
        • Metabase Metadata Extracted
      • Mixpanel
        • Mixpanel Metadata Extracted
      • Mode
        • Mode Metadata Extracted
      • Power BI
        • Power BI Metadata Extracted
      • QuickSight
        • QuickSight Metadata Extracted
      • Retool
        • Retool Metadata Extracted
      • Redash
        • Redash Metadata Extracted
      • Sigma
        • Sigma Metadata Extracted
      • Tableau
        • Tableau Metadata Extracted
      • ThoughtSpot
        • ThoughtSpot Metadata Extracted
      • Cluvio
        • Cluvio Metadata Extracted
      • Hashboard
        • Hashboard Metadata Extracted
      • Lightdash
        • Lightdash Metadata Extracted
      • Preset
        • Preset Metadata Extracted
      • Superset
        • Superset Metadata Extracted
      • SQL Server Reporting Services
        • SQL Server Reporting Services Metadata Extracted
      • Hex
        • Hex Metadata Extracted
      • Omni
        • Omni Metadata Extracted
    • Data Pipeline Tools
      • Census
        • Census Metadata Extracted
      • Stitch
        • Stitch Metadata Extracted
      • Airflow
        • Airflow Metadata Extracted
      • Dagster
        • Dagster Metadata Extracted
      • Fivetran
        • Fivetran Metadata Extracted
      • Glue
        • Glue Metadata Extracted
      • Hightouch
        • Hightouch Metadata Extracted
      • Apache Kafka
        • Apache Kafka Metadata Extracted
      • Confluent Cloud
        • Confluent Cloud Metadata Extracted
      • Polytomic
        • Polytomic Metadata Extracted
      • Matillion
        • Matillion Metadata Extracted
      • Airbyte
        • Airbyte Extracted Metadata
      • Informatica
        • Informatica Metadata Extracted
      • Azure Data Factory
        • Azure Data Factory Metadata Extracted
    • Data Transformation Tools
      • dbt
        • dbt Cloud
          • dbt Cloud Metadata Extracted
        • dbt Core
          • dbt Core Metadata Extracted
      • Coalesce
        • Coalesce Metadata Extracted
    • Data Quality Tools
      • Cyera
      • Dataplex
        • Dataplex Metadata Extracted
      • Great Expectations
        • Great Expectations Metadata Extracted
      • Monte Carlo
        • Monte Carlo Metadata Extracted
      • Soda
        • Soda Metadata Extracted
    • Data Lakes
      • Google Cloud Storage
        • GCS Metadata Extracted
      • AWS S3
        • S3 Metadata Extracted
    • Query Engines
      • Trino
        • Trino Metadata Extracted
    • Custom Integrations
      • File Upload
        • CSV File Format
        • JSONL File Format
        • Maintain your Resources
      • Marketplace
        • Secoda SDK
        • Upload and Connect your Marketplace Integration
        • Publish the Integration
        • Example Integrations
      • Secoda Fields Explained
    • Security
      • Connecting via Reverse SSH Tunnel
      • Connecting via SSH Tunnel
      • Connecting via VPC Peering
      • Connecting via AWS Cross Account Role
      • Connecting via AWS PrivateLink
        • Snowflake via AWS PrivateLink
        • AWS Service via AWS PrivateLink
      • Recommendations to Improve SSH Tunnel Concurrency on SSH Bastion
    • Push Metadata to Source
  • Extensions
    • Chrome
    • Confluence
      • Confluence Metadata Extracted
      • Confluence best practices
    • Git
    • GitHub
    • Jira
      • Jira Metadata Extracted
    • Linear
    • Microsoft Teams
    • PagerDuty
    • Slack
      • Slack user guide
  • Features
    • Access Requests
    • Activity Log
    • Analytics
    • Announcements
    • Audit Log
    • Automations
      • Automations Use Cases
    • Archive
    • Bookmarks
    • Catalog
    • Collections
    • Column Profiling
    • Data Previews
    • Data Quality Score
    • Documents
      • Comments
      • Embeddings
    • Filters
    • Glossary
    • Homepage
    • Inbox
    • Lineage
      • Manual Lineage
    • Metrics
    • Monitors
      • Monitoring Use Cases
    • Notifications
    • Policies
    • Popularity
    • Publishing
    • Queries
      • Query Blocks
        • Chart Blocks
      • Extracted Queries
    • Questions
    • Search
    • Secoda AI
      • Secoda AI User Guide
      • Secoda AI Use Cases
      • Secoda AI Security FAQs
      • Secoda MCP Server
    • Sharing
    • Views
  • Enterprise
    • SAML
      • Okta SAML
      • OneLogin SAML
      • Microsoft Azure AD SAML
      • Google SAML
      • SCIM
      • SAML Attributes
    • Self-Hosted
      • Additional Resources
        • Additional Environment Variables
          • PowerBI OAuth Application (on-premise)
          • Google OAuth Application (on-premise)
          • Github Application (on-premise)
          • OpenAI API Key Creation (on-premise)
          • AWS Bucket with Access Keys (on-premise)
        • TLS/SSL (Docker compose)
        • Automatic Updates (Docker compose)
        • Backups (Docker compose)
        • Outbound Connections
      • Self-Hosted Changelog
    • SIEM
      • Google Chronicle
  • API
    • Get Started
    • Authentication
    • Example Workflows
    • API Reference
      • Getting Started
      • Helpful Information
      • Audit Logs
      • Charts
      • Collections
      • Columns
      • Custom Properties
      • Dashboards
      • Databases
      • Documents
      • Events
      • Groups
      • Integrations
      • Lineage
      • Monitors
      • Resources
      • Schemas
      • Tables
      • Tags
      • Teams
      • Users
      • Questions
      • Queries
      • Getting Started
      • Helpful Information
      • Audit Logs
      • Charts
      • Collections
      • Columns
      • Custom Properties
      • Dashboards
      • Databases
      • Documents
      • Events
      • Groups
      • Integrations
      • Lineage
      • Monitors
      • Resources
      • Schemas
      • Tables
      • Tags
      • Teams
      • Users
      • Questions
      • Queries
  • FAQ
  • Policies
    • Terms of Use
    • Secoda AI Terms
    • Master Subscription Agreement
    • Privacy Policy
    • Security Policy
    • Accessibility Statement
    • Data Processing Agreement
    • Subprocessors
    • Service Level Agreement
    • Bug Bounty Program
  • System Status
  • Changelog
Powered by GitBook
On this page
  • Example: Step-by-Step Integration Development
  • Prerequisites

Was this helpful?

  1. Integrations
  2. Custom Integrations

Marketplace

Learn how to build a Marketplace integration using the Secoda SDK.

Last updated 9 months ago

Was this helpful?

To create a Marketplace integration, you'll need to use the Secoda Software Development Kit (SDK). The SDK is a Python-based toolkit that allows for the creation of custom Marketplace integrations. These integrations can be used to add resources and lineage to Secoda.

Check out our webinar on the Secoda SDK !

To start building with the Secoda SDK, there are a few steps to follow. First, you need to write your Python code. You can use the documentation below or scroll down for a !

Once the code is written, you can upload and connect your integration.

And share it!

Example: Step-by-Step Integration Development

In this guide, we'll build an example integration with a data visualization service called . By the end, you'll be able to write scripts that can be executed locally and ready to be submitted to Secoda.

Prerequisites

Before you begin, ensure you have:

  • Python (3.11 or later recommended) installed on your system.

  • Secoda SDK installed via pip install --upgrade secodadk.

Step 1: Set Up Your Development Environment

Create a new directory for your integration project. Inside this directory, create a virtual environment and activate it:

python -m venv venv
source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

With your virtual environment activated, install the Secoda SDK (and if exists, upgrade to the latest version)

pip install --upgrade secodadk

Step 2: Initialize Your Integration Script

Create a new Python file, fake_cluvio_integration.py, which will contain your integration code.

Step 3: Import SDK Components and define your Integration Class

At the top of your script, import the necessary components from the Secoda SDK:

from secodadk import SecodaIntegration, Resource, DeclaredLineage, InternalResource, ExternalTable

Extend the SecodaIntegration class to create your own integration:

class FakeCluvioIntegration(SecodaIntegration):
    def extract(self):
        # Your extraction logic goes here.

Step 4: Implement the Extraction Logic

The extract method is where the integration interacts with Cluvio's API. We'll authenticate using credentials and then extract data:

# Within your extract method
token_response = self.http_post(
    f"https://api.cluvio.com/users/sign_in",
    json={
        "user": {
            "email": self.credentials.get("email"),
            "password": self.credentials.get("password"),
        }
    },
    headers={"Accept": "application/json"},
)
token = token_response.json().get("token")

# Authentication requests
auth = dict(
    headers={"token": token},
    verify=True,
)


# Logic for fetching and declaring resources and lineage
...

Note that:

  • self.credentials is a dictionary provided by the SecodaIntegration base class, allowing you to access user inputs in the integration connect form.

  • We use self.http_post for HTTP requests. You must use our self.http_{method} for HTTP requests. Other library like requests or httpx will not work. self.http_{method} returns a httpx.Response instance.

With the data obtained from Cluvio, you can now declare resources (e.g., dashboards) via self.declare_resource

# Continue your extract method
group_databuilder_id = "dashboard_group.default"
self.declare_resource(
    Resource(
        entity_type="dashboard_group",
        databuilder_id=group_databuilder_id,
        title="default",
    )
)

dashboards = self.http_get(f"https://api.cluvio.com/dashboards/", **auth).json().get("data", [])
for dashboard in dashboards:
    dashboard_databuilder_id = f"dashboard.{dashboard.get('id')}"
    self.declare_resource(
        Resource(
            entity_type="dashboard",
            databuilder_id=dashboard_databuilder_id,
            title=dashboard.get("attributes", {}).get("name"),
            description=dashboard.get("attributes", {}).get("description"),
            parent_databuilder_id=group_databuilder_id,
            external_updated_at=dashboard.get("attributes", {}).get(
                "updated_at"
            ),
            product="cluvio",
            native_type="dashboard",
        )
    )

Lineage between resources is also crucial for understanding the relationship and flow of data. This is declared via the self.declare_lineage method. Lineage can be between resources from the same source.

self.declare_lineage(
    DeclaredLineage(
        from_identifier=InternalResource(
            databuilder_id=dashboard_databuilder_id
        ),
        to_identifier=InternalResource(databuilder_id=group_databuilder_id),
    )
)

Lineage can also flow from an existing resource in the workspace, to a resource brought in from the custom integration.

self.declare_lineage(
    DeclaredLineage(
        from_identifier=ExternalTable(
            schema="public",
            table="orders",
        ),
        to_identifier=InternalResource(databuilder_id=dashboard_databuilder_id)
    )
)

Step 5: Testing Locally

To ensure that your integration works as expected, test it on your local machine. Set the credentials in your environment for security:

if __name__ == "__main__":
    FakeCluvioIntegration(
        credentials={
            "email": os.environ["EMAIL"],
            "password": os.environ["PASSWORD"],
        }
    ).start()

Run your script with the environment variables set to your Cluvio credentials:

EMAIL=your-email@example.com
PASSWORD=yourpassword
python fake_cluvio_integration.py

Step 6: Finalize Your Integration for Upload

See for more information on the fields of Resource .

Once you've tested your script and are happy with the results, it's time to .

For the full code of this integration, see .

Secoda Fields Explained
Upload and Connect your Custom Integration
here
here
Secoda SDK
Upload and Connect your Marketplace Integration
Publish the Integration
Cluvio
step by step example