This page will describe how to create a JSONL file for upload into Secoda.
Create a newline-delimited JSON (JSONL) file that contains a JSON object for each resource or lineage entry that is meant to be imported into Secoda.
Resources and Lineage
Secoda expects metadata files to follow a newline-delimited JSON (JSONL) format. Two types of files can be uploaded into Secoda into this way:
resources.jsonl: Contains metadata for various resources, such as tables, columns, dashboards, and charts.
lineage.jsonl: Contains lineage information that describes relationships within Resources in the resouces.jsonl file (this is referred to as Internal Lineage) and relationships between Resources in the resources.jsonl file with Resources that already exist in Secoda (this is referred to as External Lineage).
Learn more about the available fields for the JSON objects here.
These JSON objects have several required and optional fields. See an outline of the Secoda fields here.
Example - Generating Metadata Files
There are several ways to generate a JSONL file. If you plan on using Python, we recommending defining the Secoda resources as Pydantic models. The code below shows how Secoda has defined the Resource and Lineage models using Pydantic. All the fields as part of the models are explained in the documentation above.
class Resource(pydantic.BaseModel):
entity_type: SecodaEntityTypes
databuilder_id: str
title: Optional[str] = ""
description: Optional[str] = None
definition: Optional[str] = None
external_updated_at: Optional[datetime] = None
native_type: Optional[str] = None
# parent_databuilder_id is an optional string depending on the resource that is being created.
# Please see notes above to determine when it must be included for your resource.
parent_databuilder_id: Optional[str] = None
# Column specific
sort_order: Optional[int] = None
type: Optional[str] = None
is_pk: Optional[bool] = False
# Table specific, required for table
schema: Optional[str] = None
database: Optional[str] = None
# Dashboard specific
group: Optional[str] = None
# Chart specific
product: Optional[str] = None