Great Expectations

An overview of the Great Expectations integration with Secoda

Great Expectations Metadata Extracted

Secoda currently supports retrieving expectation and validation metadata from Google Cloud Storage and AWS S3.

Getting Started with Great Expectations and Google Cloud Storage

There are two steps to get started using Great Expectations with Secoda through Google Cloud Storage:

1. Create a service account for Secoda

2. Connect Google Cloud Storage to Secoda

Create a service account for Secoda

To provide least privilege to Secoda for extracting Big Query metadata, you can create a new service account following the steps below. Refer to Google Cloud’s documentation about service accounts for more information.

1. From the Navigation panel on the left, go to IAM & admin > Service accounts

2. Click Create Service Account along the top

3. Enter a name (for example: “secoda”) and click Create

4. When assigning permissions, make sure to grant the following roles:

Storage Object Viewer

5. Create a JSON key. The downloaded file will be used to create your warehouse in the next section.

Connecting to Secoda

Log into your Secoda profile at https://app.secoda.co

1. From the Navigation panel on the left go Integrations > Add new integration

2. Select Great Expectations and fill in the fields based off of the Great Expectations configuration YAML file.

3. Make sure to upload the Great Expectations configuration YAML file.

4. You will be asked to map each datasource outlined in your configuration file to an existing integration in Secoda.

Getting Started with Great Expectations and AWS S3

There are two steps to get started using Great Expectations with Secoda through AWS S3

1. Create a new AWS IAM user

2. Connect AWS S3 to Secoda

Create a AWS IAM user for Secoda

You can create an IAM user using one of three methods:

• Creating IAM users (console)
• Creating IAM users (AWS CLI)
• Creating IAM users (AWS API)

Note: Ensure that Access Key - Programmatic Access is checked.

Once you create the user you can save the Access Key ID and Secret Access Key generated for the user.

Attach the following policy to the newly created user. Make sure to change <your-bucket-name>

{
    "Statement": [
        {
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:DeleteObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<your-bucket-name>",
                "arn:aws:s3:::<your-bucket-name>/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Connecting to Secoda

1. Log into your Secoda profile at https://app.secoda.co
2. From the Navigation panel on the left go Integrations > Add new integration

3. Select Great Expectations and fill in the fields based off of the Great Expectations configuration YAML file.

4. Make sure to upload the Great Expectations configuration YAML file.

5. You will be asked to map each datasource outlined in your configuration file to an existing integration in Secoda.