Monitoring
Create data monitors for visibility into the health of your data stack
Monitoring is an essential player in maintaining data quality. Within Secoda, you can configure essential monitors to be alerted about changes to your data. Automatically schedule and create thresholds with the ability to track the history of runs and visualize performance of your monitors.
If you've chosen Automatic thresholds, it can take up to a week for Secoda to finish learning what the right thresholds should be for your monitors.
Choose from the following monitor types:
Row Count - The number of rows over time
Freshness - The time elapsed since last update
Cardinality - The number of distinct values of a given column
Maximum - The highest value of a numeric column
Minimum - The lowest value of a numeric column
Mean - The arithmetic mean of a numeric column
Null Percentage - The percentage of values in a column that are null
Unique Percentage - The percentage of values in a column that are unique
Custom SQL - Define a monitor by writing your own SQL query
The monitor will alert if any of these values are higher or lower than expected.
Note: Read permissions for the source data (in addition to the metadata) are required for the monitoring feature.
Admins and Editors can find existing Monitors in the Monitors page from the side panel. You can see all the monitors and incidents across the entire platform, and also create new ones from here.
Creating Monitors
There are 2 ways to create monitors in Secoda. You can either use the Monitors section in the sidebar navigation or through the Monitors tab on the resource page.
Select the "Monitors" section from the sidebar navigation and click "Add monitor" (also found within the monitors tab under the resource):
Select the monitor type you want to create and choose the integration you want to create the monitor for (if adding a new monitor from the resource itself, the integration will be pre-selected):
Select one or multiple resources that you'd like to add the monitor to:
Adjust the Threshold and Schedule to your preferred configuration
Schedule: Choose between Daily, Every 12, 6 or 3 hours, or Hourly
Threshold: Automatic or Manual
Note: For Automatic thresholds to be set, it can take 4 days for hourly, 6 days for multiple times a day, and 8-9 days for daily monitors.
Once configured, click add monitor and it show now show up within the list of monitors. You can view and edit the configurations from the sidebar on the monitor page
Note: You can only add a monitor type which each of the columns support.
For example, if you have 3 numeric columns selected, you can add a "MIN" or "MAX" monitor, but you cannot do it if even one string column is selected in the modal.
Custom SQL Monitors
A user is be able to create a monitor that runs custom SQL to create an output. The only requirement is that the final output of the custom SQL must be a single value.
Follow the same steps as above, but choose "Custom SQL" as the Monitor type. After creating, click into it so that you can add your desired query in the right side panel.
WHERE clause
Standard monitors such as nullness, row count, etc can be modified with custom SQL that’s added as a WHERE clause within the standard SQL.
Managing Monitors
View Status, Last and Next Run details, and a Chart Visualization of the monitor's historical performance
Thresholds and Incidents
The lighter green surrounding the main line represent the threshold limits - once the threshold is passed, it'll show a red dot indicating an incident
Scroll down to see Run history, and filter for triggered Incident Reports. Here you can see any Downstream Resources that may be impacted by the incident.
You can either Acknowledge or Resolve the incident by click these buttons below. The incident will be automatically resolved if the numbers go back into a good state (within the threshold).
Errors
You may receive an error on your Monitors for various reasons. The Error will appear under Status. You are able to click into the error to see exactly what went wrong with the Monitor. In the example below, a Custom SQL monitor was chosen but a query was never provided, causing it to error out.
Best Practices
With monitors, as we are querying your source, we do recommend being selective around what you're monitoring and how often your monitoring based on your team's cost constraints. Some best practices are to:
Be selective about which columns/tables are monitored - focus on critical data
Reduce the number of times a monitor is run to daily
Monitoring Notifications
Ensure that you receive notifications about your monitors by going into your Notification settings. Check off where you'd like to be notified, whether that's in Slack DMs, by email, and/or in the app.
Slack Channel for Monitoring notifications
In the Slack integration Channels settings, Admins are able to set the monitoring notifications to go to a specific channel. This can be the same or different channel than what you've set up for other notifications. Learn more here.
Video resource
Last updated
