Monitoring
Create data monitors for visibility into the health of your data stack
Monitoring is an essential player in maintaining data quality. Within Secoda, you can configure essential monitors to be alerted about changes to your data. Automatically schedule and create thresholds with the ability to track the history of runs and visualize performance of your monitors.
Choose from the following monitor types:
- Nullness - The percentage of values in a column that are null
- Cardinality - The number of distinct values of a given column
- Uniqueness - The percentage of values in a column that are unique
- Freshness - The time elapsed since last update
- Row Count - The number of rows over time
- Maximum - The highest value of a numeric column
- Mean - The arithmetic mean of a numeric column
- Minimum - The lowest value of a numeric column
The monitor will alert if any of these values are higher or lower than expected.
Note: Read permissions for the source data (in addition to the metadata) are required for the monitoring feature.
Admins and Editors can find existing Monitors in the Monitors page from the side panel. You can see all the monitors and incidents across the entire platform, and also create new ones from here.
- 1.Navigate to the table or column that you'd like to add a monitor for (or search it from the main Monitors page)
- 2.Click into the "Monitors" tab > "New Monitor"
- 3.Select one or multiple resources that you'd like to add the Monitor for.
Select resource - 4.Select which monitor type(s) (can be more than one) and click "Add Monitors".
Monitor types - 5.Once created, you can find it under the "Monitors" tab in the Catalog
- 6.Click into the monitor to further edit and see the history of the monitor's runs
- Edit Configuration details like Schedule, Threshold and Sensitivity
- Schedule - Choose between Daily, Every 12, 6 or 3 hours, or Hourly
- Threshold - Automatic or Manual
- Sensitivity - More or less sensitive threshold
Configuration details
Note: You can only add a monitor type which each of the columns support.
For example, if you have 3 numeric columns selected, you can add a "MIN" or "MAX" monitor, but you cannot do it if even one string column is selected in the modal.
A user is be able to create a monitor that runs custom SQL to create an output. The only requirement is that the final output of the custom SQL must be a single value.
Follow the same steps as above, but choose "Custom SQL" as the Monitor type. After creating, click into it so that you can add your desired query in the right side panel.
Custom SQL monitor
Standard monitors such as nullness, row count, etc can be modified with custom SQL that’s added as a WHERE clause within the standard SQL.
Adding WHERE clause
View Status, Last and Next Run details, and a Chart Visualization of the monitor's historical performance
The lighter green surrounding the main line represent the threshold limits - once the threshold is passed, it'll show a red dot indicating an incident
Scroll down to see Run history, and filter for triggered Incident Reports. Here you can see any Downstream Resources that may be impacted by the incident.
You can either Acknowledge or Resolve the incident by click these buttons below. The incident will be automatically resolved if the numbers go back into a good state (within the threshold).
Ensure that you receive notifications about your monitors by going into your Notification settings. Check off where you'd like to be notified, whether that's in Slack DMs, by email, and/or in the app.
Monitor notifications
In the Slack integration Channels settings, Admins are able to set the monitoring notifications to go to a specific channel. This can be the same or different channel than what you've set up for other notifications. Learn more here.
.png?alt=media&token=c912fb46-aea0-4e2d-b2af-e9d850303eec)
Last modified 21h ago