Monitoring
Create data monitors for visibility into the health of your data stack
Introduction
Monitoring plays a crucial role in maintaining data quality by allowing you to configure alerts for changes in your data. Automatically schedule monitors and set thresholds to track run history and visualize monitor performance.
Admins and Editors can access existing Monitors from the Monitors page accessible via the side panel. Here, you can view all monitors and incidents across the platform and create new ones.
To learn about how our current customers are using Monitors in Secoda to improve their data quality, check out this list of Monitoring Use Cases.
Note: Read permissions for the source data (in addition to the metadata) are required for the monitoring feature.
Types of Monitors
Select from a variety of Monitors to suit your needs:
Row Count - The number of rows over time
Freshness - The time elapsed since last update
Cardinality - The number of distinct values of a given column
Maximum - The highest value of a numeric column
Minimum - The lowest value of a numeric column
Mean - The arithmetic mean of a numeric column
Null Percentage - The percentage of values in a column that are null
Unique Percentage - The percentage of values in a column that are unique
Custom SQL - Define a monitor by writing your own SQL query
The monitor will alert if any of these values are higher or lower than expected.
Creating Monitors
Monitors can be created via the Monitors section in the sidebar or through the Monitors tab on the resource page:
Navigate to "Monitors" and click "Add monitor."
Choose the monitor type and select the integration.
(if adding a new Monitor from the resource itself, the integration will be pre-selected):
Select one or multiple resources that you'd like to add the monitor to.
Adjust the Threshold and Schedule to your preferred configuration.
Schedule Options: Daily, Every 12, 6 or 3 hours, or Hourly
Threshold: Automatic or Manual
Note: For Automatic thresholds to be set, it can take 4 days for hourly, 6 days for multiple times a day, and 8-9 days for daily monitors.
Once configured, click add monitor and it show now show up within the list of monitors. You can view and edit the configurations from the sidebar on the monitor page
Note: You can only add a monitor type which each of the columns support.
For example, if you have 3 numeric columns selected, you can add a "MIN" or "MAX" monitor, but you cannot do it if even one string column is selected in the modal.
Custom SQL Monitors
A user is be able to create a monitor that runs custom SQL to create an output. The only requirement is that the final output of the custom SQL must be a single value.
Follow the same steps as above, but choose "Custom SQL" as the Monitor type. After creating, click into it so that you can add your desired query in the right side panel.
WHERE clause
Standard monitors such as nullness, row count, etc can be modified with custom SQL that’s added as a WHERE clause within the standard SQL.
Managing Monitors
View Status, Last and Next Run details, and a Chart Visualization of the monitor's historical performance.
Thresholds and Incidents
If you've chosen Automatic thresholds, it can take up to a week for Secoda to finish learning what the right thresholds should be for your monitors.
The lighter green surrounding the main line represent the threshold limits - once the threshold is passed, it'll show a red dot indicating an incident
Scroll down to see Run history, and filter for triggered Incident Reports. Here you can see any Downstream Resources that may be impacted by the incident.
You can either Acknowledge or Resolve the incident by click these buttons below. The incident will be automatically resolved if the numbers go back into a good state (within the threshold).
Errors
You may receive an error on your Monitors for various reasons. The Error will appear under Status. You are able to click into the error to see exactly what went wrong with the Monitor. In the example below, a Custom SQL monitor was chosen but a query was never provided, causing it to error out.
Best Practices
To optimize the effectiveness of data monitoring and manage resource utilization, consider these best practices:
Selective Monitoring: Focus on the most critical data elements. Prioritize columns and tables that are essential for your business operations to avoid unnecessary strain on resources.
Optimize Frequency: Set monitoring frequencies that balance timeliness and resource consumption. For many applications, configuring monitors to run daily is sufficient to catch issues without incurring excessive costs.
Regular Reviews: Periodically review data quality monitoring configurations. This ensures that your monitoring strategies stay aligned with evolving business needs and data landscapes.
Workflow Integration: Embed monitoring alerts into your team’s daily workflows using tools like Slack or email (see Monitoring Notifications). This ensures that the right personnel are promptly notified, enabling swift action.
Documentation and Training: Keep detailed documentation of your monitor setups and procedures. Train your team on the importance of monitoring and the actions required when specific alerts are triggered.
Trend Analysis: Leverage historical data from your monitoring activities to identify trends and patterns. This analysis can help refine your data management practices and predictive monitoring over time.
By following these guidelines, you can ensure your monitoring processes are both efficient and effective, providing critical insights while maintaining control over costs and resource use.
Monitoring Permissions
Monitoring functionality is primarily intended for users with Edit or Admin roles, rather than end business users who typically have Viewer permissions.
Creating:
Any Admin or Editor can now create monitors, without requiring specific integration permissions.
Editing and Owning:
Initially, any Admin or Editor can edit monitors. However, once an owner is designated for a monitor, only the owner and Admins will have editing rights.
Owners and Admins will also have the ability to invite additional owners to the monitors.
Deleting, Running, and Resolving:
The permissions for deleting, running, and resolving monitors will mirror those for editing. This means that only the monitor's owner and Admins can perform these actions once an owner is assigned.
Viewing:
Viewing permissions remain unchanged, with any Admin or Editor able to view monitors. We are planning future updates that will allow more granular control over who can view monitor outputs, enhancing privacy and data security.
Monitoring Notifications
Stay informed about the status of your monitors by adjusting your Notification settings. Specify your preferred channels for receiving alerts—whether through Slack DMs, email, or directly within the app.
Notifications for monitor incidents are issued only after the first occurrence following a successful run. The same incident will not trigger new alerts unless the issue has been resolved and another incident occurs. This policy minimizes repetitive alerts and ensures that notifications remain meaningful and actionable.
Configuring Slack Channel Notifications
Admins can direct monitoring notifications to specific Slack channels, distinct from other notification settings. This ensures that the right team members are alerted promptly. For detailed steps on setting this up, visit here.
Email Monitoring Notifications
Email notifications provide direct links to the relevant sections in Secoda. As shown in the image below, clicking the "Open Secoda" link takes you to the Inbox notification, while other links direct you to specific incidents or tables.
Video resource
Last updated
