Implementing data quality monitoring using Data Metric Functions in Snowflake

Data is now abundant in virtually every organisation. Nevertheless, many data-driven initiatives fail not because of a lack of data, but because of a lack of confidence in its quality. Uncertainty over how up-to-date the data is, unexpected NULL values or inconsistent data records lead to analyses being called into question and decisions being delayed. This is precisely where Snowflake comes in with its Data Metric Functions (DMFs), offering a native way to make data quality measurable and monitorable directly within the platform.

DMFs: The Basics

DMFs are functions that return a specific metric, such as the number of NULL values in a column, the number of unique values, or the freshness of a table. The system DMFs provided by Snowflake in the SNOWFLAKE.CORE schema cover many standard requirements and enable a quick start without additional development effort. For more complex or domain-specific use cases, custom DMFs can be created and flexibly stored in the desired database and schema.

As soon as a DMF is assigned to a table or view, Snowflake takes over the automatic execution. The metrics are calculated at a defined time interval, for example several times a day or tailored to the respective data loading cycle. All results are stored centrally in a dedicated event table, creating a consistent and traceable history of data quality. This approach not only makes data quality visible but also allows it to be compared over time and across different data objects.

DMFs can be applied to a wide range of objects, including tables, views, materialised views, dynamic tables, external tables and Apache Iceberg tables. This makes the approach suitable for both traditional analytics workloads and modern data architectures. At the same time, there are deliberately set limits – for example, on the maximum number of DMF mappings or on usage in shared or trial accounts – which should be taken into account during planning.

Added value and potential applications of DMFs

DMFs enable the continuous measurement of quality metrics such as completeness, uniqueness and timeliness, rather than checking these only sporadically or manually. This allows error-correction measures to be automated and ensures high-quality data on an ongoing basis, which in turn strengthens all stakeholders’ confidence in this data. To this end, Snowflake provides both predefined, system-native DMFs and the option to define custom metrics that are precisely tailored to business or technical requirements.

Data Quality and DMFs are available as a feature of the Snowflake Enterprise Edition. One advantage of the Snowflake approach lies in its technical implementation. DMFs utilise serverless compute resources that are fully managed by Snowflake. There is no need to provision or manage additional virtual warehouses, and billing is based solely on the scheduled execution of the metrics. Creating DMFs is free of charge, and ad hoc calls – for example, as part of analyses – do not incur any costs either. Usage is transparently reported under the ‘Data Quality Monitoring’ section and can be tracked in detail.

Practical application of DMFs

An example implementation scenario in an e-commerce business illustrates what such a data quality workflow looks like in practice.

Specifically, the following quality rules, amongst others, are implemented in the example:

• Uniqueness: CUSTOMER_IDs must not occur more than once
• Timeliness: Customer data must not be more than 10 days old
• Business plausibility: Order totals must not be negative

First, a central schedule for Data Metric Functions is set up, which defines the frequency at which the quality checks are carried out. On this basis, the system’s own DMFs – for example, those used to identify NULL values, check uniqueness or measure data freshness – are specifically linked to the customer data table. This ensures that customers can be uniquely identified by their CUSTOMER_ID and that, for example, their data records are no older than 10 days. In addition, a user-defined DMF is used to map a business-specific quality requirement. This describes the check for negative values in the order total (TOTAL_AMOUNT) of an order table. This ensures, for example, that no undesirable discount combinations result in negative order totals and that the company does not mistakenly pay out money.

A crucial aspect here is that a DMF initially merely measures, but does not yet make any judgement as to whether a value is acceptable. To turn a measurement into a genuine quality check, so-called expectations are used. These are used to define which range of values is considered ‘good’. The result of the DMF is automatically compared with this expectation and assessed as ‘pass’ or ‘fail’. In this way, formal and reproducible data quality rules can be established without having to build logic outside Snowflake.

The example shown illustrates how DMFs are recorded in a structured manner, executed automatically, and their results made available centrally. This takes place entirely within Snowflake and without any additional infrastructure.

From measurement to active monitoring

When combined with alerts, data quality ultimately becomes an active monitoring mechanism. In the event of deviations or threshold breaches, automated notifications can be triggered, enabling problems to be identified and resolved at an early stage. Various roles within the organisation benefit equally from this: data stewards gain transparency into the current state of their data, data engineers can respond quickly and in a targeted manner, and platform administrators can keep an eye on costs, performance and consistency. Data quality thus evolves from a reactive task into an integral part of data governance.

Conclusion

Overall, Data Metric Functions demonstrate how data quality can be directly embedded within the data platform. Instead of using external tools, additional infrastructure or manual verification processes, quality is measured right where the data is generated and processed. Companies thereby create a reliable foundation for compliance, stable SLAs and, above all, trust in their data. Anyone wishing to make data-driven decisions in the long term will find it hard to do without a structured, automated data quality approach such as the DMFs in Snowflake.