Create and use data products with Astro Observe

Create observability tools on Astro to gain insight into the health and performance of your data pipelines and lifecycle.

In Observe, a data product is a composition of assets that deliver a result with business relevance. Observe automatically infers upstream dependencies for the assets that define a data product, like an Airflow dag that populates a dashboard or a Snowflake table powering a recommendation engine.

For guidance on identifying data products in your organization, see how to identify data products in your organization.

The Data Products landing page provides an overview of all your data products and their SLA statuses. You can see at a glance which data products are late or stale, who owns them, and when they were last updated to see a snapshot of data product health.

The following procedures describe how to create data products and leverage them to gain insight across pipelines and deployments. After you create a data product, then you can view data product details.

You can follow a comprehensive walkthrough of setting up a Data Product and testing an alert in the

Get started with Observe

quickstart.

Prerequisites

See Observe prerequisites

Create a data product

1. Start data product creation
2. From the Asset Catalog

   From the Asset Catalog

   • In the sidebar, go to Observe > Asset Catalog.
   • Click + Create Data Product.

   From the Data Products page

   From the Asset Catalog

   • In the sidebar, go to Observe > Data Products.
   • Click + Data Product.
3. Select assets
5. Define data product details
7. Optionally configure a monitor
8. See Create a data product monitor for information on monitoring data products.

After completing these steps, your new data product will be available to view and manage under Data Products.

View your data product details

After you create a data product, you can view in-depth details about the performance of your assets by selecting your data product for a closer look.

1. Click Observe in the Astro UI, and then click Data Products.

2. Choose the data product that you want to view details about.

When you see information about a specific data product, you can see summary performance data. This includes general statistics, like the average SLA success rate and information about when the data product was created and last updated. Additional details can be found in the following tabs.

Overview

The overview tab provides summary information about the assets in your data product and the rates of overall SLA success rates, consolidated into daily, weekly, and monthly overall rates. This view allows you to quickly identify trends in historical data for your business-critical pipelines.

Event timeline

Each Data Product in Observe has an Event Timeline that reflects activity from the assets, like dag and task completion, and key issues, like breached SLAs and triggered alerts, in that data product. The event timeline supports filtering by event status along with more fine-grained filters. Selecting any particular event displays metadata about the event, including the notification history of a triggered alert.

The event timeline view allows you to see a record of events associated with your data product. These are categorized into Success, Neutral, and Failure events.

Success events

• SLA Success
• Task Success
• Task Start

Neutral events

• Airflow dataset write
• OpenLineage dataset write

Failure events

• Alert notification
• SLA Breach
• Task failure

For more information about diagnosing failures, see Root cause analysis and AI log summaries.

Assets

The Assets tab shows the assets included in the specific data product.

If you select an asset from the list to examine in-depth, you can see additional asset-specific details. See Assets for descriptions.

Metrics

Data products that include Airflow task assets report the following metrics by default:

• Task retries
• Task failures
• Task duration

Alerts

The Alerts tab consolidates all alerts and SLAs that are specific to your selected data product. From this page you can Add an alert, find an existing alert, and view the notification history for an alert.