Ad Astro, Astronomer ‘Launches’ Data Orchestration Observability Cloud

Data moves, constantly. While there are a percentage of enterprise information resources that remain comparatively static, the vast majority of data inside any organization is always on the move, channelling between applications, data repositories, through services, across network connections and down application programming interfaces.

Because there is such a high degree of inherent movement within the data universe, an entire sub-sector of the IT industry has developed to look after the analysis, management, monitoring and orchestration of data itself. Some technology vendors specialize in just one of those data actions, but most offer an amalgam of more than one function to provide a more platform-level service.

Combining both orchestration and observability into its core technology proposition, Astronomer, the company behind the Astro data orchestration platform, has now created Astro Observe.

Astrological Automation

Because modern digital enterprises now find themselves consuming a variety of different “data products” (a term we have defined before here), there is an unquestioned need to try and get a single view (tech vendors love to call it a single source of truth) across the entire data estate. That’s what Astro Observe seeks to provide.

By adding observability to the existing data orchestration capabilities of Astro, Astronomer says that enterprises now have every element they need to achieve full-stack data orchestration on one platform.

According to Astronomer, reliance on data has evolved over the past decade from powering basic internal dashboards to data becoming a product itself, with source data packaged alongside “artefacts” (a term used to describe any explanatory classification information that clarifies, validates and explains what data itself relates to, so logically here this includes metadata, transformation logic, documentation, test information and access policies) for maximum information management potential.

In terms of use, Astro Observe “exposes” all that information about information as an application programming interface (simply known as an API in developer circles) and aims to help organizations more closely align their data conduits to business objectives.

“Whether it’s retail recommendations with dynamic pricing, automated customer support, financial trading strategy, regulatory reporting, or AI—full stack data orchestration has become the linchpin of modern software solutions” suggests Astronomer, in a product statement. “Now that data products have become business-critical, any failure can have an outsized impact on revenue and reputation. By unifying orchestration and observability across the full data stack on a single platform, the reliability and trust of data products are improved, development velocity is increased, costs are lowered, and critical data assets are better secured.”

Data’s Connective Tissues

The company insists that the addition of observability to Astro is a pivotal step in providing a single platform for modern data orchestration. Central and fundamental to Astronomer’s observability capabilities is the use of OpenLineage, a technology framework that works to make data pipelines observable by automatically collecting and correlating detailed information about their behaviour and data movement.

Inside Astro Observe users get a service level agreement dashboard so that data product owners (usually developers working in data science roles) can be assigned. This step is necessary to provide a clear picture of accountability and ownership. Owners of data products can set policies and monitor thresholds for data freshness and delivery times to better align with business outcomes.

“The Astro Observe lineage graph provides a view of upstream and downstream dependencies across data products, offering essential context for issue remediation and compliance,” notes the company. “Further here, a recommendation engine makes use of best practices from Airflow [Apache Airflow is an open-source tool to programmatically author, schedule and monitor workflows] alongside a holistic view of deployments to surface actionable recommendations for optimizing trust and reliability of data products.”

Data As A Working Asset

It is at this point in the data management market that we start to hear vendors talking about data as a working asset inside an organization. It is, if you will, a formalization of what the so-called digitally transformed business is supposed to look like. With Astronomer providing functions such as predictive alerting, teams are able to pinpoint and understand risk before the delivery of data products are impacted… almost as if data itself has become an employee or at least a part of the physical office assets that a company operates with.

“Astronomer provides a single destination for teams to orchestrate and manage data pipelines along with the visibility they need to monitor performance and make adjustments as needed to solve problems or make proactive optimizations; minimizing context switching, and allowing for one interface to understand and take action,” details the company, in its technology literature.

As modern data stacks become more complex with dependencies that sprawl across multiple platforms, teams and deployments, Astronomer’s centralized view aims to help teams assess and make decisions about the management of their projects. This technology could take off, pun intended.