Astronomer gives boost to Apache Airflow platform data orchestration for AI

Getting data to and from different systems is often the domain of data orchestration. It is among the most widely used tools in the open-source Apache Airflow technology, originally created by Airbnb.

Today Astronomer, the guide commercial sponsor behind the Apache Airflow project is out with its latest Astro platform update, providing enterprise uphold, security and management enhancements. While Airflow got its start helping with the orchestration of data pipelines for data analytics and business intelligence, the technology is now increasingly being used in uphold of artificial intelligence (AI) and machine learning (ML) workloads.

“Airflow is very good at a couple of things, one is just basically writing and running data pipelines,” Julian LaNeve, CTO at Astronomer told VentureBeat. “Airflow lets you define pipelines as code, so you can do anything that the code will let you do which is essentially boundless.” 

Why Airflow is critical for modern data orchestration

LaNeve explained that Airflow has become popular in recent years as it enables organizations to define, build and deploy data pipelines more easily.

Airflow integrates with major data platforms and cloud provider systems including Snowflake, Databricks, AWS, Microsoft and Google Cloud. LaNeve noted that while the open-source project is fairly straightforward to use for a single team, it becomes increasingly complex to manage at an enterprise scale. That’s where Astronomer comes into play, offering a managed service for Apache Airflow.

Astronomer also offers additional capabilities on top of the core open-source technology.

“We’ve built a layer on top of Airflow that we call the Astronomer runtime, where we’ve taken the open source project and made a couple of modifications to make it run more efficiently,” LaNeve said. 

Astronomer has also developed tools as part of its Astro platform that make it easier to write data pipelines. The company’s Astro Cloud IDE provides a notebook-based tool for writing data pipelines. LaNeve added that Astronomer has also started doing some work in the observability space understanding how data moves across the entire ecosystem.

Astronomer gets better connections and upgrades

With its new Astro platform update, Astronomer is now bringing a series of enhanced capabilities to its platform.

One of the most fundamental and difficult things with any data pipeline is getting connected to the data in a safeguard manner. With the Astro update, there is a new connection management update to help overcome that challenge.

The connection management feature acts as a central point of governance, visibility and security for the data pipelines.

“We’ve built a connection management feature into the Astro platform that lets an administrator come in and define connections to Snowflake, Databricks and anywhere that Airflow can access,” LaNeve said.

The Astro platform update also enables easier upgrades and rollbacks for data pipeline configurations. When and if a data pipeline fails, a user can now easily fall back to a previous configuration for production workloads. For upgrades, the platform will also execute a series of checks to first ensure that the code in the update is compatible and will run as expected.

Astronomer doubles down on AI

Astronomer is increasingly being used for AI workflows.

At the end of November, Astronomer announced a series of integrations across the AI landscape with vendors including OpenAI, Cohere, Pinecone, OpenSearch, Weaviate and pgvector. 

Astronomer has also built a reference architecture for how organizations can build and deploy large language model (LLM) applications. The ask.astronomer.io application is a public demonstration of the architecture and can pull in documentation across over a dozen different sources in a retrieval augmented generation (RAG) approach.

LaNeve said he also sees Airflow and the Astro platform being widely used in uphold of training AI models.

“Ultimately you want your models to be trained with the latest data and you want that to happen reliably,” LaNeve said. “That’s  exactly what Astronomer and Airflow was built for.”