Choose a code repository strategy

Choose a code repository strategy

Astro supports a range of options when it comes to organizing your project code.

This guide covers the options along with their pros and cons, so you can choose the best option for your team.

Feature overview

Astro supports two basic repository strategies. You can:

1. Use a monorepo: host an Astro project in one repository for deploys to one or multiple Astro Deployments, based, for example, on permanent dev and prod branches with their own Deployments or a single permanent branch. Astronomer recommends this approach for most teams.
2. Use a multirepo: separate your Dags from other files in your project and maintain multiple repositories linked to one or multiple Astro Deployments. Astronomer recommends this approach for teams needing to meet strict security requirements and teams planning to automate the creation of Deployments.

Best practice guidance

Depending on your organization’s structure and needs, Astronomer recommends either a monorepo or multirepo approach to organizing the code you deploy to Astro. In addition to a repository strategy, Astronomer recommends implementing version control and CI/CD.

Option 1: Monorepo

This is the most common strategy.

When your Dags are critical to your business, the ability to test your Dags is critical, as well. Using a monorepo, you can set up automated deploys from multiple branches of a single repository to support multiple Deployments of the same codebase, enabling implementation of robust testing of the code you intend to deploy to production, support for multiple teams, and more.

Within a monorepo, you can also maintain multiple Astro projects, all with Astro Deployments and testing.

This strategy requires:

• One Astro Workspace for your project(s).
• One or more permanent branches in your repository, each representing an environment. Many teams use two permanent branches named main and dev.
• One or more Astro Deployments, each representing an environment.

Pros of this approach:

• You can test code changes to your Dags in an isolated environment on Astro before deploying the changes to production.
• It is flexible and scalable. Mapping branches to additional Deployments allows for permanent testing environments or ephemeral sandbox Deployments for feature testing. See Manage dev Deployments on Astro. Also, if you work at a larger organization, you can deploy to different Workspaces and Astro projects to support multiple teams or organize Deployments according to specific business use cases.
• It supports separate environment configurations. For example, your development Deployment can query a development database, and your production Deployment can query a production database without needing complex logic to switch between the two.

Cons of this approach:

• Dependencies and environments might be more difficult to manage, to the extent that they differ from project to project.

For guidance on setting up CI/CD for deploys to multiple Deployments from a single repository, see

Multiple environments

.

Individuals and small teams getting started on Astro can keep their instances simple with one permanent branch mapped to an Astro Deployment.

Option 2: Multirepo

A common use case for multiple repositories is keeping Astro Project configuration (such as the Dockerfile, requirements.txt, or Deployment Configuration-as-Code) separate from Dag code. Astronomer recommends a multirepo approach when:

• You have strict security requirements for who can manage Deployments.
• You want to minimize complexity for project contributors, automate the creation of Deployments, or manage Deployments more efficiently.
• You can set up and maintain a more complex CI/CD pipeline.

This strategy requires:

• A Workspace for your project(s).
• Multiple repositories.
• One or more Astro Deployments.
• Dag-only deploys on the target Deployment and CI/CD pipeline setup for each repository.

Pros of this approach:

• Keeping project configuration separate from Dag code enables the use of a single repository for configuring multiple Deployments.
• The possibility of accidental changes to Astro Deployment settings is minimized.
• You can automate the creation of Deployments and streamline your Deployment management process.

Cons of this approach:

• You must keep any local copies of an Astro project synchronized with multiple repositories in order to test Deployment code locally.
• Team members might have inconsistencies in their local environments if they lack access to the configuration repository. For this reason, Astronomer recommends setting up a Dev Deployment where Dag authors can see and modify project configuration for testing purposes.

Version control and CI/CD

In addition to choosing a repository strategy suited to your use case, Astronomer recommends implementing version control and CI/CD.

Implementing version control is a longstanding software development best practice that enables:

• Simultaneous collaboration.
• Easy review of code changes.
• Safe and secure experimentation on new features and fixes.
• Improved traceability and auditing.

Using CI/CD is a general software development best practice that enables automated building, testing, merging, and delivery of code. If you haven’t yet implemented CI/CD pipelines for your Deployments, benefits of doing so include:

• Faster feedback loops.
• Improved code quality.
• Streamlined development workflows.
• Consistent deployments across environments.
• Less time spent getting code to production.
• Enhanced collaboration.

With a version control system in place, you can set up CI/CD by following the guidance in Develop a CI/CD workflow for deploying code to Astro. The option you choose for organizing the code in your shared repository and deploying to Astro should reflect the size and needs of your team. Deployment API tokens are required for implementing CI/CD for automated deploys to Astro.

Once you have set up CI/CD, Astronomer recommends enabling CI/CD enforcement so that code pushes can be completed only when using a Deployment or Workspace token.