Creating dependency aware dev pipelines in Monorepos

Edit (2024-06) - As of Turborepo v2.0.4 you can now use turbo watch [tasks] to re-run turbo tasks when their dependencies change. It intelligently ignores persistent tasks so they do not block execution. This makes it easier to create dependency-aware dev pipelines in Turborepo.

Running multiple dev tasks in parallel common when working with a monorepo. It's also a source of frustration, as monorepo tools don't support depending on persistent tasks.

This is problem for packages that bundle workspace dependencies into a single file. If the dev tasks run in parallel, packages might use a previous build of a dependency.

Unsurprisingly, lots of folks ask for some version of this feature across some of the most popular monorepo tools.

Turborepo #1497 | Support executing multiple dependent long-running tasks in parallel

Nx #5570 | Support for dependency-aware "watch" tasks on nx run-many

The simplest solution would be to tell the dev task to wait for the dependency to build before starting the dev server. Then, on change, rebuild the dependencies and restart the server.

Below I'll cover how to do exactly that, and make your long-running dev tasks "dependency aware". I'll also suggest a different way of thinking about dev pipelines which can help you avoid this issue in the first place.

Adding a watch script that rebuilds dependencies on change

A common solution to this problem is to use a watch script to re-run the build pipelines when a dependency changes. You can see an example of this on TanStack/query's package.json. The watch script will re-run the build pipeline anytime a workspace package changes.

// query/package.json
{
  "scripts": {
    "build": "nx affected --target=build --exclude=examples/**",
    "build:all": "nx run-many --target=build --exclude=examples/**",
    "watch": "pnpm run build:all && nx watch --all -- pnpm run build:all",
    // `dev` is really just running `build` for all packages on change
    "dev": "pnpm run watch"
  }
}

You could do something similar with Turborepo by configuring nodemon to run turbo build whenever it detects a change.

{
  "scripts": {
    "dev": "nodemon --config nodemon.config.json -x 'pnpm build:packages'",
    "build": "turbo run build --cache-dir=.turbo",
    "build:packages": "turbo run build --filter=!docs"
  }
}

But this only works if all the dev tasks can run in parallel. If your dev task depends on another packages build outputs, you'll need to wait for that task to finish before starting your dev server.

This is problem for packages that bundle workspace dependencies into a single file. If the dev tasks run in parallel, the package will be using the previous build of the dependency. This can lead to unexpected and annoying behavior.

Forcing pipelines to wait for persistent task outputs

Many monorepo tools allow you to configure certain task to wait for other tasks to finish before running. This is typically done using a dependsOn configuration.

{
  "tasks": {
    //...
    "dev": {
      "cache": false,
      "persistent": true,
      "dependsOn": ["^dev"] // this doesn't work with persistent tasks :(
    }
  }
}

However, this won't work. Tools like Turborepo will throw an error saying that the base package @acme/base cannot be depended on.

$ turbo run dev
error: Invalid persistent configuration:
"@acme/base#dev" is a persistent task, "@acme/app#dev" cannot depend on it

Turborepo doesn't allow you to depend on a persistent task. The dependsOn configuration will wait for all tasks to finish before running the next task, so you can't have @acme/app's dev depend on @acme/base' package dev task because it never ends.

Using a wait-on script to run a dependency's setup task first

To get around this issue, you can create a task that waits for @acme/base to finish building before exiting; or a "setup" task.

We can use the wait-on package to wait for the build files to be present before starting the dev server. You can then set all dev pipelines to dependOn the wait-on script.

// turbo.json
{
  "pipeline": {
    // ...
    "dev": {
      "dependsOn": ["setup-dev"],
      "cache": false
    },
    "setup-dev": {
      "cache": false
    }
  }
}
// packages/@acme/base/package.json
{
  "scripts": {
    "dev": "vite",
    "build": "vite build",
    "setup-dev": "wait-on dist/@acme/base.js"
  }
}

The task will exit once the build files are present. This allows @acme/app to depend on @acme/base's setup-dev task, ensuring that @acme/app is using the latest build of @acme/base.

Now, when resolving task dependencies, Turborepo will run @acme/base's setup-dev and wait for it to emit an exit code before running the @acme/app:dev task.

Don't treat setup tasks as dev tasks

After a very helpful conversation with Anthonly Shew on Twitter, I realized one of the reasons a lot of folks, like myself, run into this problem is because we're thinking about our dev pipelines the wrong way.

Just because a build artifact "must" exist for a dev task to run it doesn't mean that it's part of the dev task itself.

Breaking the eggs, chopping the veggies, and mixing the ingredients are all part of the dev task. Turning on the stove is a setup task.

It's corny, but it helps me remember to avoid linking tasks that don't really belond together.