Skip to main content

Choosing the right storage for your workflow

Understand your storage options in Tines 3B.

Written by Jamie Gaynor

Tines 3B gives your steps real filesystem storage through named volumes. A volume is a directory mounted at /storage/<name> that a step can open, read, write, rename, delete, and list, just like a local disk. What makes volumes flexible is that you configure them along three independent dimensions: how long the data lives, whether a step can write to it, and whether writes can happen at the same time. This article walks through each option and what it's best used for.

How a volume works

Before the options, one idea that shapes all of them: a step writes into its own private view while it runs. Other steps always see the last committed version, never half-written files. When a step succeeds, its changed files publish. If it fails, nothing publishes. That is what keeps your stored data consistent even when steps error out or run concurrently.

You declare a volume in a step's Dockerfile, for example:

VOLUME ["state"]

Reach for a volume when your code needs filesystem behavior: many related files, random access, downloaded artifacts, extracted archives, SQLite databases, or third-party tools that expect file paths. For small handoffs between two adjacent steps, such as a JSON payload or a bit of generated text, use stdin and stdout instead of a volume.

Choose the lifetime

The first choice is how long the data should stick around.

  • Durable (the default). Files survive and stay visible to later Live workflow runs in the same space. This is the right choice for uploads, reports, dashboards, caches, cursors, logs people need later, and any generated artifacts that future runs should reuse.

  • Run-scoped. Files exist only for the single workflow run that created them, then they're gone. Declare it with scope=run. This is best for downloads, extracted archives, intermediate files, temporary SQLite databases, and scratch space handed between steps in one run.

VOLUME ["state"]              # durable
VOLUME ["work:scope=run"] # run-scoped

A good way to decide is to ask a product question rather than a technical one: should this data still be there for later runs, or only for this run? Also note that every step sharing the same run-scoped store must declare scope=run; a plain VOLUME ["work:ro"] is the read-only view of the durable volume, not the run-scoped one.

Choose the access

The second choice is whether a step can change the files.

  • Read-only. Add :ro to the volume name. Use it for any step that only inspects files: dashboards, report APIs, export steps, validators, and summarizers.

  • Writable. Grant a writable mount only to the smallest step that actually publishes changes.

VOLUME ["state:ro"]           # read-only reader
VOLUME ["state"] # writable

Keeping readers read-only makes a workflow easier to reason about, and it avoids a common mistake: giving a dashboard or report step a writable mount when all it ever does is read.

Choose the write concurrency

The third choice matters when workflow runs can overlap, because it decides whether two writable steps can publish at the same time.

  • Shared (the default). Concurrent writers can all publish to the same volume. If they touch different paths, every path survives. If two writers touch the same file, the release fails with a conflict rather than silently letting the last write win. Shared mode is best when each writer owns a distinct path or directory, for example one folder per delivery, job, customer, or worker.

  • Exclusive. The volume stays locked for the whole step, so writers take turns. Declare it with concurrency=exclusive. This is best for logical read-modify-write state: counters, cursors, ledgers, sessions, a single append-only file, a package cache, or one database or JSON file that many runs update.

VOLUME ["incoming"]                      # shared writers, each owning a path
VOLUME ["state:concurrency=exclusive"] # one writer at a time

Two cautions worth keeping in mind:

  • Lifetime and concurrency are separate. A durable volume can use shared mode when writers create disjoint paths, and a run-scoped volume can still need exclusive mode if parallel steps update the same file group.

  • Exclusive locks cover the whole step. A model call, an API request, or a download inside an exclusive step makes every other writer wait for that slow work to finish. When work is slow but its results feed shared state, let the workers run concurrently and hand each finished result to a small exclusive step that only updates the shared files.

Common shapes at a glance

These patterns cover most workflows:

  • Independent work. Give each item, job, customer, or worker its own directory in a shared writable volume, then batch or import from it later with read-only readers.

  • One shared record. Put a counter, index, database path, or summary behind a single exclusive writer, with every reader mounting it :ro.

  • Per-run scratch space. Use a run-scoped volume for downloads, extracted archives, and intermediate files that later Live runs don't need.

  • Best-effort cache. Write independent, content-addressed files in a shared volume where an occasional miss or overwrite is acceptable. If cache correctness matters, treat it like a shared record and use one writer.

A note on structured data

Volumes are the built-in storage for files and filesystem state, and SQLite works well on a durable or run-scoped volume for structured data local to a workflow. When you need to read from or write to an existing external database, connect to it directly with a database connector (Postgres, MySQL, Microsoft SQL Server, or MongoDB) rather than storing that data in a volume.

Did this answer your question?