The 3B v3 upgrade includes a Blobstore upgrade. New installations start with empty Blobstore storage automatically. Existing installations must choose whether to retain their Blobstore data or start with empty storage.
freshskips copying the old data. Existing Blobstore data becomes unavailable, while new execution and filesystem data works normally.retainmigrates existing execution results, workflow histories, and filesystem and volume data during an offline maintenance window.
Do not change this choice after 3B v3 accepts new writes. Contact Tines before migrating a production installation.
Before either upgrade, take your normal PostgreSQL backup and snapshot the Blobstore persistent volume. The retain migration preserves the existing data and creates the upgraded copy beside it, so confirm that the volume has enough free space for both copies.
Single-server installation
To retain existing Blobstore data, run the installed setup script with the version and migration choice:
sudo VERSION=<V3_VERSION> BLOBSTORE_ROLLOUT=retain \ bash /opt/3b/setup.sh
The setup script saves BLOBSTORE_ROLLOUT=retain in /opt/3b/.env, stops every Blobstore caller, waits for the pods to terminate, runs the non-retrying migration Job, and starts 3B v3 after the Job succeeds. If the Job fails, the script leaves the callers stopped and prints the retained Job diagnostics.
If losing access to existing Blobstore data is acceptable, select fresh explicitly. This skips the migration:
sudo VERSION=<V3_VERSION> BLOBSTORE_ROLLOUT=fresh \ bash /opt/3b/setup.sh
The setup script uses the same one-time stop for fresh, but it starts 3B v3 without running the migration Job.
When S3-compatible storage is configured, the migration uses the existing endpoint, bucket, region, credentials, and workload identity. Without object storage, every required legacy payload must still be present on the persistent volume.
Kubernetes with Helm
A new Helm installation starts with empty Blobstore storage. An upgrade from v2 must set blobstore.rollout explicitly in the same values file that you use for every helm upgrade.
Use fresh when existing Blobstore data may be left behind:
blobstore: rollout: fresh
Use retain when existing Blobstore data must be migrated:
blobstore: rollout: retain
Both choices require a one-time maintenance window because the v2 workloads must stop before 3B v3 starts. Retaining data adds a migration operation between stopping v2 and starting v3.
1. Record and stop the Blobstore callers
Put the ingress or load balancer into maintenance mode. Suspend any HorizontalPodAutoscaler or external autoscaler that targets api, orchestrator, public, worker, or blobstore.
Record the current replicas:
export NAMESPACE=<NAMESPACE>
{
kubectl -n "$NAMESPACE" get deployment api orchestrator public worker \
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.replicas}{"\n"}{end}'
kubectl -n "$NAMESPACE" get statefulset blobstore \
-o jsonpath='{.metadata.name}{"="}{.spec.replicas}{"\n"}'
} | tee 3b-replicas-before-v3.txt
Scale the callers down and wait for every pod to stop:
kubectl -n "$NAMESPACE" scale \
deployment/api deployment/orchestrator deployment/public deployment/worker \
statefulset/blobstore --replicas=0
kubectl -n "$NAMESPACE" rollout status deployment/api --timeout=5m
kubectl -n "$NAMESPACE" rollout status deployment/orchestrator --timeout=5m
kubectl -n "$NAMESPACE" rollout status deployment/public --timeout=5m
kubectl -n "$NAMESPACE" rollout status deployment/worker --timeout=5m
kubectl -n "$NAMESPACE" rollout status statefulset/blobstore --timeout=5m
for app in api orchestrator public worker blobstore; do
mapfile -t pods < <(kubectl -n "$NAMESPACE" get pod -l "app=$app" -o name)
((${#pods[@]} == 0)) || kubectl -n "$NAMESPACE" wait --for=delete "${pods[@]}" --timeout=5m
done
Do not continue while any of those workloads has a running pod.
2. Run the migration Job for retain
Skip this step if you selected fresh. If you selected retain, use a unique attempt name. The chart keeps every Blobstore caller at zero replicas while this value is present.
export VERSION=<V3_VERSION> export VALUES_FILE=<PATH_TO_YOUR_VALUES_FILE> export ATTEMPT_ID="v2-v3-$(date -u +%Y%m%dT%H%M%SZ)" helm upgrade 3b oci://oci.tines.com/3b/charts/3b \ --version "$VERSION" \ --namespace "$NAMESPACE" \ -f "$VALUES_FILE" \ --set-string blobstore.backfillAttemptId="$ATTEMPT_ID" \ --timeout 15m
Follow the Job and save its log:
kubectl -n "$NAMESPACE" logs -f job/blobstore-backfill kubectl -n "$NAMESPACE" wait --for=condition=complete job/blobstore-backfill --timeout=3h kubectl -n "$NAMESPACE" logs job/blobstore-backfill > blobstore-backfill.log
The Job does not retry automatically. If it fails, preserve the Job, its log, and the persistent volume, then contact Tines. Do not start the application or create another attempt until Tines reviews the failure.
After Tines approves a retry, save the failed Job log, delete the failed Job, choose a new ATTEMPT_ID, and repeat step 2:
kubectl -n "$NAMESPACE" logs job/blobstore-backfill > blobstore-backfill-failed.log kubectl -n "$NAMESPACE" delete job blobstore-backfill export ATTEMPT_ID="v2-v3-$(date -u +%Y%m%dT%H%M%SZ)"
The failed Job must be deleted because Kubernetes does not allow its pod definition to be changed in place.
3. Start 3B v3
If you selected retain, apply the chart again after the Job succeeds. Keep blobstore.rollout: retain in your normal values. The command below ends the migration phase and starts 3B v3 on the retained data. If you selected fresh, run the same command once after stopping the old workloads and keep blobstore.rollout: fresh in your normal values.
helm upgrade 3b oci://oci.tines.com/3b/charts/3b \ --version "$VERSION" \ --namespace "$NAMESPACE" \ -f "$VALUES_FILE" \ --set-string blobstore.backfillAttemptId= \ --timeout 15m
Confirm that Blobstore and the application are ready before removing maintenance mode. Run one normal workflow that writes execution and filesystem data, then read its result. Later v3 upgrades do not rerun the migration or require another full maintenance window. Keep the selected blobstore.rollout value in your normal values throughout v3. The final command above clears its temporary migration input automatically. Keep the pre-upgrade volume snapshot until you complete the 3B v4 upgrade. That upgrade removes the Blobstore rollout setting and migration tooling because every installation then uses the upgraded Blobstore unconditionally.