#7. CI/CD: From Merge to Running Release

Status, 27 October 2026: CI is live on main. Continuous deployment (#21) is committed on feat/21-continuous-deployment and not merged yet. Files marked † below exist on that branch. The operator's runbook is docs/guides/deployment.md †.

#The idea in one picture

flowchart LR
    PR[Pull request] --> CI1[CI: verify]
    CI1 -->|green| MERGE[Merge to main]
    MERGE --> CI2[CI on main]
    CI2 -->|green| DEP[Deploy workflow]
    DEP --> BUILD[Build that exact commit<br/>tar it]
    BUILD -->|scp| SRV[VPS: incoming/]
    SRV -->|ssh: release.sh deploy| ACT{Started and<br/>stayed up?}
    ACT -->|yes| LIVE[current → new release]
    ACT -->|no| RB[Roll back to previous<br/>workflow fails → email]

CI (continuous integration) answers "is this change safe?" CD (continuous deployment) answers "then ship it, with no human steps".

#CI: .github/workflows/ci.yml

It runs on every PR to main and every push to main. One job, verify, on Ubuntu with Node from .nvmrc (24):

Details worth explaining:

• npm ci, not npm install. ci installs exactly what package-lock.json says and fails if it disagrees with package.json. It's reproducible, which is what you want on a build server.
• Concurrency: a newer push to a PR cancels the older CI run on it. Runs on main are never cancelled, because each one might lead to a deploy.
• permissions: contents: read: the workflow's token can read the repo and nothing else. That's least privilege.
• Lint had to be fixed first. When CI was planned, npm run lint failed on main with 51 errors (#18). A gate that is always red is no gate, so #18 blocked #20. Some of those errors were real risks: a no-misused-promises error on cron.schedule or process.on can hide an unhandled rejection that crashes the daemon.

#"Why isn't main protected?"

On GitHub Free, a private repo can't have branch protection. So "only merge green PRs" is a convention here, not something GitHub enforces. The deploy design makes up for it: deploys are triggered by CI passing on main, so even a direct push that fails CI never ships.

#CD: .github/workflows/deploy.yml †

#Trigger

on:
  workflow_run:            # when the CI workflow…
    workflows: [CI]
    types: [completed]     # …finishes on main
    branches: [main]
  workflow_dispatch:       # or by hand, with an optional dry run

The deploy job runs only if all of these hold:

• the repository variable DEPLOY_ENABLED == 'true', which is an off switch you can flip without changing code
• CI's conclusion was success
• the CI run came from a push, not a PR

It checks out github.event.workflow_run.head_sha, the commit CI tested, not whatever main points to now. Otherwise, a fast second merge could ship an untested commit.

concurrency: deploy-production with cancel-in-progress: false means deploys queue up instead of interrupting each other. Interrupting a deploy halfway is worse than waiting.

How the plan changed: the issue originally had the deploy workflow call CI as a reusable workflow (workflow_call), so it would run every check again. The final design chains off CI's result instead (workflow_run), so the checks run once and the deploy reacts to them. A manual run hasn't been through CI, so it runs lint and tests itself.

#Build once, ship the artefact

GitHub builds dist/ and packs a tarball: dist package.json package-lock.json ecosystem.config.js scripts/deploy. The server never runs the TypeScript compiler or sees the source. Its only job is npm ci --omit=dev (runtime dependencies only). Build once, deploy the artefact is the principle: what you tested is what you run.

#Getting onto the server

• A dedicated ed25519 deploy key. Its private half is a GitHub secret, and its public half is in the subscription user's authorized_keys. The key can't do anything that user can't do, and that user isn't root.
• DEPLOY_KNOWN_HOSTS: the server's host key, pinned. Without it, SSH either prompts (and hangs in CI) or blindly trusts whatever answers, which opens the door to a man-in-the-middle.
• BatchMode=yes makes SSH fail instead of waiting for a password.
• scp uploads the tarball. Then ssh … "bash -s -- deploy <id>" < scripts/deploy/release.sh streams the new release script into bash on the server. This runs the new version of the script, not whatever the previous release left on disk.
• The dry run (workflow_dispatch with dry_run) only logs in and prints whoami, the Node version and the folders. It's a cheap way to prove connectivity, and it replaced a throwaway test branch.

#On the server: release.sh †

This is the heart of CD. Its two commands are deploy <id> and rollback [<id>]. In order, deploy:

1. Validates the release id against a strict regex. It ends up in paths and rm -rf, so that's not optional.
2. Preflight: refuses (exit 2) if shared/config/settings.json is missing or shared/data/ is empty. A fresh or wiped server must never start an app with no state, because that app would create duplicate playlists.
3. Unpacks into releases/<id>/, runs npm ci --omit=dev, and replaces config, data and logs with symlinks into shared/.
4. Atomic switch: ln -sfn … current.tmp && mv -Tf current.tmp current. A rename is atomic on Linux, so current is never missing or half-written.
5. pm2 delete the old app, which gives it up to 6 minutes to finish a job, then pm2 start ecosystem.config.js from the new release.
6. Health check: wait up to 90 seconds for a new Application startup complete line, by recording the log's byte size before the start and only reading what was written after it. Then check that the PID is unchanged 10 seconds later. A process that logs "started" and then crashes doesn't pass.
7. Pass → pm2 save (so a reboot brings this release back), delete the tarball, prune to the newest 5 releases. Fail → print the last 50 log lines, switch back, start the previous release, and exit 1. GitHub then emails you about the failed workflow.

Why look for a log line instead of an HTTP health endpoint? The app has no HTTP server. Adding one just for deploys would mean an open port and more code. "It authenticated with Spotify, started the scheduler and logged startup complete" proves more than "the port is open" would.

How the plan changed: the issue planned two scripts (remote-deploy.sh and rollback.sh) and a health check based on pm2 jlist restart counts with a 60-second timeout. The build merged them into one script that shares activate() between deploy and rollback, so rollback gets the same health check. It also switched to the fresh log line plus a stable PID, which is easier to test with stubs, and raised the timeout to 90 seconds.

#Questions you might get asked

"Describe your deployment pipeline." PRs run lint, type-check, tests, build and shell checks. A merge to main runs the same checks, and if they pass, a deploy workflow builds that exact commit into a tarball and copies it over SSH with a pinned host key and a deploy-only key. A release script installs it into a timestamped folder, switches a current symlink atomically, restarts it under PM2, and waits for proof that it started. If the new release doesn't come up healthy, it rolls back automatically and the workflow fails.

"Why deploy on every merge? Isn't that risky for production?" For one maintainer with good tests, a release branch adds ceremony without adding safety. The safety comes from CI as a gate, the deploy checking out the exact commit CI tested, a real health check, automatic rollback, and an off switch.

"How do you roll back?" Automatically, if the new release fails its health check. By hand, I run release.sh rollback, which uses the same health check, against any of the last five releases still on disk. A rollback is just a symlink switch and a restart, because state lives in shared/ and never moves.

"How are secrets handled?" App secrets (Spotify, SMTP) exist only on the server in a chmod 600 file and never touch GitHub. GitHub holds only what it needs to reach the server: host, port, user, a deploy key and the pinned host key.

"What's zero-downtime here?" There isn't any, and that's deliberate. Running the old and new processes at the same time would mean two writers to the same state and the same playlists. A gap of a few seconds in a cron daemon costs nothing, and startup catch-up covers a job that should have fired during that gap.