BBC 6 Music → Spotify Sync: Learning Guide

1. The Big Picture

The 30-second version

One long-running Node.js process. Inside it, three cron jobs:

Job When (London time) What it does
Monthly creator 00:00 on the 1st Creates "BBC 6 Music " on Spotify
Weekly updater 00:00 every Tuesday Scrapes the BBC, matches songs, adds new ones
Daily checker 01:00 every day Notices missed runs and re-runs them

Under the jobs sit five services, each with one responsibility:

flowchart TD
    subgraph Process["One Node process (PM2: bbc-6music-sync)"]
        S[Scheduler<br/>node-cron + a job queue]
        S --> M[MonthlyPlaylistCreator]
        S --> W[WeeklyPlaylistUpdater]
        S --> D[DailyStatusChecker]
        D -.re-runs.-> M
        D -.re-runs.-> W
        W -.creates month first if needed.-> M

        W --> B[BBCScraper<br/>axios + cheerio]
        W --> SM[SongMatcher<br/>scoring + verification]
        SM --> SP[SpotifyService<br/>spotify-web-api-node]
        W --> SP
        M --> SP
        W --> ST[StateManager<br/>JSON files]
        M --> ST
        W --> N[NotificationService<br/>nodemailer]
        M --> N
    end
    B -->|HTTPS| BBC[(bbc.co.uk playlist page)]
    SP -->|HTTPS| SPOT[(Spotify Web API)]
    ST --> FS[(data/*.json)]
    N -->|SMTP| MAIL[(Plesk mailbox)]

Everything is wired together by hand in src/index.ts: new each service, pass it into the jobs that need it, start the scheduler. There's no dependency-injection framework, and it doesn't need one. That style is called constructor injection, and it's why every class can be unit tested with fakes.

One Tuesday, end to end

Here's what happens at 00:00 on a Tuesday, in order (weeklyPlaylistUpdater.ts):

  1. Is there a playlist for this month? If processing-state.json still says last month, create this month's playlist first. (This exists because of a real race. See #11.)
  2. Scrape the BBC page. You get back about 30–40 {artist, title} pairs.
  3. Find out what's already in the playlist. Read the current playlist live from Spotify, plus our own records for this month and last month. This yields three sets: URIs we've seen, song identities we've seen (artist|title), and tombstones (songs you deleted by hand, which must never come back).
  4. For each BBC song:
    • tombstoned → skip
    • identity already seen (this month or last) → skip as a duplicate
    • otherwise → ask the matcher. On success, queue the URI. On ambiguity, add it to the review list.
  5. Add the queued URIs to the Spotify playlist, in batches of 100.
  6. Write the records back: month metadata (now mirroring Spotify), the week's song batch file, and the processing state.
  7. Email a review list if anything needs a human.
  8. Clean up song batch files older than 28 days.

All the thinking happens in steps 3 and 4. Everything else is plumbing.

Shape of the design, and why

Choice Instead of Because
One process with in-process cron System cron starting a fresh process per job Shared state and auth in one place. One thing to deploy and watch
JSON files SQLite or Postgres A few kilobytes, one writer, written weekly. Files are readable with cat and get backed up with the folder
Spotify playlist as the source of truth Our JSON as the source of truth You edit playlists by hand. Learned the hard way (#10)
Flag for review when unsure Always pick something A wrong song in your playlist is worse than a missing one (#13)
Email on failure Dashboards or monitoring One user. Email reaches you, and a dashboard nobody looks at doesn't
Services behind small classes One big script Each part can be tested alone. The original Python version was one script (History)

The ADRs call this "appropriate tools for the scale". The point is that boring is a feature: every dependency is something that can break at 00:00 on a Tuesday while you're asleep.

Where things live

src/
  index.ts                 wiring, startup, graceful shutdown
  config/config.ts         load + validate settings.json
  schedulers/              the three jobs + the Scheduler that runs them
  services/                BBC, Spotify, matcher, state, notifications
  utils/                   dates (Luxon), logger (Winston), title/artist cleaners
  tools/sendTestEmail.ts   one-off "does SMTP actually work?" command
scripts/                   manual harnesses (test-matcher, inspect-playlist, …)
scripts/deploy/release.sh  † runs on the server: install, health check, roll back
ecosystem.config.js        † PM2's definition of the process
                           († = on the feat/21 branch until it merges)
.github/workflows/         ci.yml and deploy.yml
tests/unit, tests/integration, tests/deploy, tests/fixtures
data/                      state (gitignored): state/, playlists/, songs/
config/settings.json       secrets (gitignored, chmod 600)

Questions you might get asked

"Walk me through the architecture." It's a single Node/TypeScript daemon run by PM2 on a VPS. Inside it, node-cron triggers three jobs on London time. The jobs are thin orchestrators over five services: scraping, Spotify, matching, state and notifications, and every service is constructor-injected so it can be tested on its own. State is JSON on disk. Spotify is the real record of what's in a playlist, and we read it back every run.

"Why not serverless, like a Lambda on a schedule?" It would work, and it would be a reasonable choice. I already had a VPS, and the job needs persistent state on disk and a long-lived refresh token. A serverless version would need a bucket or a table for state and a secrets store, which means more moving parts for a job that runs once a week. If I didn't have the VPS, I'd seriously consider it.

"Why does one weekly job need a daily job?" Because the weekly job can fail: the BBC is down, Spotify is down, or the server was rebooting at midnight. The daily check is a cheap safety net. It looks at state, sees "this week's run didn't complete", and runs it again. It also runs once at startup, so a deploy at 00:00 doesn't cost a week.

"What's the riskiest part?" The matcher. Everything else either works or fails loudly. The matcher can fail quietly, by confidently picking the wrong track. Most of the bug history is about that.