#5. Scheduling, State and Recovery

The matcher is the clever part, and this part is what keeps the system running. Most production bugs in this project lived here.

#Cron, in-process

Cron syntax is five fields: minute hour day-of-month month day-of-week.

scheduler.ts registers all three with node-cron and { timezone: 'Europe/London' }.

Why Tuesday? The BBC updates on Monday, at no fixed time. Running at Tuesday 00:00 makes sure Monday's update is in.

Why in-process instead of system cron? System cron would start a fresh Node process for each job, so every run would load config and authenticate from scratch, and the timezone would need configuring on the OS. In-process, it's one daemon, one config, one Spotify client, and PM2 watches it. The cost is that if the process is down at 00:00, the job simply doesn't happen. That cost is why the daily checker and startup catch-up exist.

#The job queue (#19)

private queue: Promise<void> = Promise.resolve();

private enqueue(name, task) {
  this.queue = this.queue.then(run);   // run after whatever is already running
  return this.queue;
}

This is a promise chain used as a mutex. Each job waits for the previous one to finish, so two jobs never read and write the JSON files at the same time. It's three lines with no locking library. Errors are caught inside run, so one failed job never breaks the chain for the jobs after it.

#Graceful shutdown (#19)

When PM2 stops the app (every deploy), it sends SIGINT (or SIGTERM, from systemd). main() catches it and calls scheduler.stop():

1. Stop cron from triggering anything new.
2. Wait up to 5 minutes for the running job to finish.
3. Flush the logs and exit.

PM2's kill_timeout is 6 minutes, longer than 5 on purpose. If it were shorter, PM2 would kill the process halfway through a weekly run's Spotify writes. A unit test checks this relationship (ecosystem.test.ts), which is why DEFAULT_SHUTDOWN_TIMEOUT_MS is now exported.

#Startup catch-up (#19)

On every start, runStartupCheck() runs the daily checker immediately. Since every deploy is a restart, a deploy at 00:01 on a Tuesday would otherwise lose that week until 01:00. If startup fails (bad config, Spotify auth), the app emails once before exiting. PM2's backoff settings keep a crash loop from turning into an email storm.

#Time zones: the GMT/BST trap

Every date the system acts on is computed in Europe/London with Luxon (dateHelper.ts). Timestamps are stored in UTC.

Things worth being able to explain:

• Why not UTC everywhere? "Midnight on the 1st" is a local idea. In BST, London midnight is 23:00 UTC the previous day. A UTC cron would create October's playlist on 30 September.
• Locale is pinned (en-GB for display, en-US for playlist names). Luxon otherwise uses the machine's default locale, which differed between the dev PC, CI and the server (#30).
• The clock-change hour. UK clocks change at 01:00 GMT in March (01:00 jumps to 02:00) and at 02:00 BST in October (01:00–01:59 happens twice). The daily check is scheduled at 01:00, right inside that window. In March, 01:00 doesn't exist on that Sunday, and in October it exists twice. The midnight jobs are safe, because the clocks change on a Sunday and nothing important runs then. Whether node-cron skips or doubles the daily check is untested. Either outcome is harmless here (the check is idempotent), but it's a good thing to have reasoned about when someone asks. The plan's validation scenario "GMT/BST transition" is still to be observed.

#State: three kinds of JSON file

stateManager.ts, under data/:

Why separate files: each one changes for a different reason, the song batches can be cleaned up without touching state, and you can cat any of them.

#Duplicates: harder than it looks

The BBC list rolls over week to week, and most songs stay on it for several weeks. So "don't add it twice" is the main job. It took four bug fixes to get right:

Identity is checked before matching, which also saves Spotify calls.

#The source-of-truth flip (#10)

This is the most important design change in the project.

Before: playlists/YYYY-MM.json was treated as the record of what's in the playlist. But you edit playlists in the Spotify app:

• You add a track by hand (after a review email). Our file doesn't know, so next week the song is flagged for review again, every week.
• You delete a track you don't like. Our file still lists it, and if it was ever rebuilt, the song came back while it stayed on the BBC list.

After: every run reads the live playlist from Spotify and treats it as the truth. Our file becomes a cache that mirrors it, plus things only we know:

• Manually added = live URIs we have no record of. They're recorded, so they're never flagged again.
• Manually removed = URIs we recorded adding that are no longer live. These become tombstones (removedIdentities), and a tombstoned song is never re-added. A tombstone is cleared if you put the track back.

"Tombstone" is a standard distributed-systems term: a record that says "this was deliberately deleted", so the deletion isn't mistaken for "never existed".

#The tombstone bug (#12)

The first version compared identities to detect removals. But recorded identities use the BBC's naming ("Love") and the live read returns Spotify's naming ("Concept of Love"). So a track that was still in the playlist looked deleted and got tombstoned, permanently. The fix compares URIs, the one identifier that's the same on both sides. The general rule: compare things using a key both sides agree on.

#The 1st-of-the-month race (#11)

When the 1st falls on a Tuesday (next time: 1 December 2026), the monthly and weekly jobs both fire at 00:00. If the weekly job ran first, it added the new month's songs to last month's playlist and recorded them there. The cross-month check then treated them as duplicates forever, so they never reached the right playlist.

The fix: the weekly updater checks state.currentMonth first and runs the monthly creator itself if needed. If that fails, it abandons the run rather than writing to the wrong month. The job queue added later in #19 also orders the two jobs, so there are now two separate protections.

#The daily checker

dailyStatusChecker.ts:

• Is state.currentMonth this month? If not, run the monthly creator.
• Is it Tuesday or later and weeklyRunCompleted false? If so, run the weekly updater.
• Clean up old song files.

Known gap (#15, open): weeklyRunCompleted is only reset to false by the monthly creator. After the first successful week of a month it stays true, so a missed or failed run in week 2, 3 or 4 is never retried. The safety net only covers the first week of each month.

#Email and logs

Email (notificationService.ts), via nodemailer over SMTP to a Plesk mailbox, is sent for:

• errors (scrape failed, playlist creation failed, startup failed)
• review lists (songs that need a human)
• "new playlist created"

It sits behind a NotificationChannel interface, so a second channel (the ADR imagines GitHub Issues) could be added without touching the callers. npm run send-test-email (#25) proves the production SMTP settings work before you depend on them. That's worth doing because alert paths that are never exercised quietly break. For local development, Mailpit catches everything on port 1025.

Logs (logger.ts) use Winston, with JSON lines to logs/application.log and logs/error.log, each capped at 10 MB × 5 files, plus a one-line console format that PM2 captures. Before #17, the files grew without limit and the console was silenced, so pm2 logs showed nothing. The plan had assumed pm2-logrotate would rotate the app's files, but it only rotates PM2's own captured output.

#Questions you might get asked

"How do you stop two jobs corrupting state?" There's one process, and jobs run through a promise-chain queue, so only one runs at a time. Deploys wait for the running job (5 minutes) before PM2's 6-minute kill.

"What if the server is down at midnight on Tuesday?" On restart, the startup check runs the daily checker, which sees the week isn't done and runs it. There's an honest caveat: that only works for the first week of each month until #15 is fixed.

"Why read the playlist back from Spotify every run?" Because I edit it by hand. Anything that a person can change outside your system can't be cached as truth. Our JSON is a cache, plus the one thing only we know: what we added, which is how we detect a deliberate deletion.

"What's a tombstone?" A record of a deliberate deletion, kept so the system doesn't re-add the item. We key it by Spotify URI, because the BBC's and Spotify's names for the same song differ. Keying by name caused #12.

"How do you handle time zones?" Cron and all date logic run on Europe/London through Luxon, storage is in UTC, and the locale is pinned so that the playlist names don't depend on the machine. I've reasoned through the DST hour and the 1st-on-a-Tuesday case, and the second one has a regression test.