#2. The BBC Side: Scraping a Page With No API

#The source

https://www.bbc.co.uk/programmes/articles/5JDPyPdDGs3yCLdtPhGgWM7/bbc-radio-6-music-playlist

It's an editorial article, not a data feed. A producer updates it on Mondays. It looks roughly like this:

<div class="text--prose">
  <p><strong>NEW ENTRIES ARE IN BOLD ◼️ Entries marked ↑ have moved category</strong></p>
  <h3>29TH JUNE-5TH JULY 2026</h3>
  <p><strong>Bleachers - you and forever</strong></p>
  <p>↑ Courtney Barnett - Mantis</p>
  <p>James Blake - Tracks from 'Trying Times'</p>
  ...
</div>

There's no API, no RSS and no JSON, so we scrape it: download the HTML and pick it apart. The code is src/services/bbcScraper.ts (about 230 lines).

A nice detail: the 2022 Python script this project grew out of used the selector div.text--prose > p. The TypeScript scraper uses the same selector four years later. The BBC has redesigned around that block, but not the block itself (see History).

#How it works

Three steps: fetch, parse, date.

#Fetch, with retries

for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
  try {
    return (await axios.get(this.url, { timeout: 30000, headers: { 'User-Agent': 'BBC6MusicPlaylistSync/1.0' } })).data;
  } catch (error) {
    if (this.isClientError(error)) throw ...;        // 4xx: retrying won't help
    if (attempt < this.maxRetries) await this.sleep(this.retryDelayMs * attempt);  // 5s, then 10s
  }
}

There are three ideas worth being able to explain here:

• Linear backoff (wait 5s, then 10s). The ADR's reasoning: it's a public page with no rate limit to respect, and exponential backoff (5s, 25s, 125s) would just be slower. Spotify gets different treatment, because Spotify does rate-limit.
• Don't retry 4xx. A 404 means the URL is wrong, and asking again won't change that. 5xx and network errors are usually temporary, so those get retried.
• An honest User-Agent. It names the bot, which is polite. Some sites block unknown agents, though, and this is a likely place for a future breakage.

#Parse

const $ = cheerio.load(html);
const container = $('div.text--prose');
if (container.length === 0) throw new BBCScraperError('container not found ...');
container.find('p').each(...)

Cheerio is jQuery for the server. It parses HTML into a tree you can query with CSS selectors, and it doesn't run a browser. For each <p>:

1. Skip noise. SKIP_PATTERNS covers lines in brackets, "NEW ENTRIES", "moved category" and the ◼️ emoji.
2. Strip the movement arrows (↑ ↓) from the start of the line.
3. Split on the separator. The regex is /\s+[-–—]\s+/, which matches a hyphen, en dash or em dash with spaces around it. The spaces matter: "Jay-Z" must not split. Only the first separator counts, so a title with a dash in it stays intact.
4. Reject editorial titles. EDITORIAL_TITLE_PATTERNS catches lines like "Tracks from 'Trying Times'" and "selected tracks" (#4). These look like Artist - Title but aren't songs.

Interesting fact: the album field exists in the data model (BBCSong.album) and the matcher gives +1000 points for an album match, but the BBC page never lists albums. That tier never fires in practice.

#Date

The scraper tries to read <time datetime="..."> to learn which week the playlist is for. That element doesn't exist on the page (#14, open). The code always falls back to "the most recent Monday" and logs a warning that nobody reads. The real date is in the <h3>, written as a range: 29TH JUNE-5TH JULY 2026.

This matters less than it sounds, because the month the songs go into no longer comes from this date. It comes from the processing state (#10). It's still a silent lie in the logs, though.

#Why axios + cheerio

If a reviewer asks "why axios in 2026?", the honest answer is the last row: it was a reasonable default when the plan was written, and it isn't worth replacing now.

#How scraping breaks, and what protects you

Scrapers fail in three ways. It's worth knowing all three:

1. Loudly. The container is missing, so we throw, so you get an email. That's the good case.
2. Quietly wrong. The page changes shape but still parses. For example, editorial lines start being read as songs (#4), or the date guess is silently wrong (#14).
3. Looks like success. The BBC's outage behaviour is a 302 redirect to /error.html, which returns 200 OK with a "Service Temporarily Unavailable" page. Axios follows the redirect, sees 200, and reports success. Then parsing fails outside the retry loop, so there's no retry, and the error message says the layout changed when the real cause is an outage (#15, open, high priority). The fixture tests/fixtures/bbc-service-unavailable.html is already waiting for that fix.

The defence is fixtures: saved copies of real pages in tests/fixtures/ (sample, empty, malformed, outage) that the parser is tested against. When the BBC changes the page, you save the new version as a fixture, watch the test go red, then fix the parser. That's TDD applied to someone else's HTML.

#Questions you might get asked

"Isn't scraping fragile?" Yes, and the design accepts that rather than hiding it. Parsing is isolated in one class and tested against saved real pages. If the container disappears the job fails loudly, and an email goes out. The remaining risk is quiet breakage, where the page changes but still parses. The open issues #14 and #15 are both examples of that, and they're the next things I'd fix.

"Is it legal or polite to scrape the BBC?" It's one request a week (plus retries) to a public page, with a User-Agent that identifies the bot, for a personal playlist. That's about as light as scraping gets. I'd check the BBC's terms before turning this into a public service.

"Why not use a headless browser to be safe?" Because the content is in the initial HTML. A browser would add around 100 MB of memory and a slow, flaky dependency to solve a problem we don't have. If the BBC moved to client-side rendering, that would be the moment to switch.

"How do you know the parser handles real-world input?" The fixtures are real pages, and every bug that came from a real page (editorial lines, an outage page) became a fixture and a test first.