nico/metalfrom.eu
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

fix(crawler): corriger le parsing de l'endpoint archive (6 colonnes)

Browse source 
L'endpoint /archives/ajax-band-list retourne 6 colonnes :
  [date, band_link, country_link, genre, time, user]
et non 4 comme supposé initialement. Le band_link est à row[1],
country à row[2], genre à row[3].

Les dates MA n'incluent pas l'année (ex: "June 1") donc date_str=None :
la logique d'arrêt incrémental sur date est désactivée, on re-fetch
les ~800 entrées récentes à chaque run (idempotent, coût négligeable).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Nicolas Fryder 2026-06-30 20:35:53 +02:00
parent dc242d1390
commit 94d94d926b
1 changed files with 15 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

21
apps/crawler/src/ma_http.py
View file

@ -240,24 +240,33 @@ def _parse_list_row(row, country_code: str) -> Optional[Dict]:
def _parse_archive_row(row, order_by: str) -> Optional[Dict]: def _parse_archive_row(row, order_by: str) -> Optional[Dict]:
"""Ligne des archives : [name_html, country_html, genre, date_str]""" """
Ligne des archives MA (6 colonnes) :
[date_label, band_link, country_link, genre, time_label, user_link]
ex: ["June 1", "<a href='/bands/Foo/123'>Foo</a>", "<a href='/lists/DE'>Germany</a>",
"Black Metal", "Jun 1st, 02:04", "<a href='/users/bar'>bar</a>"]
Note : les dates ("June 1", "Jun 1st, 02:04") n'incluent pas l'année → date_str=None,
la logique d'arrêt incrémental sur date est désactivée, on re-fetch toutes les ~800
entrées à chaque run (coût négligeable).
"""
if len(row) < 4: if len(row) < 4:
return None return None
href, name = _extract_link(row[0]) href, name = _extract_link(row[1])
ma_id = _extract_ma_id(href) ma_id = _extract_ma_id(href)
if not ma_id or not name: if not ma_id or not name:
return None return None
country_href, _ = _extract_link(row[1]) country_href, _ = _extract_link(row[2])
m = re.search(r"/lists/([A-Z]{1,3})", country_href or "") m = re.search(r"/lists/([A-Z]{1,3})", country_href or "")
cc = m.group(1) if m else _clean(row[1]) cc = m.group(1) if m else _clean(row[2])
return { return {
"ma_id": ma_id, "ma_id": ma_id,
"name": name, "name": name,
"url": href, "url": href,
"country": cc, "country": cc,
"genre": _clean(row[2]), "genre": _clean(row[3]),
"date_str": _clean(row[3]), "date_str": None, # pas d'année dans le format MA → pas de filtre date
"date_type": order_by, "date_type": order_by,
} }