metalfrom.eu/apps/crawler/requirements.txt at b5e3ef91ab00c290025e874a050fcd9ab85c5dfa - nico/metalfrom.eu - MEGA FORGE A NICO: CA FORGE UN MAX ICI POTO

nico/metalfrom.eu

Nicolas Fryder 260200e251 feat(crawler): nouveau service crawler server-side via FlareSolverr

- apps/crawler/ : service Python complet, remplace les scripts locaux
  - FlareSolverr pour bypasser Cloudflare (cookies CF → session requests)
  - Crawl incrémental : /archives/band-list/by/created et /by/modified
  - Crawl complet Europe : pagination AJAX /browse/ajax-country/
  - Enrichissement : pages individuelles de bands (themes, membres, label, hash)
  - Écriture directe en DB (upserts bulk, idempotents)
  - Scheduler intégré (schedule library) : incrémental 4h, enrich 2h, full le 1er du mois
  - Tracking via crawl_run et crawl_checkpoint (migration 004)
- docker-compose.dev.yml : flaresolverr + crawler ajoutés
  Full crawl désactivé en dev (CRAWLER_SCHED_FULL_DAY=0)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-27 15:15:29 +02:00

5 lines

91 B

Text

Raw Blame History

 requests==2.32.3
 psycopg2-binary==2.9.9
 beautifulsoup4==4.12.3
 lxml==5.3.0
 schedule==1.2.2