Nicolas Fryder
260200e251
- apps/crawler/ : service Python complet, remplace les scripts locaux - FlareSolverr pour bypasser Cloudflare (cookies CF → session requests) - Crawl incrémental : /archives/band-list/by/created et /by/modified - Crawl complet Europe : pagination AJAX /browse/ajax-country/ - Enrichissement : pages individuelles de bands (themes, membres, label, hash) - Écriture directe en DB (upserts bulk, idempotents) - Scheduler intégré (schedule library) : incrémental 4h, enrich 2h, full le 1er du mois - Tracking via crawl_run et crawl_checkpoint (migration 004) - docker-compose.dev.yml : flaresolverr + crawler ajoutés Full crawl désactivé en dev (CRAWLER_SCHED_FULL_DAY=0) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
12 lines
267 B
Python
12 lines
267 B
Python
import random
|
|
import time
|
|
|
|
|
|
def sleep_range(min_s: float, max_s: float):
|
|
time.sleep(random.uniform(min_s, max_s))
|
|
|
|
|
|
def cooldown(min_s: float, max_s: float):
|
|
secs = random.uniform(min_s, max_s)
|
|
print(f"[polite] cooldown {secs:.1f}s")
|
|
time.sleep(secs)
|