Compare commits
No commits in common. "b5e3ef91ab00c290025e874a050fcd9ab85c5dfa" and "228c678789217ebe5eb8ba5e48f9430b8b429853" have entirely different histories.
b5e3ef91ab
...
228c678789
56 changed files with 527 additions and 2156 deletions
|
|
@ -49,23 +49,15 @@ jobs:
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v4
|
- uses: actions/checkout@v4
|
||||||
|
|
||||||
# requirements-dev.txt, et non une liste recopiée : l'ancienne installait
|
|
||||||
# ruff/pytest/pip-audit seulement. psycopg2, beautifulsoup4, lxml et
|
|
||||||
# hypothesis manquaient, et le job aurait échoué dès la collecte — ce que
|
|
||||||
# personne n'a jamais vu, faute de runner enregistré.
|
|
||||||
- name: Installer l'outillage
|
- name: Installer l'outillage
|
||||||
run: pip install --no-cache-dir -r requirements-dev.txt
|
run: pip install --no-cache-dir ruff pytest pip-audit
|
||||||
|
|
||||||
- name: Ruff (lint + règles de sécurité bandit)
|
- name: Ruff (lint + règles de sécurité bandit)
|
||||||
run: ruff check .
|
run: ruff check .
|
||||||
|
|
||||||
- name: Tests du geocoder
|
- name: Tests du parseur de géocodage
|
||||||
run: cd apps/geocoder && python -m pytest tests -q
|
run: cd apps/geocoder && python -m pytest tests -q
|
||||||
|
|
||||||
# La suite du crawler n'était pas lancée du tout.
|
|
||||||
- name: Tests du crawler
|
|
||||||
run: cd apps/crawler && python -m pytest tests -q
|
|
||||||
|
|
||||||
- name: Audit des dépendances
|
- name: Audit des dépendances
|
||||||
run: pip-audit -r apps/crawler/requirements.txt -r apps/geocoder/requirements.txt
|
run: pip-audit -r apps/crawler/requirements.txt -r apps/geocoder/requirements.txt
|
||||||
continue-on-error: true
|
continue-on-error: true
|
||||||
|
|
|
||||||
53
README.md
53
README.md
|
|
@ -1,53 +1,2 @@
|
||||||
# Stack metalfrom.eu
|
# Stack metalfrom.eu
|
||||||
|
test 4
|
||||||
Carte des groupes de metal européens, données issues de Metal Archives
|
|
||||||
(avec accord du propriétaire du site source).
|
|
||||||
|
|
||||||
## Services
|
|
||||||
|
|
||||||
| Service | Rôle |
|
|
||||||
|--------------------|-----------------------------------------------------------------|
|
|
||||||
| `web` | Site public (carte Leaflet) |
|
|
||||||
| `admin` | Back-office (nginx + reverse-proxy vers l'API) |
|
|
||||||
| `api` | API Fastify (public + routes admin), migrations DB |
|
|
||||||
| `crawler` | Scraping incrémental/complet de Metal Archives via FlareSolverr |
|
|
||||||
| `flaresolverr` | Contournement Cloudflare (Chrome headless) pour le crawler |
|
|
||||||
| `geocoder-enqueue` | Peuple `band_locations` depuis `bands.location_text` |
|
|
||||||
| `geocoder-worker` | Géocodage Geoapify |
|
|
||||||
| `groq-worker` | Désambiguïsation LLM (Groq) des lieux non géocodés |
|
|
||||||
|
|
||||||
`crawler` et `flaresolverr` ne tournent **qu'en dev** pour l'instant : ils sont
|
|
||||||
absents de `docker-compose.yml`. La prod partage la même base, alimentée depuis
|
|
||||||
l'environnement de dev.
|
|
||||||
|
|
||||||
## Déploiement
|
|
||||||
|
|
||||||
- `docker-compose.yml` : production
|
|
||||||
- `docker-compose.dev.yml` : environnement de dev (branche `dev`, auto-déployé
|
|
||||||
via webhook Forgejo → Coolify)
|
|
||||||
|
|
||||||
Base PostgreSQL + PostGIS gérée séparément par Coolify. Variables d'env : voir
|
|
||||||
`infra/.env.example`.
|
|
||||||
|
|
||||||
## Qualité
|
|
||||||
|
|
||||||
Le déploiement se déclenche sur webhook à chaque push : rien ne s'interpose
|
|
||||||
entre `git push` et la mise en ligne. La porte de qualité est donc **locale**,
|
|
||||||
et doit être installée une fois par clone :
|
|
||||||
|
|
||||||
```
|
|
||||||
npm install
|
|
||||||
npm run hooks:install
|
|
||||||
```
|
|
||||||
|
|
||||||
| Commande | Portée |
|
|
||||||
|--------------------------------|------------------------------------------------------------|
|
|
||||||
| `npm run check` | lint + types + tests JS + ruff + pytest (~6 s) |
|
|
||||||
| `npm run test:e2e` | parcours Playwright du dashboard admin (~22 s) |
|
|
||||||
| `npm run test:integration:full`| SQL validé contre un vrai PostgreSQL+PostGIS (exige Docker)|
|
|
||||||
| `npm run check:full` | tout ci-dessus + mutation + audits de dépendances |
|
|
||||||
|
|
||||||
Le hook `pre-push` lance `check` puis `test:e2e`. `.forgejo/workflows/ci.yml`
|
|
||||||
existe mais reste inerte tant qu'aucun runner Forgejo n'est enregistré.
|
|
||||||
|
|
||||||
Outillage Python de test : `pip install -r requirements-dev.txt`.
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
FROM nginx:1.27-alpine
|
FROM nginx:alpine
|
||||||
ARG API_UPSTREAM=http://api:3000
|
ARG API_UPSTREAM=http://api:3000
|
||||||
COPY site/ /usr/share/nginx/html
|
COPY site/ /usr/share/nginx/html
|
||||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
||||||
|
|
|
||||||
|
|
@ -3,18 +3,6 @@ server {
|
||||||
root /usr/share/nginx/html;
|
root /usr/share/nginx/html;
|
||||||
index index.html;
|
index index.html;
|
||||||
|
|
||||||
server_tokens off;
|
|
||||||
|
|
||||||
# En-têtes de sécurité : le helmet de l'API ne couvre que les réponses JSON,
|
|
||||||
# pas les pages HTML/JS servies ici. Admin = actions destructrices → strict.
|
|
||||||
add_header X-Frame-Options "DENY" always;
|
|
||||||
add_header X-Content-Type-Options "nosniff" always;
|
|
||||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
|
||||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
|
||||||
# Tout est self-hosted, à une police Google près (<link> dans index.html).
|
|
||||||
# script-src 'self' est tenable : app.js n'utilise aucun handler inline.
|
|
||||||
add_header Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com; img-src 'self' data:; connect-src 'self'; frame-ancestors 'none'; base-uri 'self'; form-action 'self'" always;
|
|
||||||
|
|
||||||
# Résolveur DNS interne Docker : force nginx à re-résoudre "api" à chaque
|
# Résolveur DNS interne Docker : force nginx à re-résoudre "api" à chaque
|
||||||
# requête au lieu de mettre l'IP en cache au démarrage (sinon un redeploy
|
# requête au lieu de mettre l'IP en cache au démarrage (sinon un redeploy
|
||||||
# du service api laisse nginx pointer vers un conteneur mort -> 502/404).
|
# du service api laisse nginx pointer vers un conteneur mort -> 502/404).
|
||||||
|
|
@ -23,8 +11,6 @@ server {
|
||||||
location /admin/api/ {
|
location /admin/api/ {
|
||||||
set $upstream_api http://api:3000;
|
set $upstream_api http://api:3000;
|
||||||
proxy_pass $upstream_api;
|
proxy_pass $upstream_api;
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Connection "";
|
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
|
@ -34,8 +20,6 @@ server {
|
||||||
location /admin/auth/ {
|
location /admin/auth/ {
|
||||||
set $upstream_api http://api:3000;
|
set $upstream_api http://api:3000;
|
||||||
proxy_pass $upstream_api;
|
proxy_pass $upstream_api;
|
||||||
proxy_http_version 1.1;
|
|
||||||
proxy_set_header Connection "";
|
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
proxy_set_header X-Real-IP $remote_addr;
|
proxy_set_header X-Real-IP $remote_addr;
|
||||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||||
|
|
|
||||||
|
|
@ -33,21 +33,11 @@ const state = {
|
||||||
// API
|
// API
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
async function api(path, opts = {}) {
|
async function api(path, opts = {}) {
|
||||||
// Timeout explicite : sans lui, une API qui ne répond jamais (pool DB saturé,
|
const res = await fetch(path, {
|
||||||
// upstream nginx muet) laisse la vue bloquée sur « Chargement… » indéfiniment.
|
credentials: "include",
|
||||||
const ctrl = new AbortController();
|
headers: { "Content-Type": "application/json", ...(opts.headers || {}) },
|
||||||
const t = setTimeout(() => ctrl.abort(), opts.timeoutMs || 20000);
|
...opts,
|
||||||
let res;
|
});
|
||||||
try {
|
|
||||||
res = await fetch(path, {
|
|
||||||
credentials: "include",
|
|
||||||
headers: { "Content-Type": "application/json", ...(opts.headers || {}) },
|
|
||||||
signal: ctrl.signal,
|
|
||||||
...opts,
|
|
||||||
});
|
|
||||||
} finally {
|
|
||||||
clearTimeout(t);
|
|
||||||
}
|
|
||||||
if (res.status === 401) {
|
if (res.status === 401) {
|
||||||
showLogin();
|
showLogin();
|
||||||
throw new Error("unauthorized");
|
throw new Error("unauthorized");
|
||||||
|
|
@ -259,7 +249,6 @@ async function renderPilotage(token = renderToken) {
|
||||||
<button class="btn btn-danger" data-danger="reset-all">🔥 Réinitialiser tout le géocodage</button>
|
<button class="btn btn-danger" data-danger="reset-all">🔥 Réinitialiser tout le géocodage</button>
|
||||||
<button class="btn btn-danger" data-danger="purge-nominatim">🧹 Purger le cache Nominatim</button>
|
<button class="btn btn-danger" data-danger="purge-nominatim">🧹 Purger le cache Nominatim</button>
|
||||||
<button class="btn btn-danger" data-danger="requeue-all">♻️ Remettre TOUT en file (dont « done »)</button>
|
<button class="btn btn-danger" data-danger="requeue-all">♻️ Remettre TOUT en file (dont « done »)</button>
|
||||||
<button class="btn btn-danger" data-danger="reset-llm">🤖 Relancer les lieux abandonnés par le LLM</button>
|
|
||||||
</div>
|
</div>
|
||||||
<div id="p-danger-feedback" class="feedback" role="status"></div>
|
<div id="p-danger-feedback" class="feedback" role="status"></div>
|
||||||
</details>
|
</details>
|
||||||
|
|
@ -363,14 +352,6 @@ const DANGER_ACTIONS = {
|
||||||
body: { include_done: true },
|
body: { include_done: true },
|
||||||
confirm: "Remettre TOUTES les localisations en file, y compris celles déjà résolues ?",
|
confirm: "Remettre TOUTES les localisations en file, y compris celles déjà résolues ?",
|
||||||
},
|
},
|
||||||
// Cette route existait, était testée, et n'était appelée par aucun bouton :
|
|
||||||
// c'était pourtant le seul moyen de relancer les lieux passés en 'manual'
|
|
||||||
// après épuisement des tentatives LLM. La fonctionnalité existait sans que
|
|
||||||
// personne puisse l'atteindre.
|
|
||||||
"reset-llm": {
|
|
||||||
url: "/admin/api/locations/reset-llm",
|
|
||||||
confirm: "Relancer les lieux que le LLM n'a pas su résoudre ?\n\nIls repartent en file de géocodage — appels Geoapify et Groq facturés.",
|
|
||||||
},
|
|
||||||
};
|
};
|
||||||
|
|
||||||
function wireDangerZone() {
|
function wireDangerZone() {
|
||||||
|
|
@ -1187,7 +1168,7 @@ function activitySummary(row) {
|
||||||
if (sum.bands_new) parts.push(`${fmtNum(sum.bands_new)} nouveaux`);
|
if (sum.bands_new) parts.push(`${fmtNum(sum.bands_new)} nouveaux`);
|
||||||
if (sum.bands_updated) parts.push(`${fmtNum(sum.bands_updated)} MAJ`);
|
if (sum.bands_updated) parts.push(`${fmtNum(sum.bands_updated)} MAJ`);
|
||||||
if (sum.bands_enriched) parts.push(`${fmtNum(sum.bands_enriched)} enrichis`);
|
if (sum.bands_enriched) parts.push(`${fmtNum(sum.bands_enriched)} enrichis`);
|
||||||
if (sum.error) parts.push(`erreur : ${esc(sum.error)}`);
|
if (sum.error) parts.push(`erreur : ${sum.error}`);
|
||||||
return parts.join(" · ") || "—";
|
return parts.join(" · ") || "—";
|
||||||
}
|
}
|
||||||
return `${esc(sum.target_table || "")} #${sum.target_id ?? ""}`;
|
return `${esc(sum.target_table || "")} #${sum.target_id ?? ""}`;
|
||||||
|
|
|
||||||
|
|
@ -7,8 +7,5 @@ RUN npm ci --omit=dev
|
||||||
COPY src ./src
|
COPY src ./src
|
||||||
COPY migrations ./migrations
|
COPY migrations ./migrations
|
||||||
|
|
||||||
# node:20-alpine fournit déjà un utilisateur non-root "node"
|
|
||||||
USER node
|
|
||||||
|
|
||||||
ENV PORT=3000
|
ENV PORT=3000
|
||||||
CMD ["sh", "-c", "node src/migrate.js && node src/server.js"]
|
CMD ["sh", "-c", "node src/migrate.js && node src/server.js"]
|
||||||
|
|
|
||||||
|
|
@ -1,48 +0,0 @@
|
||||||
-- 016_updated_at_only_on_change.sql
|
|
||||||
-- Corrige la boucle de ré-enrichissement infinie.
|
|
||||||
--
|
|
||||||
-- Le trigger bands_set_geom (migration 002) faisait `NEW.updated_at := now()`
|
|
||||||
-- de façon INCONDITIONNELLE sur toute UPDATE. Or upsert_bands (crawler) fait un
|
|
||||||
-- `ON CONFLICT (ma_id) DO UPDATE SET name = COALESCE(EXCLUDED.name, bands.name), ...`
|
|
||||||
-- SANS clause WHERE : Postgres exécute donc l'UPDATE (et déclenche le trigger)
|
|
||||||
-- pour chaque ligne existante, même quand AUCUNE valeur ne change réellement.
|
|
||||||
--
|
|
||||||
-- Conséquence : chaque crawl incrémental/complet remettait updated_at=now() sur
|
|
||||||
-- des dizaines de milliers de bands inchangées. get_bands_to_enrich considérait
|
|
||||||
-- alors `updated_at > crawled_at + 1 min` comme vrai pour quasi toute la table,
|
|
||||||
-- ré-enrichissant en boucle des pages inchangées → charge inutile et permanente
|
|
||||||
-- vers metal-archives.com.
|
|
||||||
--
|
|
||||||
-- Fix : ne bumper updated_at que si la ligne change VRAIMENT (NEW IS DISTINCT
|
|
||||||
-- FROM OLD). Un upsert qui réécrit des valeurs identiques ne déclenche plus rien.
|
|
||||||
|
|
||||||
CREATE OR REPLACE FUNCTION bands_set_geom() RETURNS trigger AS $$
|
|
||||||
BEGIN
|
|
||||||
-- Ne recalculer geom que si lat ou lon a vraiment changé
|
|
||||||
IF TG_OP = 'INSERT'
|
|
||||||
OR OLD.lat IS DISTINCT FROM NEW.lat
|
|
||||||
OR OLD.lon IS DISTINCT FROM NEW.lon
|
|
||||||
THEN
|
|
||||||
IF NEW.lat IS NOT NULL AND NEW.lon IS NOT NULL THEN
|
|
||||||
NEW.geom := ST_SetSRID(ST_MakePoint(NEW.lon, NEW.lat), 4326)::geography;
|
|
||||||
ELSE
|
|
||||||
NEW.geom := NULL;
|
|
||||||
END IF;
|
|
||||||
END IF;
|
|
||||||
|
|
||||||
-- Ne bumper updated_at que sur changement réel de la ligne.
|
|
||||||
-- À ce stade NEW.updated_at == OLD.updated_at (pas encore modifié), donc la
|
|
||||||
-- comparaison ne se compare pas elle-même ; si toutes les autres colonnes sont
|
|
||||||
-- identiques, NEW IS DISTINCT FROM OLD est faux et updated_at reste inchangé.
|
|
||||||
IF TG_OP = 'INSERT' OR NEW IS DISTINCT FROM OLD THEN
|
|
||||||
NEW.updated_at := now();
|
|
||||||
END IF;
|
|
||||||
|
|
||||||
RETURN NEW;
|
|
||||||
END;
|
|
||||||
$$ LANGUAGE plpgsql;
|
|
||||||
|
|
||||||
DROP TRIGGER IF EXISTS trg_bands_set_geom ON bands;
|
|
||||||
CREATE TRIGGER trg_bands_set_geom
|
|
||||||
BEFORE INSERT OR UPDATE ON bands
|
|
||||||
FOR EACH ROW EXECUTE FUNCTION bands_set_geom();
|
|
||||||
|
|
@ -1,22 +0,0 @@
|
||||||
-- 017_enrich_pending_signal.sql
|
|
||||||
-- Découple « a besoin d'être ré-enrichi » de updated_at.
|
|
||||||
--
|
|
||||||
-- Après la migration 016, updated_at ne bouge plus que sur changement réel d'un
|
|
||||||
-- champ de listing (nom, pays, genre, statut, lieu). Problème : quand Metal
|
|
||||||
-- Archives modifie un groupe sur un champ visible UNIQUEMENT sur la page du
|
|
||||||
-- groupe (line-up, albums, thèmes), la liste archives/modified ne montre aucun
|
|
||||||
-- diff au niveau listing → sans signal dédié, ce groupe ne serait ré-enrichi
|
|
||||||
-- qu'au bout de 30 jours (filet stale).
|
|
||||||
--
|
|
||||||
-- Solution : le crawler incrémental « modified » capture l'horodatage "modified"
|
|
||||||
-- que MA affiche lui-même dans sa liste (ma_modified_seen). Quand ce texte change
|
|
||||||
-- pour un groupe, on lève enrich_pending — une seule fois par modification MA,
|
|
||||||
-- sans boucle. L'enrichissement remet enrich_pending à false.
|
|
||||||
|
|
||||||
ALTER TABLE bands
|
|
||||||
ADD COLUMN IF NOT EXISTS ma_modified_seen TEXT,
|
|
||||||
ADD COLUMN IF NOT EXISTS enrich_pending BOOLEAN NOT NULL DEFAULT false;
|
|
||||||
|
|
||||||
-- Index partiel pour la file d'enrichissement (peu de lignes à true à la fois)
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_bands_enrich_pending
|
|
||||||
ON bands (enrich_pending) WHERE enrich_pending;
|
|
||||||
|
|
@ -1,32 +0,0 @@
|
||||||
-- 018_protect_admin_locations.sql
|
|
||||||
-- Les corrections manuelles de géocodage n'étaient protégées par rien.
|
|
||||||
--
|
|
||||||
-- Corriger à la main les coordonnées d'un groupe écrit un band_locations avec
|
|
||||||
-- geocode_provider='admin' et geocode_confidence=1.0 : la donnée la plus
|
|
||||||
-- autoritaire du système, saisie par un humain, impossible à régénérer.
|
|
||||||
--
|
|
||||||
-- Or le trigger de la migration 013 supprime TOUTES les band_locations d'un
|
|
||||||
-- groupe dès que son location_text change — override compris. Et locked_fields
|
|
||||||
-- ne protège que les colonnes de `bands`, jamais les points. Une correction
|
|
||||||
-- disparaissait donc silencieusement à la première modification du lieu, y
|
|
||||||
-- compris quand ce lieu venait d'être verrouillé par l'admin lui-même.
|
|
||||||
--
|
|
||||||
-- Le trigger épargne désormais les points posés par un admin. Ils restent
|
|
||||||
-- supprimables explicitement (DELETE depuis la route, reset-all ciblé), mais
|
|
||||||
-- plus par effet de bord.
|
|
||||||
|
|
||||||
CREATE OR REPLACE FUNCTION bands_geocode_dirty() RETURNS trigger AS $$
|
|
||||||
BEGIN
|
|
||||||
IF TG_OP = 'UPDATE' AND OLD.location_text IS DISTINCT FROM NEW.location_text THEN
|
|
||||||
DELETE FROM band_locations
|
|
||||||
WHERE ma_id = NEW.ma_id
|
|
||||||
AND COALESCE(geocode_provider, '') <> 'admin';
|
|
||||||
END IF;
|
|
||||||
RETURN NEW;
|
|
||||||
END;
|
|
||||||
$$ LANGUAGE plpgsql;
|
|
||||||
|
|
||||||
DROP TRIGGER IF EXISTS trg_bands_geocode_dirty ON bands;
|
|
||||||
CREATE TRIGGER trg_bands_geocode_dirty
|
|
||||||
AFTER UPDATE OF location_text ON bands
|
|
||||||
FOR EACH ROW EXECUTE FUNCTION bands_geocode_dirty();
|
|
||||||
|
|
@ -1,93 +0,0 @@
|
||||||
-- 019_status_constraints_and_retention.sql
|
|
||||||
--
|
|
||||||
-- A) Contraintes de domaine sur les statuts
|
|
||||||
--
|
|
||||||
-- Dix-huit migrations, zéro CHECK. crawl_run.status, job_triggers.job_type et
|
|
||||||
-- band_locations.geocode_status sont du texte libre dont les valeurs admises ne
|
|
||||||
-- vivaient que dans des commentaires. C'est le dénominateur commun de plusieurs
|
|
||||||
-- bugs de ce dépôt : un statut 'error' que plus personne n'écrivait mais qu'un
|
|
||||||
-- bouton d'admin ciblait encore, des lignes 'processing' qu'aucune requête ne
|
|
||||||
-- reprenait, un job_type réclamé par le mauvais daemon. Une faute de frappe
|
|
||||||
-- écrivait une ligne durablement invisible, sans jamais rien faire échouer.
|
|
||||||
--
|
|
||||||
-- Les valeurs sont relevées sur le code qui écrit réellement ces colonnes
|
|
||||||
-- (crawler/src/db.py, geocoder/src/*.py, api/src/adminRoutes.js).
|
|
||||||
--
|
|
||||||
-- NOT VALID est délibéré et DÉFINITIF ici : la contrainte s'applique à toutes
|
|
||||||
-- les écritures futures, mais l'historique n'est pas relu. Cette migration
|
|
||||||
-- s'exécute au démarrage de chaque conteneur API ; valider exigerait un scan
|
|
||||||
-- complet, et une seule ligne héritée hors domaine ferait échouer la migration,
|
|
||||||
-- donc le démarrage de l'API. Le but est d'arrêter la dérive, pas de réécrire
|
|
||||||
-- le passé. Pour valider un jour, à froid :
|
|
||||||
-- ALTER TABLE crawl_run VALIDATE CONSTRAINT crawl_run_status_check;
|
|
||||||
|
|
||||||
ALTER TABLE crawl_run DROP CONSTRAINT IF EXISTS crawl_run_status_check;
|
|
||||||
ALTER TABLE crawl_run ADD CONSTRAINT crawl_run_status_check
|
|
||||||
CHECK (status IN ('running', 'done', 'error', 'cancelled')) NOT VALID;
|
|
||||||
|
|
||||||
ALTER TABLE job_triggers DROP CONSTRAINT IF EXISTS job_triggers_status_check;
|
|
||||||
ALTER TABLE job_triggers ADD CONSTRAINT job_triggers_status_check
|
|
||||||
CHECK (status IN ('pending', 'running', 'done', 'error', 'cancelled')) NOT VALID;
|
|
||||||
|
|
||||||
ALTER TABLE job_triggers DROP CONSTRAINT IF EXISTS job_triggers_type_check;
|
|
||||||
ALTER TABLE job_triggers ADD CONSTRAINT job_triggers_type_check
|
|
||||||
CHECK (job_type IN ('enrich', 'incremental', 'full_crawl', 'geocoder_enqueue')) NOT VALID;
|
|
||||||
|
|
||||||
ALTER TABLE band_locations DROP CONSTRAINT IF EXISTS band_locations_geocode_status_check;
|
|
||||||
ALTER TABLE band_locations ADD CONSTRAINT band_locations_geocode_status_check
|
|
||||||
CHECK (geocode_status IN (
|
|
||||||
'queued', 'processing', 'done', 'country_only', 'error', 'llm_needed', 'manual'
|
|
||||||
)) NOT VALID;
|
|
||||||
|
|
||||||
-- ------------------------------------------------------------------
|
|
||||||
-- B) Table morte
|
|
||||||
--
|
|
||||||
-- geocode_queue date du pipeline Nominatim, remplacé par band_locations. Plus
|
|
||||||
-- aucune ligne de code ne la référence ; elle ne subsistait que dans le schéma.
|
|
||||||
-- ------------------------------------------------------------------
|
|
||||||
DROP TABLE IF EXISTS geocode_queue;
|
|
||||||
|
|
||||||
-- ------------------------------------------------------------------
|
|
||||||
-- C) Rétention
|
|
||||||
--
|
|
||||||
-- crawl_log, admin_audit_log et admin_login_attempts grossissent sans fin. La
|
|
||||||
-- migration 015 avait pourtant explicitement conçu service_health pour éviter
|
|
||||||
-- « une table qui grossit » — la leçon n'avait pas été appliquée aux autres.
|
|
||||||
-- crawl_log prend une ligne par échec d'enrichissement, et GET /admin/api/activity
|
|
||||||
-- relit admin_audit_log en entier à chaque affichage.
|
|
||||||
--
|
|
||||||
-- La purge n'est pas automatique (pas de pg_cron ici) : elle est exposée par
|
|
||||||
-- l'API, qui l'appelle au démarrage. Les durées sont volontairement longues,
|
|
||||||
-- l'objectif est de borner la croissance, pas d'effacer l'historique utile.
|
|
||||||
-- ------------------------------------------------------------------
|
|
||||||
CREATE OR REPLACE FUNCTION purge_historique(
|
|
||||||
jours_logs INT DEFAULT 90,
|
|
||||||
jours_audit INT DEFAULT 365,
|
|
||||||
jours_tentatives INT DEFAULT 30
|
|
||||||
) RETURNS TABLE (table_purgee TEXT, lignes BIGINT) AS $$
|
|
||||||
BEGIN
|
|
||||||
RETURN QUERY
|
|
||||||
WITH d AS (
|
|
||||||
DELETE FROM crawl_log WHERE created_at < now() - make_interval(days => jours_logs)
|
|
||||||
RETURNING 1
|
|
||||||
) SELECT 'crawl_log'::TEXT, count(*) FROM d;
|
|
||||||
|
|
||||||
RETURN QUERY
|
|
||||||
WITH d AS (
|
|
||||||
DELETE FROM admin_audit_log WHERE created_at < now() - make_interval(days => jours_audit)
|
|
||||||
RETURNING 1
|
|
||||||
) SELECT 'admin_audit_log'::TEXT, count(*) FROM d;
|
|
||||||
|
|
||||||
RETURN QUERY
|
|
||||||
WITH d AS (
|
|
||||||
DELETE FROM admin_login_attempts WHERE created_at < now() - make_interval(days => jours_tentatives)
|
|
||||||
RETURNING 1
|
|
||||||
) SELECT 'admin_login_attempts'::TEXT, count(*) FROM d;
|
|
||||||
END;
|
|
||||||
$$ LANGUAGE plpgsql;
|
|
||||||
|
|
||||||
-- Index manquant : la purge et le filtre `since` de /admin/api/logs balayaient
|
|
||||||
-- crawl_log en entier faute d'index sur created_at seul (celui de la 006 est
|
|
||||||
-- DESC, utilisable, mais on garde l'intention explicite ici).
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_admin_login_attempts_created ON admin_login_attempts (created_at);
|
|
||||||
CREATE INDEX IF NOT EXISTS idx_admin_audit_log_created ON admin_audit_log (created_at);
|
|
||||||
|
|
@ -8,13 +8,12 @@
|
||||||
"start": "node src/server.js"
|
"start": "node src/server.js"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@fastify/compress": "^7.0.3",
|
|
||||||
"@fastify/cookie": "^9.4.0",
|
|
||||||
"@fastify/helmet": "^11.1.1",
|
|
||||||
"@fastify/rate-limit": "^9.0.0",
|
|
||||||
"bcryptjs": "^2.4.3",
|
|
||||||
"fastify": "^4.28.1",
|
"fastify": "^4.28.1",
|
||||||
"jsonwebtoken": "^9.0.2",
|
"@fastify/rate-limit": "^9.0.0",
|
||||||
"pg": "^8.12.0"
|
"@fastify/helmet": "^11.1.1",
|
||||||
|
"@fastify/cookie": "^9.4.0",
|
||||||
|
"pg": "^8.12.0",
|
||||||
|
"bcryptjs": "^2.4.3",
|
||||||
|
"jsonwebtoken": "^9.0.2"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -4,20 +4,7 @@ import jwt from "jsonwebtoken";
|
||||||
export const ADMIN_COOKIE_NAME = "admin_session";
|
export const ADMIN_COOKIE_NAME = "admin_session";
|
||||||
const SESSION_TTL_S = 12 * 3600; // 12h
|
const SESSION_TTL_S = 12 * 3600; // 12h
|
||||||
const LOCKOUT_WINDOW_MIN = 15;
|
const LOCKOUT_WINDOW_MIN = 15;
|
||||||
// Deux compteurs distincts, volontairement asymetriques.
|
const LOCKOUT_MAX_ATTEMPTS = 5;
|
||||||
//
|
|
||||||
// L'ancienne requete comptait `username = $2 OR ip = $3` dans un seul total :
|
|
||||||
// cinq echecs avec le nom d'un admin, depuis n'importe quelle adresse,
|
|
||||||
// verrouillaient ce compte un quart d'heure. N'importe qui connaissant le nom
|
|
||||||
// d'utilisateur pouvait donc mettre l'admin dehors, indefiniment et sans cout.
|
|
||||||
//
|
|
||||||
// L'IP reste stricte : c'est elle qui freine une attaque par force brute, et
|
|
||||||
// s'auto-verrouiller n'a aucun interet pour un attaquant. Le compteur par
|
|
||||||
// username subsiste contre une attaque repartie sur plusieurs adresses, mais
|
|
||||||
// avec un seuil bien plus haut : un attaquant devra bruler 4 IP avant de
|
|
||||||
// commencer a genner le titulaire du compte.
|
|
||||||
const LOCKOUT_MAX_PER_IP = 5;
|
|
||||||
const LOCKOUT_MAX_PER_USERNAME = 20;
|
|
||||||
// Hash bcrypt valide mais sans correspondance, utilisé pour égaliser le temps
|
// Hash bcrypt valide mais sans correspondance, utilisé pour égaliser le temps
|
||||||
// de réponse quand le username n'existe pas (évite l'énumération de comptes).
|
// de réponse quand le username n'existe pas (évite l'énumération de comptes).
|
||||||
const DUMMY_HASH = "$2b$12$CwTycUXWue0Thq9StjUM0uJ8vKR1dlT0LYzGsv8ZE8nFI8q9Z5T9.";
|
const DUMMY_HASH = "$2b$12$CwTycUXWue0Thq9StjUM0uJ8vKR1dlT0LYzGsv8ZE8nFI8q9Z5T9.";
|
||||||
|
|
@ -58,17 +45,14 @@ export function verifyAdminSession(token) {
|
||||||
|
|
||||||
export async function isLockedOut(pool, username, ip) {
|
export async function isLockedOut(pool, username, ip) {
|
||||||
const r = await pool.query(
|
const r = await pool.query(
|
||||||
`SELECT
|
`SELECT count(*)::int AS n
|
||||||
count(*) FILTER (WHERE ip = $3)::int AS par_ip,
|
|
||||||
count(*) FILTER (WHERE username = $2)::int AS par_username
|
|
||||||
FROM admin_login_attempts
|
FROM admin_login_attempts
|
||||||
WHERE success = false
|
WHERE success = false
|
||||||
AND created_at > now() - make_interval(mins => $1)
|
AND created_at > now() - make_interval(mins => $1)
|
||||||
AND (username = $2 OR ip = $3)`,
|
AND (username = $2 OR ip = $3)`,
|
||||||
[LOCKOUT_WINDOW_MIN, username, ip]
|
[LOCKOUT_WINDOW_MIN, username, ip]
|
||||||
);
|
);
|
||||||
const { par_ip, par_username } = r.rows[0];
|
return r.rows[0].n >= LOCKOUT_MAX_ATTEMPTS;
|
||||||
return par_ip >= LOCKOUT_MAX_PER_IP || par_username >= LOCKOUT_MAX_PER_USERNAME;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function recordLoginAttempt(pool, username, ip, success) {
|
export async function recordLoginAttempt(pool, username, ip, success) {
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import { requireAdminSession, writeAuditLog } from "./adminAuth.js";
|
import { requireAdminSession, writeAuditLog } from "./adminAuth.js";
|
||||||
import { ValidationError, pagination, parseLat, parseLon, parseYear } from "./validate.js";
|
import { pagination } from "./validate.js";
|
||||||
|
|
||||||
const BAND_SORT_COLUMNS = new Set([
|
const BAND_SORT_COLUMNS = new Set([
|
||||||
"ma_id", "name", "country", "status", "genre",
|
"ma_id", "name", "country", "status", "genre",
|
||||||
|
|
@ -248,18 +248,26 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
if (Object.keys(updates).length === 0) {
|
if (Object.keys(updates).length === 0) {
|
||||||
return reply.code(400).send({ ok: false, error: "Aucun champ à modifier" });
|
return reply.code(400).send({ ok: false, error: "Aucun champ à modifier" });
|
||||||
}
|
}
|
||||||
// Mêmes règles que les routes publiques, et une seule implémentation :
|
if ("formed_year" in updates) {
|
||||||
// elles étaient recopiées à la main ici alors que validate.js les portait
|
const y = updates.formed_year === null ? null : Number(updates.formed_year);
|
||||||
// déjà, testées par 64 cas qui ne couvraient donc pas ce qui tournait.
|
if (y !== null && (!Number.isFinite(y) || y < 1800 || y > 2100)) {
|
||||||
try {
|
return reply.code(400).send({ ok: false, error: "formed_year invalide" });
|
||||||
if ("formed_year" in updates) updates.formed_year = parseYear(updates.formed_year, "formed_year");
|
|
||||||
if ("lat" in updates) updates.lat = parseLat(updates.lat);
|
|
||||||
if ("lon" in updates) updates.lon = parseLon(updates.lon);
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof ValidationError) {
|
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
|
||||||
}
|
}
|
||||||
throw err;
|
updates.formed_year = y;
|
||||||
|
}
|
||||||
|
if ("lat" in updates) {
|
||||||
|
const v = updates.lat === null ? null : Number(updates.lat);
|
||||||
|
if (v !== null && (!Number.isFinite(v) || v < -90 || v > 90)) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "lat invalide" });
|
||||||
|
}
|
||||||
|
updates.lat = v;
|
||||||
|
}
|
||||||
|
if ("lon" in updates) {
|
||||||
|
const v = updates.lon === null ? null : Number(updates.lon);
|
||||||
|
if (v !== null && (!Number.isFinite(v) || v < -180 || v > 180)) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "lon invalide" });
|
||||||
|
}
|
||||||
|
updates.lon = v;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Transaction : l'edition, la pose de l'override de localisation et
|
// Transaction : l'edition, la pose de l'override de localisation et
|
||||||
|
|
@ -348,6 +356,21 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ------------------------------------------------------------------
|
||||||
|
// Checkpoints
|
||||||
|
// (l'historique des crawl_run est exposé de façon unifiée par
|
||||||
|
// GET /admin/api/activity, avec le détail logs via GET /admin/api/logs?run_id=)
|
||||||
|
// ------------------------------------------------------------------
|
||||||
|
fastify.get("/admin/api/crawl-checkpoints", async (req, reply) => {
|
||||||
|
try {
|
||||||
|
const r = await pool.query(`SELECT key, value, updated_at FROM crawl_checkpoint ORDER BY key`);
|
||||||
|
return { ok: true, items: r.rows };
|
||||||
|
} catch (err) {
|
||||||
|
fastify.log.error(err);
|
||||||
|
return reply.code(500).send({ ok: false, error: "Erreur checkpoints" });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
// Logs crawler
|
// Logs crawler
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
|
|
@ -386,15 +409,7 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
fastify.post("/admin/api/crawl-runs/cleanup", async (req, reply) => {
|
fastify.post("/admin/api/crawl-runs/cleanup", async (req, reply) => {
|
||||||
try {
|
try {
|
||||||
// 30 minutes par défaut était plus court qu'un crawl complet Europe, qui
|
const olderThanMinutes = Math.max(1, Number(req.body?.older_than_minutes) || 30);
|
||||||
// dure des heures : déclencher ce nettoyage pendant un crawl légitime le
|
|
||||||
// marquait en erreur alors qu'il tournait toujours, et
|
|
||||||
// update_crawl_run_progress (filtré sur status='running') cessait
|
|
||||||
// silencieusement de publier — l'affichage restait figé jusqu'à la fin.
|
|
||||||
// C'est exactement le double mensonge que la migration 014 avait corrigé
|
|
||||||
// sur le bouton Annuler. Le défaut couvre désormais le plus long run
|
|
||||||
// attendu ; un seuil plus court reste possible explicitement.
|
|
||||||
const olderThanMinutes = Math.max(1, Number(req.body?.older_than_minutes) || 24 * 60);
|
|
||||||
const r = await pool.query(`
|
const r = await pool.query(`
|
||||||
UPDATE crawl_run
|
UPDATE crawl_run
|
||||||
SET status = 'error',
|
SET status = 'error',
|
||||||
|
|
@ -402,10 +417,6 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
error = 'annulé manuellement (run bloqué)'
|
error = 'annulé manuellement (run bloqué)'
|
||||||
WHERE status = 'running'
|
WHERE status = 'running'
|
||||||
AND started_at < now() - make_interval(mins => $1)
|
AND started_at < now() - make_interval(mins => $1)
|
||||||
-- Une annulation déjà demandée suit le chemin coopératif : le
|
|
||||||
-- crawler la lit et écrira lui-même 'cancelled'. Forcer 'error'
|
|
||||||
-- par-dessus reviendrait à mentir sur l'issue du run.
|
|
||||||
AND cancel_requested = FALSE
|
|
||||||
RETURNING id, run_type, started_at
|
RETURNING id, run_type, started_at
|
||||||
`, [olderThanMinutes]);
|
`, [olderThanMinutes]);
|
||||||
await writeAuditLog(pool, req.adminUsername, "cleanup_stuck_runs", "crawl_run", null, null, { cleaned: r.rows });
|
await writeAuditLog(pool, req.adminUsername, "cleanup_stuck_runs", "crawl_run", null, null, { cleaned: r.rows });
|
||||||
|
|
@ -613,9 +624,6 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
geocode_tries_geo=0, geocode_tries_llm=0,
|
geocode_tries_geo=0, geocode_tries_llm=0,
|
||||||
geocode_next_at=now(), updated_at=now()
|
geocode_next_at=now(), updated_at=now()
|
||||||
WHERE is_country_only = FALSE
|
WHERE is_country_only = FALSE
|
||||||
-- Les saisies manuelles sont épargnées : elles n'ont pas de source
|
|
||||||
-- automatique à re-jouer, les effacer les perd définitivement.
|
|
||||||
AND COALESCE(geocode_provider, '') <> 'admin'
|
|
||||||
`);
|
`);
|
||||||
const count = r.rowCount;
|
const count = r.rowCount;
|
||||||
await pool.query(
|
await pool.query(
|
||||||
|
|
@ -758,20 +766,17 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
return reply.code(400).send({ ok: false, error: "bad id" });
|
return reply.code(400).send({ ok: false, error: "bad id" });
|
||||||
}
|
}
|
||||||
const { lat, lon } = req.body || {};
|
const { lat, lon } = req.body || {};
|
||||||
let latN, lonN;
|
const latN = lat === null || lat === undefined || lat === "" ? null : Number(lat);
|
||||||
try {
|
const lonN = lon === null || lon === undefined || lon === "" ? null : Number(lon);
|
||||||
latN = parseLat(lat);
|
|
||||||
lonN = parseLon(lon);
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof ValidationError) {
|
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
|
||||||
}
|
|
||||||
throw err;
|
|
||||||
}
|
|
||||||
// parseLat/parseLon acceptent null (effacement) ; ici les deux sont requis.
|
|
||||||
if (latN === null || lonN === null) {
|
if (latN === null || lonN === null) {
|
||||||
return reply.code(400).send({ ok: false, error: "lat et lon requis" });
|
return reply.code(400).send({ ok: false, error: "lat et lon requis" });
|
||||||
}
|
}
|
||||||
|
if (!Number.isFinite(latN) || latN < -90 || latN > 90) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "lat invalide" });
|
||||||
|
}
|
||||||
|
if (!Number.isFinite(lonN) || lonN < -180 || lonN > 180) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "lon invalide" });
|
||||||
|
}
|
||||||
|
|
||||||
const before = await pool.query(`SELECT * FROM band_locations WHERE id = $1`, [id]);
|
const before = await pool.query(`SELECT * FROM band_locations WHERE id = $1`, [id]);
|
||||||
if (!before.rows.length) return reply.code(404).send({ ok: false, error: "not found" });
|
if (!before.rows.length) return reply.code(404).send({ ok: false, error: "not found" });
|
||||||
|
|
@ -867,19 +872,13 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
// Géocodage — actions sur band_locations (nouveau pipeline)
|
// Géocodage — actions sur band_locations (nouveau pipeline)
|
||||||
// ------------------------------------------------------------------
|
// ------------------------------------------------------------------
|
||||||
// Débloque les lignes coincées en 'processing' (worker tué en plein
|
|
||||||
// traitement) en plus de l'ancien statut 'error' (legacy). Le worker se répare
|
|
||||||
// aussi tout seul au bout de GEOCODE_STUCK_PROCESSING_MIN ; ce bouton est un
|
|
||||||
// forçage manuel pour ne pas attendre.
|
|
||||||
fastify.post("/admin/api/locations/reset-errors", async (req, reply) => {
|
fastify.post("/admin/api/locations/reset-errors", async (req, reply) => {
|
||||||
try {
|
try {
|
||||||
const r = await pool.query(`
|
const r = await pool.query(`
|
||||||
UPDATE band_locations
|
UPDATE band_locations
|
||||||
SET geocode_status='queued', geocode_tries_geo=0,
|
SET geocode_status='queued', geocode_tries_geo=0,
|
||||||
geocode_error=NULL, geocode_next_at=now(), updated_at=now()
|
geocode_error=NULL, geocode_next_at=now(), updated_at=now()
|
||||||
WHERE (geocode_status = 'error'
|
WHERE geocode_status = 'error'
|
||||||
OR (geocode_status = 'processing' AND updated_at < now() - interval '5 minutes'))
|
|
||||||
AND COALESCE(geocode_provider, '') <> 'admin'
|
|
||||||
`);
|
`);
|
||||||
const count = r.rowCount;
|
const count = r.rowCount;
|
||||||
await pool.query(
|
await pool.query(
|
||||||
|
|
@ -925,9 +924,6 @@ export default async function adminRoutes(fastify, opts) {
|
||||||
geocode_query=NULL, geocode_error=NULL, geocode_next_at=now(), updated_at=now()
|
geocode_query=NULL, geocode_error=NULL, geocode_next_at=now(), updated_at=now()
|
||||||
WHERE geocode_status = ANY($1::text[])
|
WHERE geocode_status = ANY($1::text[])
|
||||||
AND is_country_only = FALSE
|
AND is_country_only = FALSE
|
||||||
-- Idem reset-all : include_done englobait les corrections manuelles,
|
|
||||||
-- que rien ne permet de reconstituer.
|
|
||||||
AND COALESCE(geocode_provider, '') <> 'admin'
|
|
||||||
`, [statuses]);
|
`, [statuses]);
|
||||||
const count = r.rowCount;
|
const count = r.rowCount;
|
||||||
await pool.query(
|
await pool.query(
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,6 @@
|
||||||
*/
|
*/
|
||||||
import Fastify from "fastify";
|
import Fastify from "fastify";
|
||||||
import rateLimit from "@fastify/rate-limit";
|
import rateLimit from "@fastify/rate-limit";
|
||||||
import compress from "@fastify/compress";
|
|
||||||
import helmet from "@fastify/helmet";
|
import helmet from "@fastify/helmet";
|
||||||
import cookie from "@fastify/cookie";
|
import cookie from "@fastify/cookie";
|
||||||
import { timingSafeEqual } from "crypto";
|
import { timingSafeEqual } from "crypto";
|
||||||
|
|
@ -22,17 +21,7 @@ import {
|
||||||
requireAdminSession,
|
requireAdminSession,
|
||||||
} from "./adminAuth.js";
|
} from "./adminAuth.js";
|
||||||
import adminApiRoutes from "./adminRoutes.js";
|
import adminApiRoutes from "./adminRoutes.js";
|
||||||
import {
|
import { ValidationError, sanitizeSearchString, parseLimitOffset } from "./validate.js";
|
||||||
ValidationError,
|
|
||||||
sanitizeSearchString,
|
|
||||||
parseLimitOffset,
|
|
||||||
parseBbox,
|
|
||||||
parseZoom,
|
|
||||||
cellSizeForZoom,
|
|
||||||
parseCsvList,
|
|
||||||
parseYear,
|
|
||||||
parseMaId,
|
|
||||||
} from "./validate.js";
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @param {object} [opts]
|
* @param {object} [opts]
|
||||||
|
|
@ -45,12 +34,6 @@ import {
|
||||||
* @param {number} [opts.adminRateLimitMax] requêtes/min sur /admin/import
|
* @param {number} [opts.adminRateLimitMax] requêtes/min sur /admin/import
|
||||||
* @param {number} [opts.authRateLimitMax] requêtes/min sur /admin/auth/login
|
* @param {number} [opts.authRateLimitMax] requêtes/min sur /admin/auth/login
|
||||||
*/
|
*/
|
||||||
// Plafonds de /api/clusters. Nommés plutôt qu'écrits en dur dans le SQL : la
|
|
||||||
// réponse doit pouvoir dire qu'elle a été tronquée, ce qui suppose de comparer
|
|
||||||
// au même nombre des deux côtés.
|
|
||||||
const BANDS_AT_ZOOM_LIMIT = 2000;
|
|
||||||
const CLUSTERS_LIMIT = 1000;
|
|
||||||
|
|
||||||
export async function buildServer(opts = {}) {
|
export async function buildServer(opts = {}) {
|
||||||
const {
|
const {
|
||||||
pool = null,
|
pool = null,
|
||||||
|
|
@ -77,28 +60,6 @@ export async function buildServer(opts = {}) {
|
||||||
trustProxy: true // derrière Traefik (+ nginx pour /admin/*) : lit X-Forwarded-For
|
trustProxy: true // derrière Traefik (+ nginx pour /admin/*) : lit X-Forwarded-For
|
||||||
});
|
});
|
||||||
|
|
||||||
// Compression.
|
|
||||||
//
|
|
||||||
// Le site public charge l'intégralité du corpus une seconde après l'ouverture
|
|
||||||
// de la page (loadAllHeatPoints), pour la heatmap et la recherche. Mesuré sur
|
|
||||||
// 100 000 groupes avec des données réalistes : 22 Mo non compressés contre
|
|
||||||
// 1,9 Mo en gzip. Rien ne compressait dans la chaîne — ni ici, ni Traefik.
|
|
||||||
// Ce n'était donc pas un risque théorique mais la facture de chaque visiteur,
|
|
||||||
// à chaque visite.
|
|
||||||
//
|
|
||||||
// Seuil à 1 Ko : en dessous, l'en-tête et le temps CPU coûtent plus que ce
|
|
||||||
// qu'ils économisent.
|
|
||||||
//
|
|
||||||
// gzip est préféré à brotli, contre l'intuition : mesuré sur la réponse
|
|
||||||
// complète de /api/bands, gzip sort 1,9 Mo contre 2,3 Mo pour brotli. La
|
|
||||||
// qualité brotli par défaut est réglée pour la vitesse, et sur ce JSON très
|
|
||||||
// répétitif elle perd sur les deux tableaux à la fois — taille ET CPU.
|
|
||||||
await fastify.register(compress, {
|
|
||||||
global: true,
|
|
||||||
threshold: 1024,
|
|
||||||
encodings: ["gzip", "br", "deflate"],
|
|
||||||
});
|
|
||||||
|
|
||||||
// Headers de sécurité
|
// Headers de sécurité
|
||||||
await fastify.register(helmet, {
|
await fastify.register(helmet, {
|
||||||
contentSecurityPolicy: false,
|
contentSecurityPolicy: false,
|
||||||
|
|
@ -112,14 +73,6 @@ export async function buildServer(opts = {}) {
|
||||||
const origin = request.headers.origin;
|
const origin = request.headers.origin;
|
||||||
const allowedOrigins = corsOrigins;
|
const allowedOrigins = corsOrigins;
|
||||||
|
|
||||||
// `Vary: Origin` est indispensable des lors que l'en-tete de reponse DEPEND
|
|
||||||
// de l'origine demandee : sans lui, un cache intermediaire (CDN, proxy)
|
|
||||||
// peut servir a une origine la reponse mise en cache pour une autre, ce qui
|
|
||||||
// revient a autoriser une origine qui ne l'est pas. Pose systematiquement,
|
|
||||||
// y compris quand l'origine est refusee -- sinon le refus lui-meme se
|
|
||||||
// retrouve mis en cache pour une origine legitime.
|
|
||||||
reply.header('Vary', 'Origin');
|
|
||||||
|
|
||||||
if (allowedOrigins.includes(origin)) {
|
if (allowedOrigins.includes(origin)) {
|
||||||
reply.header('Access-Control-Allow-Origin', origin);
|
reply.header('Access-Control-Allow-Origin', origin);
|
||||||
}
|
}
|
||||||
|
|
@ -292,23 +245,36 @@ export async function buildServer(opts = {}) {
|
||||||
year_max,
|
year_max,
|
||||||
} = /** @type {Record<string, string|undefined>} */ (req.query || {});
|
} = /** @type {Record<string, string|undefined>} */ (req.query || {});
|
||||||
|
|
||||||
// Validation déléguée à validate.js. Ces règles y étaient déjà écrites ET
|
if (!bbox || !zoom) {
|
||||||
// couvertes par 64 tests, mais la production les réimplémentait à la main
|
return reply.code(400).send({ ok: false, error: "bbox and zoom are required" });
|
||||||
// ici : les tests garantissaient donc une implémentation qui ne tournait
|
|
||||||
// nulle part. Une divergence existait déjà (parseMaId exige un entier, le
|
|
||||||
// code en ligne acceptait un flottant).
|
|
||||||
let minLon, minLat, maxLon, maxLat, zoomLevel, cellSize;
|
|
||||||
try {
|
|
||||||
({ minLon, minLat, maxLon, maxLat } = parseBbox(bbox));
|
|
||||||
zoomLevel = parseZoom(zoom);
|
|
||||||
cellSize = cellSizeForZoom(zoomLevel);
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof ValidationError) {
|
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
|
||||||
}
|
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const bboxParts = String(bbox).split(",").map(Number);
|
||||||
|
if (bboxParts.length !== 4) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "bbox must have 4 values" });
|
||||||
|
}
|
||||||
|
|
||||||
|
const [minLon, minLat, maxLon, maxLat] = bboxParts;
|
||||||
|
const zoomLevel = Number(zoom);
|
||||||
|
|
||||||
|
if ([minLon, minLat, maxLon, maxLat, zoomLevel].some(v => !Number.isFinite(v))) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "Invalid bbox or zoom values" });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (minLat < -90 || maxLat > 90 || minLon < -180 || maxLon > 180) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "Coordinates out of range" });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (minLon >= maxLon || minLat >= maxLat) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "Invalid bbox bounds" });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (zoomLevel < 0 || zoomLevel > 22) {
|
||||||
|
return reply.code(400).send({ ok: false, error: "Zoom level must be between 0 and 22" });
|
||||||
|
}
|
||||||
|
|
||||||
|
const cellSize = Math.max(0.01, 180 / Math.pow(2, zoomLevel));
|
||||||
|
|
||||||
// Source: band_locations (un point par localisation géocodée) JOIN bands.
|
// Source: band_locations (un point par localisation géocodée) JOIN bands.
|
||||||
const where = [
|
const where = [
|
||||||
"bl.geom IS NOT NULL",
|
"bl.geom IS NOT NULL",
|
||||||
|
|
@ -321,29 +287,28 @@ export async function buildServer(opts = {}) {
|
||||||
vals.push(minLon, minLat, maxLon, maxLat);
|
vals.push(minLon, minLat, maxLon, maxLat);
|
||||||
i += 4;
|
i += 4;
|
||||||
|
|
||||||
try {
|
if (countries) {
|
||||||
if (countries) {
|
const cs = String(countries).split(",").map(s => s.trim().toUpperCase()).filter(Boolean);
|
||||||
const cs = parseCsvList(countries, { max: 100, label: "pays", transform: (x) => x.toUpperCase() });
|
if (cs.length > 100) {
|
||||||
if (cs.length) {
|
return reply.code(400).send({ ok: false, error: "Max 100 pays simultanés" });
|
||||||
where.push(`b.country = ANY($${i}::text[])`);
|
|
||||||
vals.push(cs);
|
|
||||||
i++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
if (cs.length) {
|
||||||
|
where.push(`b.country = ANY($${i}::text[])`);
|
||||||
|
vals.push(cs);
|
||||||
|
i++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (status) {
|
if (status) {
|
||||||
const st = parseCsvList(status, { max: 50, label: "statuts" });
|
const st = String(status).split(",").map(s => s.trim()).filter(Boolean);
|
||||||
if (st.length) {
|
if (st.length > 50) {
|
||||||
where.push(`COALESCE(NULLIF(trim(b.status), ''), 'Unknown') = ANY($${i}::text[])`);
|
return reply.code(400).send({ ok: false, error: "Max 50 statuts simultanés" });
|
||||||
vals.push(st);
|
|
||||||
i++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
} catch (err) {
|
if (st.length) {
|
||||||
if (err instanceof ValidationError) {
|
where.push(`COALESCE(NULLIF(trim(b.status), ''), 'Unknown') = ANY($${i}::text[])`);
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
vals.push(st);
|
||||||
|
i++;
|
||||||
}
|
}
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (genre) {
|
if (genre) {
|
||||||
|
|
@ -357,24 +322,24 @@ export async function buildServer(opts = {}) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
try {
|
if (year_min) {
|
||||||
const yearMin = parseYear(year_min, "year_min");
|
const yearMin = Number(year_min);
|
||||||
if (yearMin !== null) {
|
if (!Number.isFinite(yearMin) || yearMin < 1800 || yearMin > 2100) {
|
||||||
where.push(`b.formed_year >= $${i}`);
|
return reply.code(400).send({ ok: false, error: 'year_min invalide' });
|
||||||
vals.push(yearMin);
|
|
||||||
i++;
|
|
||||||
}
|
}
|
||||||
const yearMax = parseYear(year_max, "year_max");
|
where.push(`b.formed_year >= $${i}`);
|
||||||
if (yearMax !== null) {
|
vals.push(yearMin);
|
||||||
where.push(`b.formed_year <= $${i}`);
|
i++;
|
||||||
vals.push(yearMax);
|
}
|
||||||
i++;
|
|
||||||
|
if (year_max) {
|
||||||
|
const yearMax = Number(year_max);
|
||||||
|
if (!Number.isFinite(yearMax) || yearMax < 1800 || yearMax > 2100) {
|
||||||
|
return reply.code(400).send({ ok: false, error: 'year_max invalide' });
|
||||||
}
|
}
|
||||||
} catch (err) {
|
where.push(`b.formed_year <= $${i}`);
|
||||||
if (err instanceof ValidationError) {
|
vals.push(yearMax);
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
i++;
|
||||||
}
|
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const whereSql = where.join(" AND ");
|
const whereSql = where.join(" AND ");
|
||||||
|
|
@ -399,20 +364,14 @@ export async function buildServer(opts = {}) {
|
||||||
JOIN bands b ON b.ma_id = bl.ma_id
|
JOIN bands b ON b.ma_id = bl.ma_id
|
||||||
WHERE ${whereSql}
|
WHERE ${whereSql}
|
||||||
ORDER BY bl.ma_id ASC, bl.step_order ASC
|
ORDER BY bl.ma_id ASC, bl.step_order ASC
|
||||||
LIMIT ${BANDS_AT_ZOOM_LIMIT};
|
LIMIT 2000;
|
||||||
`;
|
`;
|
||||||
const r = await p.query(sql, vals);
|
const r = await p.query(sql, vals);
|
||||||
// La réponse était plafonnée sans le dire : dans une zone dense, des
|
|
||||||
// groupes disparaissaient de la carte sans aucun signal, ni côté API ni
|
|
||||||
// côté client. `truncated` permet enfin à l'interface d'inviter à
|
|
||||||
// zoomer au lieu de mentir par omission.
|
|
||||||
return {
|
return {
|
||||||
ok: true,
|
ok: true,
|
||||||
type: "bands",
|
type: "bands",
|
||||||
items: r.rows,
|
items: r.rows,
|
||||||
count: r.rows.length,
|
count: r.rows.length
|
||||||
limit: BANDS_AT_ZOOM_LIMIT,
|
|
||||||
truncated: r.rows.length >= BANDS_AT_ZOOM_LIMIT
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -469,7 +428,7 @@ export async function buildServer(opts = {}) {
|
||||||
END as sample_bands
|
END as sample_bands
|
||||||
FROM clusters
|
FROM clusters
|
||||||
ORDER BY band_count DESC
|
ORDER BY band_count DESC
|
||||||
LIMIT ${CLUSTERS_LIMIT};
|
LIMIT 1000;
|
||||||
`;
|
`;
|
||||||
vals.push(cellSize);
|
vals.push(cellSize);
|
||||||
|
|
||||||
|
|
@ -479,8 +438,6 @@ export async function buildServer(opts = {}) {
|
||||||
type: "clusters",
|
type: "clusters",
|
||||||
items: r.rows,
|
items: r.rows,
|
||||||
total_clusters: r.rows.length,
|
total_clusters: r.rows.length,
|
||||||
limit: CLUSTERS_LIMIT,
|
|
||||||
truncated: r.rows.length >= CLUSTERS_LIMIT,
|
|
||||||
cell_size: cellSize,
|
cell_size: cellSize,
|
||||||
zoom: zoomLevel
|
zoom: zoomLevel
|
||||||
};
|
};
|
||||||
|
|
@ -524,39 +481,31 @@ export async function buildServer(opts = {}) {
|
||||||
const vals = [];
|
const vals = [];
|
||||||
let i = 1;
|
let i = 1;
|
||||||
|
|
||||||
try {
|
if (countries) {
|
||||||
if (countries) {
|
const cs = String(countries).split(",").map(s => s.trim().toUpperCase()).filter(Boolean);
|
||||||
const cs = parseCsvList(countries, { max: 100, label: "pays", transform: (x) => x.toUpperCase() });
|
if (cs.length > 100) {
|
||||||
if (cs.length) {
|
return reply.code(400).send({ ok: false, error: "Max 100 pays simultanés" });
|
||||||
where.push(`country = ANY($${i}::text[])`);
|
|
||||||
vals.push(cs);
|
|
||||||
i++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
} catch (err) {
|
if (cs.length) {
|
||||||
if (err instanceof ValidationError) {
|
where.push(`country = ANY($${i}::text[])`);
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
vals.push(cs);
|
||||||
|
i++;
|
||||||
}
|
}
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (geocoded === "1") where.push(`geom IS NOT NULL`);
|
if (geocoded === "1") where.push(`geom IS NOT NULL`);
|
||||||
if (geocoded === "0") where.push(`geom IS NULL`);
|
if (geocoded === "0") where.push(`geom IS NULL`);
|
||||||
|
|
||||||
try {
|
if (status) {
|
||||||
if (status) {
|
const st = String(status).split(",").map(s => s.trim()).filter(Boolean);
|
||||||
const st = parseCsvList(status, { max: 50, label: "statuts" });
|
if (st.length > 50) {
|
||||||
if (st.length) {
|
return reply.code(400).send({ ok: false, error: "Max 50 statuts simultanés" });
|
||||||
where.push(`COALESCE(NULLIF(trim(status), ''), 'Unknown') = ANY($${i}::text[])`);
|
|
||||||
vals.push(st);
|
|
||||||
i++;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
} catch (err) {
|
if (st.length) {
|
||||||
if (err instanceof ValidationError) {
|
where.push(`COALESCE(NULLIF(trim(status), ''), 'Unknown') = ANY($${i}::text[])`);
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
vals.push(st);
|
||||||
|
i++;
|
||||||
}
|
}
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (only_black === "1") {
|
if (only_black === "1") {
|
||||||
|
|
@ -673,18 +622,10 @@ export async function buildServer(opts = {}) {
|
||||||
try {
|
try {
|
||||||
const p = requirePool();
|
const p = requirePool();
|
||||||
const { ma_id } = /** @type {{ ma_id: string }} */ (req.params);
|
const { ma_id } = /** @type {{ ma_id: string }} */ (req.params);
|
||||||
let id;
|
const id = Number(ma_id);
|
||||||
try {
|
|
||||||
// parseMaId exige un ENTIER. Le contrôle en ligne se contentait de
|
if (!Number.isFinite(id) || id < 0) {
|
||||||
// Number.isFinite : /api/band/1.5 passait la validation et partait en
|
return reply.code(400).send({ ok: false, error: "bad ma_id" });
|
||||||
// base pour ne rien trouver. C'est la divergence que la duplication
|
|
||||||
// avait laissée s'installer.
|
|
||||||
id = parseMaId(ma_id);
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof ValidationError) {
|
|
||||||
return reply.code(400).send({ ok: false, error: err.message });
|
|
||||||
}
|
|
||||||
throw err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
const r = await p.query(
|
const r = await p.query(
|
||||||
|
|
@ -723,14 +664,9 @@ export async function buildServer(opts = {}) {
|
||||||
await adminRoutes.register(rateLimit, {
|
await adminRoutes.register(rateLimit, {
|
||||||
max: adminRateLimitMax,
|
max: adminRateLimitMax,
|
||||||
timeWindow: '1 minute',
|
timeWindow: '1 minute',
|
||||||
// La cle DOIT etre l'IP, jamais le jeton fourni. Keyer sur le Bearer
|
keyGenerator: (req) => {
|
||||||
// donnait a l'appelant le controle de son propre compteur : il suffisait
|
return authBearer(req) || req.ip;
|
||||||
// d'envoyer un jeton different a chaque requete pour obtenir un seau
|
}
|
||||||
// neuf a chaque fois et contourner entierement le plafond -- or ce
|
|
||||||
// plafond existe precisement pour ralentir la recherche du jeton par
|
|
||||||
// force brute. En prime, chaque jeton inedit consommait une entree du
|
|
||||||
// cache LRU du limiteur, evincant les compteurs legitimes.
|
|
||||||
keyGenerator: (req) => req.ip,
|
|
||||||
});
|
});
|
||||||
|
|
||||||
adminRoutes.post("/admin/import", async (req, reply) => {
|
adminRoutes.post("/admin/import", async (req, reply) => {
|
||||||
|
|
|
||||||
|
|
@ -78,20 +78,6 @@ async function run() {
|
||||||
}
|
}
|
||||||
|
|
||||||
if (count === 0) console.log('[migrate] nothing to apply');
|
if (count === 0) console.log('[migrate] nothing to apply');
|
||||||
|
|
||||||
// Purge de rétention (migration 019). Bornée par des index sur created_at, et
|
|
||||||
// volontairement tolérante : ce nettoyage ne doit jamais empêcher l'API de
|
|
||||||
// démarrer. Il n'y a pas d'ordonnanceur ici, et ce script tourne au démarrage
|
|
||||||
// de chaque conteneur — c'est le seul point d'accroche périodique disponible.
|
|
||||||
try {
|
|
||||||
const { rows: purged } = await client.query('SELECT * FROM purge_historique()');
|
|
||||||
const total = purged.reduce((n, r) => n + Number(r.lignes), 0);
|
|
||||||
if (total > 0) {
|
|
||||||
console.log('[migrate] purge: ' + purged.map(r => `${r.table_purgee}=${r.lignes}`).join(' '));
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
console.warn(`[migrate] purge ignorée: ${err.message}`);
|
|
||||||
}
|
|
||||||
// Relâché explicitement : la fermeture suffirait, mais l'expliciter rend le
|
// Relâché explicitement : la fermeture suffirait, mais l'expliciter rend le
|
||||||
// verrou visible dans les logs et évite de le garder si end() traîne.
|
// verrou visible dans les logs et évite de le garder si end() traîne.
|
||||||
await client.query('SELECT pg_advisory_unlock($1)', [MIGRATION_LOCK_ID]);
|
await client.query('SELECT pg_advisory_unlock($1)', [MIGRATION_LOCK_ID]);
|
||||||
|
|
|
||||||
|
|
@ -9,10 +9,6 @@ const {
|
||||||
verifyPassword,
|
verifyPassword,
|
||||||
requireAdminSession,
|
requireAdminSession,
|
||||||
ADMIN_COOKIE_NAME,
|
ADMIN_COOKIE_NAME,
|
||||||
seedAdminUser,
|
|
||||||
recordLoginAttempt,
|
|
||||||
markLoginSuccess,
|
|
||||||
writeAuditLog,
|
|
||||||
} = await import("../src/adminAuth.js");
|
} = await import("../src/adminAuth.js");
|
||||||
const { makeFakePool, rows } = await import("./helpers/fakePool.js");
|
const { makeFakePool, rows } = await import("./helpers/fakePool.js");
|
||||||
|
|
||||||
|
|
@ -119,98 +115,18 @@ describe("verifyPassword", () => {
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("isLockedOut", () => {
|
describe("isLockedOut", () => {
|
||||||
const compteurs = (par_ip, par_username) =>
|
it("verrouille à partir de 5 tentatives échouées", async () => {
|
||||||
makeFakePool([{ match: "admin_login_attempts", result: rows({ par_ip, par_username }) }]);
|
const under = makeFakePool([{ match: "admin_login_attempts", result: rows({ n: 4 }) }]);
|
||||||
|
const at = makeFakePool([{ match: "admin_login_attempts", result: rows({ n: 5 }) }]);
|
||||||
it("verrouille à partir de 5 échecs venant de la même IP", async () => {
|
expect(await isLockedOut(under, "nico", "1.2.3.4")).toBe(false);
|
||||||
expect(await isLockedOut(compteurs(4, 4), "nico", "1.2.3.4")).toBe(false);
|
expect(await isLockedOut(at, "nico", "1.2.3.4")).toBe(true);
|
||||||
expect(await isLockedOut(compteurs(5, 5), "nico", "1.2.3.4")).toBe(true);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("un tiers ne peut pas verrouiller un compte en échouant depuis ailleurs", async () => {
|
it("compte les tentatives par username OU par IP, sur 15 minutes", async () => {
|
||||||
// Le coeur du correctif : les deux compteurs etaient additionnes dans un
|
const pool = makeFakePool([{ match: "admin_login_attempts", result: rows({ n: 0 }) }]);
|
||||||
// seul total. Cinq echecs avec le nom d'un admin, depuis n'importe quelle
|
|
||||||
// adresse, le mettaient dehors un quart d'heure — a repeter indefiniment.
|
|
||||||
// L'admin legitime, lui, n'a aucun echec a son IP : il doit passer.
|
|
||||||
expect(await isLockedOut(compteurs(0, 19), "nico", "1.2.3.4")).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("un seuil par username subsiste contre une attaque répartie", async () => {
|
|
||||||
expect(await isLockedOut(compteurs(0, 20), "nico", "1.2.3.4")).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("compte sur une fenêtre de 15 minutes", async () => {
|
|
||||||
const pool = compteurs(0, 0);
|
|
||||||
await isLockedOut(pool, "nico", "1.2.3.4");
|
await isLockedOut(pool, "nico", "1.2.3.4");
|
||||||
const call = pool.find("admin_login_attempts");
|
const call = pool.find("admin_login_attempts");
|
||||||
expect(call.sql).toMatch(/FILTER \(WHERE ip = \$3\)/);
|
expect(call.sql).toMatch(/username = \$2 OR ip = \$3/);
|
||||||
expect(call.sql).toMatch(/FILTER \(WHERE username = \$2\)/);
|
|
||||||
expect(call.values).toEqual([15, "nico", "1.2.3.4"]);
|
expect(call.values).toEqual([15, "nico", "1.2.3.4"]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
/**
|
|
||||||
* Les quatre fonctions ci-dessous n'avaient AUCUN test direct — elles portaient
|
|
||||||
* l'essentiel des 22 mutants sans couverture du fichier (score 52 %, le plus
|
|
||||||
* bas du dépôt, sur le module qui gère JWT, bcrypt et le verrouillage).
|
|
||||||
*
|
|
||||||
* writeAuditLog est le cas le plus gênant : il peut cesser d'enregistrer sans
|
|
||||||
* qu'aucune action admin n'échoue et sans qu'aucun test ne rougisse. Un journal
|
|
||||||
* d'audit muet ne se remarque que le jour où on en a besoin.
|
|
||||||
*/
|
|
||||||
describe("écritures non couvertes jusqu'ici", () => {
|
|
||||||
it("seedAdminUser ne fait rien sans identifiants", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
delete process.env.ADMIN_SEED_USERNAME;
|
|
||||||
delete process.env.ADMIN_SEED_PASSWORD_HASH;
|
|
||||||
await seedAdminUser(pool);
|
|
||||||
expect(pool.calls).toHaveLength(0);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("seedAdminUser crée le compte sans écraser un existant", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
process.env.ADMIN_SEED_USERNAME = " nico ";
|
|
||||||
process.env.ADMIN_SEED_PASSWORD_HASH = " $2b$12$hash ";
|
|
||||||
await seedAdminUser(pool);
|
|
||||||
const call = pool.find("INSERT INTO admin_users");
|
|
||||||
expect(call.values).toEqual(["nico", "$2b$12$hash"]);
|
|
||||||
// Sans ON CONFLICT DO NOTHING, un redémarrage écraserait le mot de passe.
|
|
||||||
expect(call.sql).toMatch(/ON CONFLICT \(username\) DO NOTHING/);
|
|
||||||
delete process.env.ADMIN_SEED_USERNAME;
|
|
||||||
delete process.env.ADMIN_SEED_PASSWORD_HASH;
|
|
||||||
});
|
|
||||||
|
|
||||||
it("recordLoginAttempt enregistre l'issue réelle", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
await recordLoginAttempt(pool, "nico", "1.2.3.4", false);
|
|
||||||
expect(pool.find("INSERT INTO admin_login_attempts").values).toEqual(["nico", "1.2.3.4", false]);
|
|
||||||
await recordLoginAttempt(pool, "nico", "1.2.3.4", true);
|
|
||||||
expect(pool.findAll("INSERT INTO admin_login_attempts")[1].values[2]).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("markLoginSuccess horodate le bon compte", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
await markLoginSuccess(pool, "nico");
|
|
||||||
const call = pool.find("UPDATE admin_users");
|
|
||||||
expect(call.values).toEqual(["nico"]);
|
|
||||||
expect(call.sql).toMatch(/last_login_at = now\(\)/);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("writeAuditLog sérialise avant/après en JSON", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
await writeAuditLog(pool, "nico", "update", "bands", 42, { a: 1 }, { a: 2 });
|
|
||||||
const call = pool.find("INSERT INTO admin_audit_log");
|
|
||||||
expect(call.values[0]).toBe("nico");
|
|
||||||
expect(call.values[3]).toBe("42"); // target_id est du texte
|
|
||||||
expect(JSON.parse(call.values[4])).toEqual({ a: 1 });
|
|
||||||
expect(JSON.parse(call.values[5])).toEqual({ a: 2 });
|
|
||||||
});
|
|
||||||
|
|
||||||
it("writeAuditLog accepte l'absence d'avant/après", async () => {
|
|
||||||
const pool = makeFakePool();
|
|
||||||
await writeAuditLog(pool, "nico", "cleanup", "crawl_run", null, null, null);
|
|
||||||
const call = pool.find("INSERT INTO admin_audit_log");
|
|
||||||
expect(call.values[4]).toBeNull();
|
|
||||||
expect(call.values[5]).toBeNull();
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
|
||||||
|
|
@ -34,6 +34,7 @@ describe("routes de lecture", () => {
|
||||||
const READ_ROUTES = [
|
const READ_ROUTES = [
|
||||||
"/admin/api/stats",
|
"/admin/api/stats",
|
||||||
"/admin/api/queue",
|
"/admin/api/queue",
|
||||||
|
"/admin/api/crawl-checkpoints",
|
||||||
"/admin/api/logs",
|
"/admin/api/logs",
|
||||||
"/admin/api/geocoding",
|
"/admin/api/geocoding",
|
||||||
"/admin/api/llm",
|
"/admin/api/llm",
|
||||||
|
|
@ -367,23 +368,11 @@ describe("POST /admin/api/crawl-runs/cleanup", () => {
|
||||||
{ match: "INSERT INTO admin_audit_log", result: rows() },
|
{ match: "INSERT INTO admin_audit_log", result: rows() },
|
||||||
]);
|
]);
|
||||||
|
|
||||||
it("le seuil par défaut dépasse la durée d'un crawl complet", async () => {
|
it("utilise 30 minutes par défaut", async () => {
|
||||||
// 30 minutes était plus court qu'un crawl complet Europe, qui dure des
|
|
||||||
// heures : le nettoyage marquait alors en erreur un run parfaitement vivant,
|
|
||||||
// et update_crawl_run_progress (filtré sur status='running') cessait de
|
|
||||||
// publier — l'affichage restait figé jusqu'à la fin du run.
|
|
||||||
const handlers = cleanupPool();
|
const handlers = cleanupPool();
|
||||||
const app = buildApp(handlers);
|
const app = buildApp(handlers);
|
||||||
await app.inject({ method: "POST", url: "/admin/api/crawl-runs/cleanup", headers: auth, payload: {} });
|
await app.inject({ method: "POST", url: "/admin/api/crawl-runs/cleanup", headers: auth, payload: {} });
|
||||||
expect(pool.find("UPDATE crawl_run").values).toEqual([24 * 60]);
|
expect(pool.find("UPDATE crawl_run").values).toEqual([30]);
|
||||||
});
|
|
||||||
|
|
||||||
it("laisse tranquille un run dont l'annulation est déjà demandée", async () => {
|
|
||||||
// L'annulation est coopérative : le crawler écrira lui-même 'cancelled'.
|
|
||||||
const handlers = cleanupPool();
|
|
||||||
const app = buildApp(handlers);
|
|
||||||
await app.inject({ method: "POST", url: "/admin/api/crawl-runs/cleanup", headers: auth, payload: {} });
|
|
||||||
expect(pool.find("UPDATE crawl_run").sql).toMatch(/cancel_requested = FALSE/);
|
|
||||||
});
|
});
|
||||||
|
|
||||||
it("ne touche que les runs marqués running", async () => {
|
it("ne touche que les runs marqués running", async () => {
|
||||||
|
|
|
||||||
|
|
@ -137,58 +137,3 @@ describe("migrate.js", () => {
|
||||||
}
|
}
|
||||||
}, 60000);
|
}, 60000);
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("triggers de bands — comportement réel", () => {
|
|
||||||
beforeAll(async () => {
|
|
||||||
await migrate();
|
|
||||||
await client.query("DELETE FROM band_locations");
|
|
||||||
await client.query("DELETE FROM bands");
|
|
||||||
}, 120000);
|
|
||||||
|
|
||||||
it("updated_at ne bouge pas sur un upsert sans changement réel", async () => {
|
|
||||||
// Le crawler fait un ON CONFLICT DO UPDATE sans WHERE : Postgres exécute
|
|
||||||
// l'UPDATE pour chaque ligne vue, même identique. Quand le trigger bumpait
|
|
||||||
// updated_at sans condition, get_bands_to_enrich considérait toute la table
|
|
||||||
// comme « à ré-enrichir » après chaque crawl (migration 016).
|
|
||||||
await client.query(
|
|
||||||
`INSERT INTO bands (ma_id, name, country, location_text) VALUES (901, 'A', 'FR', 'Paris')`
|
|
||||||
);
|
|
||||||
const avant = (await client.query(`SELECT updated_at FROM bands WHERE ma_id=901`)).rows[0].updated_at;
|
|
||||||
|
|
||||||
await client.query(`
|
|
||||||
INSERT INTO bands (ma_id, name, country, location_text) VALUES (901, 'A', 'FR', 'Paris')
|
|
||||||
ON CONFLICT (ma_id) DO UPDATE SET
|
|
||||||
name = COALESCE(EXCLUDED.name, bands.name),
|
|
||||||
country = COALESCE(EXCLUDED.country, bands.country),
|
|
||||||
location_text = COALESCE(EXCLUDED.location_text, bands.location_text)
|
|
||||||
`);
|
|
||||||
const apres = (await client.query(`SELECT updated_at FROM bands WHERE ma_id=901`)).rows[0].updated_at;
|
|
||||||
expect(apres).toEqual(avant);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("updated_at bouge sur un changement réel", async () => {
|
|
||||||
const avant = (await client.query(`SELECT updated_at FROM bands WHERE ma_id=901`)).rows[0].updated_at;
|
|
||||||
await client.query(`UPDATE bands SET genre='Black Metal' WHERE ma_id=901`);
|
|
||||||
const apres = (await client.query(`SELECT updated_at FROM bands WHERE ma_id=901`)).rows[0].updated_at;
|
|
||||||
expect(apres.getTime()).toBeGreaterThan(avant.getTime());
|
|
||||||
});
|
|
||||||
|
|
||||||
it("changer le lieu efface les points automatiques mais épargne la saisie admin", async () => {
|
|
||||||
// Une correction manuelle (provider='admin', confiance 1.0) ne peut pas être
|
|
||||||
// régénérée : le trigger de nettoyage ne doit pas l'emporter au passage
|
|
||||||
// (migration 018).
|
|
||||||
await client.query(`
|
|
||||||
INSERT INTO band_locations
|
|
||||||
(ma_id, step_order, location_raw, is_country_only, lat, lon, geocode_status, geocode_provider, geocode_confidence)
|
|
||||||
VALUES (901, 0, 'Paris', FALSE, 48.85, 2.35, 'done', 'geoapify', 0.9),
|
|
||||||
(901, -1, 'override', FALSE, 48.86, 2.34, 'done', 'admin', 1.0)
|
|
||||||
`);
|
|
||||||
|
|
||||||
await client.query(`UPDATE bands SET location_text='Lyon' WHERE ma_id=901`);
|
|
||||||
|
|
||||||
const r = await client.query(
|
|
||||||
`SELECT geocode_provider FROM band_locations WHERE ma_id=901 ORDER BY geocode_provider`
|
|
||||||
);
|
|
||||||
expect(r.rows.map((x) => x.geocode_provider)).toEqual(["admin"]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
|
||||||
|
|
@ -139,7 +139,7 @@ describe.runIf(!process.env.SKIP_INTEGRATION)("SQL validé par PostgreSQL", () =
|
||||||
const ADMIN_GET = [
|
const ADMIN_GET = [
|
||||||
"/admin/api/stats", "/admin/api/queue", "/admin/api/bands",
|
"/admin/api/stats", "/admin/api/queue", "/admin/api/bands",
|
||||||
"/admin/api/bands?q=mayhem&country=NO&genre=black&location_q=oslo&themes_q=war&enriched=true&has_lat=false&has_location=true&has_conflict=true&sort=name&dir=desc",
|
"/admin/api/bands?q=mayhem&country=NO&genre=black&location_q=oslo&themes_q=war&enriched=true&has_lat=false&has_location=true&has_conflict=true&sort=name&dir=desc",
|
||||||
"/admin/api/bands/1", "/admin/api/logs",
|
"/admin/api/bands/1", "/admin/api/crawl-checkpoints", "/admin/api/logs",
|
||||||
"/admin/api/logs?level=error&run_id=1&min_id=5", "/admin/api/geocoding",
|
"/admin/api/logs?level=error&run_id=1&min_id=5", "/admin/api/geocoding",
|
||||||
"/admin/api/llm", "/admin/api/llm?model=x&only_null=1&q=oslo",
|
"/admin/api/llm", "/admin/api/llm?model=x&only_null=1&q=oslo",
|
||||||
"/admin/api/job-triggers", "/admin/api/live",
|
"/admin/api/job-triggers", "/admin/api/live",
|
||||||
|
|
|
||||||
|
|
@ -58,25 +58,6 @@ describe("/api/health et /", () => {
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("/api/clusters — troncature", () => {
|
|
||||||
it("signale la troncature quand le plafond est atteint", async () => {
|
|
||||||
// La réponse était plafonnée sans le dire : dans une zone dense, des
|
|
||||||
// groupes disparaissaient de la carte sans aucun signal.
|
|
||||||
const pleine = Array.from({ length: 2000 }, (_, i) => ({ ma_id: i }));
|
|
||||||
const res = await build([{ match: "FROM band_locations", result: { rows: pleine, rowCount: 2000 } }])
|
|
||||||
.inject({ method: "GET", url: "/api/clusters?bbox=0,0,10,10&zoom=13" });
|
|
||||||
const body = JSON.parse(res.payload);
|
|
||||||
expect(body.truncated).toBe(true);
|
|
||||||
expect(body.limit).toBe(2000);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("ne signale rien quand tout tient", async () => {
|
|
||||||
const res = await build([{ match: "FROM band_locations", result: { rows: [{ ma_id: 1 }], rowCount: 1 } }])
|
|
||||||
.inject({ method: "GET", url: "/api/clusters?bbox=0,0,10,10&zoom=13" });
|
|
||||||
expect(JSON.parse(res.payload).truncated).toBe(false);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
||||||
describe("CORS", () => {
|
describe("CORS", () => {
|
||||||
it("renvoie l'en-tête pour une origine autorisée", async () => {
|
it("renvoie l'en-tête pour une origine autorisée", async () => {
|
||||||
const res = await build().inject({
|
const res = await build().inject({
|
||||||
|
|
@ -92,19 +73,6 @@ describe("CORS", () => {
|
||||||
expect(res.headers["access-control-allow-origin"]).toBeUndefined();
|
expect(res.headers["access-control-allow-origin"]).toBeUndefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("annonce Vary: Origin, y compris quand l'origine est refusée", async () => {
|
|
||||||
// L'en-tete Allow-Origin DEPEND de l'origine demandee. Sans Vary, un cache
|
|
||||||
// intermediaire peut servir a une origine la reponse mise en cache pour une
|
|
||||||
// autre — ce qui revient a autoriser une origine qui ne l'est pas, ou a
|
|
||||||
// faire refuser une origine legitime.
|
|
||||||
for (const origin of ["https://metalfrom.eu", "https://evil.example", undefined]) {
|
|
||||||
const res = await build().inject({
|
|
||||||
method: "GET", url: "/api/health", headers: origin ? { origin } : {},
|
|
||||||
});
|
|
||||||
expect(res.headers["vary"]).toContain("Origin");
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
it("ne fait pas de match par préfixe sur l'origine", async () => {
|
it("ne fait pas de match par préfixe sur l'origine", async () => {
|
||||||
const res = await build().inject({
|
const res = await build().inject({
|
||||||
method: "GET", url: "/api/health", headers: { origin: "https://metalfrom.eu.evil.example" },
|
method: "GET", url: "/api/health", headers: { origin: "https://metalfrom.eu.evil.example" },
|
||||||
|
|
@ -369,7 +337,7 @@ describe("/admin/auth/login", () => {
|
||||||
|
|
||||||
it("répond 429 quand le compte est verrouillé, sans vérifier le mot de passe", async () => {
|
it("répond 429 quand le compte est verrouillé, sans vérifier le mot de passe", async () => {
|
||||||
const res = await build([
|
const res = await build([
|
||||||
{ match: "FROM admin_login_attempts", result: rows({ par_ip: 99, par_username: 0 }) },
|
{ match: "admin_login_attempts", result: rows({ n: 99 }) },
|
||||||
{ match: "FROM admin_users", result: rows({ password_hash: "x" }) },
|
{ match: "FROM admin_users", result: rows({ password_hash: "x" }) },
|
||||||
]).inject({
|
]).inject({
|
||||||
method: "POST", url: "/admin/auth/login",
|
method: "POST", url: "/admin/auth/login",
|
||||||
|
|
@ -384,7 +352,7 @@ describe("/admin/auth/login", () => {
|
||||||
// Hash à coût 4 : le DUMMY_HASH de production est à coût 12 (~330 ms).
|
// Hash à coût 4 : le DUMMY_HASH de production est à coût 12 (~330 ms).
|
||||||
const hash = bcrypt.hashSync("autre-chose", 4);
|
const hash = bcrypt.hashSync("autre-chose", 4);
|
||||||
const res = await build([
|
const res = await build([
|
||||||
{ match: "FROM admin_login_attempts", result: rows({ par_ip: 0, par_username: 0 }) },
|
{ match: "SELECT count(*)::int AS n", result: rows({ n: 0 }) },
|
||||||
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
||||||
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
||||||
]).inject({
|
]).inject({
|
||||||
|
|
@ -400,7 +368,7 @@ describe("/admin/auth/login", () => {
|
||||||
const bcrypt = (await import("bcryptjs")).default;
|
const bcrypt = (await import("bcryptjs")).default;
|
||||||
const hash = bcrypt.hashSync("bon", 4);
|
const hash = bcrypt.hashSync("bon", 4);
|
||||||
const res = await build([
|
const res = await build([
|
||||||
{ match: "FROM admin_login_attempts", result: rows({ par_ip: 0, par_username: 0 }) },
|
{ match: "SELECT count(*)::int AS n", result: rows({ n: 0 }) },
|
||||||
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
||||||
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
||||||
{ match: "UPDATE admin_users", result: rows() },
|
{ match: "UPDATE admin_users", result: rows() },
|
||||||
|
|
@ -421,7 +389,7 @@ describe("/admin/auth/login", () => {
|
||||||
const bcrypt = (await import("bcryptjs")).default;
|
const bcrypt = (await import("bcryptjs")).default;
|
||||||
const hash = bcrypt.hashSync("x", 4);
|
const hash = bcrypt.hashSync("x", 4);
|
||||||
await build([
|
await build([
|
||||||
{ match: "FROM admin_login_attempts", result: rows({ par_ip: 0, par_username: 0 }) },
|
{ match: "SELECT count(*)::int AS n", result: rows({ n: 0 }) },
|
||||||
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
{ match: "FROM admin_users", result: rows({ password_hash: hash }) },
|
||||||
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
{ match: "INSERT INTO admin_login_attempts", result: rows() },
|
||||||
]).inject({
|
]).inject({
|
||||||
|
|
|
||||||
|
|
@ -64,24 +64,21 @@ describe("plafond de /admin/import", () => {
|
||||||
await app.close();
|
await app.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("limite par IP, y compris pour un appelant authentifié", async () => {
|
it("limite par jeton, pas globalement", async () => {
|
||||||
// Ce test verrouillait auparavant l'inverse — un compteur par jeton, pour
|
|
||||||
// qu'un client bruyant n'affame pas les autres. L'intention se défend, mais
|
|
||||||
// le limiteur s'exécute AVANT l'authentification : la cle etait donc une
|
|
||||||
// valeur non verifiee, fournie par l'appelant. Isoler par identite reelle
|
|
||||||
// est impossible a cette couche ; on retombe sur l'IP, seule cle que
|
|
||||||
// l'appelant ne choisit pas.
|
|
||||||
const app = await appWithRealLimits({ adminRateLimitMax: 2 });
|
const app = await appWithRealLimits({ adminRateLimitMax: 2 });
|
||||||
const call = (token) =>
|
const call = (token) =>
|
||||||
app.inject({
|
app.inject({
|
||||||
method: "POST", url: "/admin/import",
|
method: "POST", url: "/admin/import",
|
||||||
headers: { "x-forwarded-for": "4.4.4.4", authorization: `Bearer ${token}` },
|
headers: { authorization: `Bearer ${token}` }, payload: { bands: [] },
|
||||||
payload: { bands: [] },
|
|
||||||
});
|
});
|
||||||
|
|
||||||
await call(TOKEN);
|
await call(TOKEN);
|
||||||
await call(TOKEN);
|
await call(TOKEN);
|
||||||
|
// Le 3e appel avec CE jeton est bloqué…
|
||||||
expect((await call(TOKEN)).statusCode).toBe(429);
|
expect((await call(TOKEN)).statusCode).toBe(429);
|
||||||
|
// …mais un autre jeton dispose de son propre compteur (il sera rejeté en
|
||||||
|
// 401, pas en 429 : c'est bien l'authentification qui tranche, pas le débit).
|
||||||
|
expect((await call("un-autre-jeton-de-la-meme-taille")).statusCode).toBe(401);
|
||||||
await app.close();
|
await app.close();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
@ -110,28 +107,6 @@ describe("plafond de /admin/auth/login", () => {
|
||||||
await app.close();
|
await app.close();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("le plafond d'import ne se contourne pas en changeant de jeton", async () => {
|
|
||||||
// Le limiteur etait keye sur le Bearer fourni, c'est-a-dire sur une valeur
|
|
||||||
// entierement controlee par l'appelant : envoyer un jeton different a
|
|
||||||
// chaque requete donnait un seau neuf a chaque fois, et le plafond ne
|
|
||||||
// freinait plus rien -- alors qu'il est justement la pour ralentir la
|
|
||||||
// recherche du jeton par force brute.
|
|
||||||
const app = await appWithRealLimits({ adminRateLimitMax: 2 });
|
|
||||||
const call = (token) =>
|
|
||||||
app.inject({
|
|
||||||
method: "POST",
|
|
||||||
url: "/admin/import",
|
|
||||||
headers: { "x-forwarded-for": "9.9.9.9", authorization: `Bearer ${token}` },
|
|
||||||
payload: { bands: [] },
|
|
||||||
});
|
|
||||||
|
|
||||||
expect((await call("faux-a")).statusCode).toBe(401);
|
|
||||||
expect((await call("faux-b")).statusCode).toBe(401);
|
|
||||||
// 3e requete depuis la meme IP : bloquee, quel que soit le jeton presente.
|
|
||||||
expect((await call("faux-c")).statusCode).toBe(429);
|
|
||||||
await app.close();
|
|
||||||
});
|
|
||||||
|
|
||||||
it("limite par IP", async () => {
|
it("limite par IP", async () => {
|
||||||
const app = await appWithRealLimits({ authRateLimitMax: 1 });
|
const app = await appWithRealLimits({ authRateLimitMax: 1 });
|
||||||
const call = (ip) =>
|
const call = (ip) =>
|
||||||
|
|
|
||||||
|
|
@ -106,6 +106,7 @@ describe("SQL des routes admin", () => {
|
||||||
{ url: "/admin/api/bands" },
|
{ url: "/admin/api/bands" },
|
||||||
{ url: "/admin/api/bands?q=mayhem&country=NO&genre=black&location_q=oslo&themes_q=war&enriched=true&has_lat=false&has_location=true&has_conflict=true&sort=name&dir=desc&page=2" },
|
{ url: "/admin/api/bands?q=mayhem&country=NO&genre=black&location_q=oslo&themes_q=war&enriched=true&has_lat=false&has_location=true&has_conflict=true&sort=name&dir=desc&page=2" },
|
||||||
{ url: "/admin/api/bands/1" },
|
{ url: "/admin/api/bands/1" },
|
||||||
|
{ url: "/admin/api/crawl-checkpoints" },
|
||||||
{ url: "/admin/api/logs" },
|
{ url: "/admin/api/logs" },
|
||||||
{ url: "/admin/api/logs?level=error&run_id=1&min_id=5&since=2026-01-01" },
|
{ url: "/admin/api/logs?level=error&run_id=1&min_id=5&since=2026-01-01" },
|
||||||
{ url: "/admin/api/geocoding" },
|
{ url: "/admin/api/geocoding" },
|
||||||
|
|
|
||||||
|
|
@ -1,39 +0,0 @@
|
||||||
import { describe, it, expect } from "vitest";
|
|
||||||
import fs from "node:fs";
|
|
||||||
import path from "node:path";
|
|
||||||
import { fileURLToPath } from "node:url";
|
|
||||||
|
|
||||||
/**
|
|
||||||
* validate.js ne doit pas redevenir du code mort bien testé.
|
|
||||||
*
|
|
||||||
* Huit de ses douze exports n'étaient appelés nulle part en production :
|
|
||||||
* app.js et adminRoutes.js réimplémentaient la même validation à la main, trois
|
|
||||||
* fois. Les 64 tests de validate.test.js et validate.property.test.js
|
|
||||||
* garantissaient donc une implémentation qui ne tournait nulle part — et une
|
|
||||||
* divergence s'était déjà installée sans que rien ne rougisse (parseMaId exige
|
|
||||||
* un entier, le contrôle en ligne acceptait un flottant, si bien que
|
|
||||||
* /api/band/1.5 était accepté).
|
|
||||||
*
|
|
||||||
* Ce test échoue si un helper cesse d'être branché. Il ne dit rien de sa
|
|
||||||
* qualité — c'est le rôle des deux autres fichiers — seulement qu'il est
|
|
||||||
* réellement sur le chemin d'exécution.
|
|
||||||
*/
|
|
||||||
const SRC = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "../src");
|
|
||||||
const lire = (f) => fs.readFileSync(path.join(SRC, f), "utf8");
|
|
||||||
|
|
||||||
const validate = lire("validate.js");
|
|
||||||
const consommateurs = ["app.js", "adminRoutes.js"].map(lire).join("\n");
|
|
||||||
|
|
||||||
const exports = [...validate.matchAll(/export\s+(?:async\s+)?(?:class|function)\s+(\w+)/g)]
|
|
||||||
.map((m) => m[1]);
|
|
||||||
|
|
||||||
describe("branchement de validate.js", () => {
|
|
||||||
it("expose au moins les douze helpers connus", () => {
|
|
||||||
expect(exports.length).toBeGreaterThanOrEqual(12);
|
|
||||||
});
|
|
||||||
|
|
||||||
it.each(exports)("%s est réellement utilisé en production", (nom) => {
|
|
||||||
const utilise = new RegExp(`\\b${nom}\\b`).test(consommateurs);
|
|
||||||
expect(utilise).toBe(true);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
@ -1,7 +0,0 @@
|
||||||
.env
|
|
||||||
*.env
|
|
||||||
__pycache__/
|
|
||||||
*.pyc
|
|
||||||
*.pyo
|
|
||||||
.git
|
|
||||||
.venv
|
|
||||||
|
|
@ -1,16 +1,9 @@
|
||||||
FROM python:3.12-slim
|
FROM python:3.12-slim
|
||||||
|
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY apps/crawler/requirements.txt .
|
COPY requirements.txt .
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
# Contexte de build = racine du depot, pour embarquer le module partage.
|
COPY src ./src
|
||||||
COPY libs/ ./libs/
|
|
||||||
COPY apps/crawler/src ./src
|
|
||||||
ENV PYTHONPATH=/app/libs
|
|
||||||
|
|
||||||
# Utilisateur non-root (le crawler parse du HTML tiers avec lxml)
|
|
||||||
RUN useradd -m -u 1000 crawler
|
|
||||||
USER crawler
|
|
||||||
|
|
||||||
CMD ["python", "-m", "src.main"]
|
CMD ["python", "-m", "src.main"]
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,6 @@
|
||||||
"""
|
"""
|
||||||
import json
|
import json
|
||||||
import logging
|
import logging
|
||||||
import re
|
|
||||||
import time
|
import time
|
||||||
from contextlib import contextmanager
|
from contextlib import contextmanager
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
@ -16,13 +15,6 @@ from .config import DATABASE_URL
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
# Types de job que CE process sait exécuter (voir main._check_job_triggers).
|
|
||||||
# 'geocoder_enqueue' en est volontairement absent : il est consommé par
|
|
||||||
# apps/geocoder/src/enqueue.py. Sans ce filtre, le crawler raflait ces triggers
|
|
||||||
# et les refermait en « job_type inconnu », selon lequel des deux daemons
|
|
||||||
# interrogeait la table en premier.
|
|
||||||
CRAWLER_JOB_TYPES = ["enrich", "incremental", "full_crawl"]
|
|
||||||
|
|
||||||
|
|
||||||
@contextmanager
|
@contextmanager
|
||||||
def get_conn():
|
def get_conn():
|
||||||
|
|
@ -70,22 +62,16 @@ def upsert_bands(bands: list[dict[str, Any]]) -> dict[str, int]:
|
||||||
b.get("crawled_hash"),
|
b.get("crawled_hash"),
|
||||||
b.get("ma_created_at"),
|
b.get("ma_created_at"),
|
||||||
b.get("ma_modified_at"),
|
b.get("ma_modified_at"),
|
||||||
b.get("ma_modified_seen"),
|
|
||||||
ts, # first_seen_at (ignoré si déjà set)
|
ts, # first_seen_at (ignoré si déjà set)
|
||||||
ts, # created_at (ignoré si déjà set)
|
ts, # created_at (ignoré si déjà set)
|
||||||
psycopg2.extras.Json(b.get("data") or {}),
|
psycopg2.extras.Json(b.get("data") or {}),
|
||||||
))
|
))
|
||||||
|
|
||||||
# enrich_pending : levé quand l'horodatage "modified" vu dans la liste MA
|
|
||||||
# change réellement (nouvelle modif côté MA). COALESCE ne peut pas servir
|
|
||||||
# ici, il faut comparer l'ancienne et la nouvelle valeur → CASE explicite.
|
|
||||||
# Toutes les expressions du SET voient la ligne AVANT update, donc l'ordre
|
|
||||||
# entre ma_modified_seen et enrich_pending n'a pas d'importance.
|
|
||||||
sql = """
|
sql = """
|
||||||
INSERT INTO bands
|
INSERT INTO bands
|
||||||
(ma_id, name, country, location_text, status, genre, formed_year,
|
(ma_id, name, country, location_text, status, genre, formed_year,
|
||||||
themes, enriched, crawled_at, crawled_hash, ma_created_at, ma_modified_at,
|
themes, enriched, crawled_at, crawled_hash, ma_created_at, ma_modified_at,
|
||||||
ma_modified_seen, first_seen_at, created_at, data)
|
first_seen_at, created_at, data)
|
||||||
VALUES %s
|
VALUES %s
|
||||||
ON CONFLICT (ma_id) DO UPDATE SET
|
ON CONFLICT (ma_id) DO UPDATE SET
|
||||||
name = COALESCE(EXCLUDED.name, bands.name),
|
name = COALESCE(EXCLUDED.name, bands.name),
|
||||||
|
|
@ -100,13 +86,6 @@ def upsert_bands(bands: list[dict[str, Any]]) -> dict[str, int]:
|
||||||
crawled_hash = COALESCE(EXCLUDED.crawled_hash, bands.crawled_hash),
|
crawled_hash = COALESCE(EXCLUDED.crawled_hash, bands.crawled_hash),
|
||||||
ma_created_at = COALESCE(EXCLUDED.ma_created_at, bands.ma_created_at),
|
ma_created_at = COALESCE(EXCLUDED.ma_created_at, bands.ma_created_at),
|
||||||
ma_modified_at= COALESCE(EXCLUDED.ma_modified_at, bands.ma_modified_at),
|
ma_modified_at= COALESCE(EXCLUDED.ma_modified_at, bands.ma_modified_at),
|
||||||
enrich_pending = CASE
|
|
||||||
WHEN EXCLUDED.ma_modified_seen IS NOT NULL
|
|
||||||
AND EXCLUDED.ma_modified_seen IS DISTINCT FROM bands.ma_modified_seen
|
|
||||||
THEN true
|
|
||||||
ELSE bands.enrich_pending
|
|
||||||
END,
|
|
||||||
ma_modified_seen = COALESCE(EXCLUDED.ma_modified_seen, bands.ma_modified_seen),
|
|
||||||
data = bands.data || EXCLUDED.data
|
data = bands.data || EXCLUDED.data
|
||||||
RETURNING (xmax = 0) AS was_inserted
|
RETURNING (xmax = 0) AS was_inserted
|
||||||
"""
|
"""
|
||||||
|
|
@ -168,8 +147,6 @@ def upsert_band_enriched(ma_id: int, data: dict[str, Any], html_hash: str) -> bo
|
||||||
"ma_created_at = COALESCE(%s, ma_created_at)",
|
"ma_created_at = COALESCE(%s, ma_created_at)",
|
||||||
"ma_modified_at = COALESCE(%s, ma_modified_at)",
|
"ma_modified_at = COALESCE(%s, ma_modified_at)",
|
||||||
"crawler_pending = %s::jsonb",
|
"crawler_pending = %s::jsonb",
|
||||||
# On vient d'enrichir : le signal de ré-enrichissement est consommé.
|
|
||||||
"enrich_pending = false",
|
|
||||||
]
|
]
|
||||||
params += [
|
params += [
|
||||||
band_data, ts, html_hash,
|
band_data, ts, html_hash,
|
||||||
|
|
@ -185,11 +162,9 @@ def get_bands_to_enrich(country: str | None = None, limit: int = 50) -> list[dic
|
||||||
"""
|
"""
|
||||||
File de priorité pour l'enrichissement :
|
File de priorité pour l'enrichissement :
|
||||||
1. Nouveaux bands (band_page absent) — jamais enrichis
|
1. Nouveaux bands (band_page absent) — jamais enrichis
|
||||||
2. enrich_pending : MA a modifié le groupe (horodatage "modified" changé),
|
2. Modifiés depuis le dernier enrichissement (updated_at > crawled_at + 1 min)
|
||||||
y compris sur des champs invisibles au niveau listing
|
3. Héritage ancien scraper (crawled_at IS NULL, band_page présent)
|
||||||
3. Modifiés depuis le dernier enrichissement (updated_at > crawled_at + 1 min)
|
4. Stale (enrichis il y a > 30 jours par le système actuel)
|
||||||
4. Héritage ancien scraper (crawled_at IS NULL, band_page présent)
|
|
||||||
5. Stale (enrichis il y a > 30 jours par le système actuel)
|
|
||||||
"""
|
"""
|
||||||
sql = """
|
sql = """
|
||||||
SELECT ma_id, data->>'url' AS url, name, country
|
SELECT ma_id, data->>'url' AS url, name, country
|
||||||
|
|
@ -198,7 +173,6 @@ def get_bands_to_enrich(country: str | None = None, limit: int = 50) -> list[dic
|
||||||
AND (
|
AND (
|
||||||
data->'band_page' IS NULL
|
data->'band_page' IS NULL
|
||||||
OR crawled_at IS NULL
|
OR crawled_at IS NULL
|
||||||
OR enrich_pending
|
|
||||||
OR (crawled_at IS NOT NULL AND updated_at > crawled_at + interval '1 minute')
|
OR (crawled_at IS NOT NULL AND updated_at > crawled_at + interval '1 minute')
|
||||||
OR crawled_at < now() - interval '30 days'
|
OR crawled_at < now() - interval '30 days'
|
||||||
)
|
)
|
||||||
|
|
@ -211,10 +185,9 @@ def get_bands_to_enrich(country: str | None = None, limit: int = 50) -> list[dic
|
||||||
ORDER BY
|
ORDER BY
|
||||||
CASE
|
CASE
|
||||||
WHEN data->'band_page' IS NULL THEN 1
|
WHEN data->'band_page' IS NULL THEN 1
|
||||||
WHEN enrich_pending THEN 2
|
WHEN crawled_at IS NOT NULL AND updated_at > crawled_at + interval '1 minute' THEN 2
|
||||||
WHEN crawled_at IS NOT NULL AND updated_at > crawled_at + interval '1 minute' THEN 3
|
WHEN crawled_at IS NULL THEN 3
|
||||||
WHEN crawled_at IS NULL THEN 4
|
ELSE 4
|
||||||
ELSE 5
|
|
||||||
END ASC,
|
END ASC,
|
||||||
updated_at DESC NULLS LAST
|
updated_at DESC NULLS LAST
|
||||||
LIMIT %s
|
LIMIT %s
|
||||||
|
|
@ -320,44 +293,6 @@ def raise_if_cancelled(run_id: int):
|
||||||
raise RunCancelled(f"run #{run_id} annulé par un administrateur")
|
raise RunCancelled(f"run #{run_id} annulé par un administrateur")
|
||||||
|
|
||||||
|
|
||||||
def recover_stuck_runs():
|
|
||||||
"""Au démarrage : referme tout crawl_run / job_trigger resté en 'running'.
|
|
||||||
|
|
||||||
Le crawler tourne en instance unique : une ligne encore 'running' ne peut
|
|
||||||
provenir que d'une instance précédente tuée brutalement (OOM, SIGKILL) avant
|
|
||||||
d'atteindre son bloc finally. On les marque en erreur pour ne pas polluer le
|
|
||||||
suivi admin et libérer la file de jobs.
|
|
||||||
|
|
||||||
Ne touche QUE les job_triggers consommés par le crawler : ceux du geocoder
|
|
||||||
(geocoder_enqueue) appartiennent à un autre process, qui fait sa propre
|
|
||||||
récupération au démarrage.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
with get_conn() as conn:
|
|
||||||
with conn.cursor() as cur:
|
|
||||||
cur.execute(
|
|
||||||
"""UPDATE crawl_run
|
|
||||||
SET status='error', finished_at=now(),
|
|
||||||
error='interrompu (redémarrage du crawler)'
|
|
||||||
WHERE status='running'
|
|
||||||
AND run_type <> 'geocoder_enqueue'"""
|
|
||||||
)
|
|
||||||
n_runs = cur.rowcount
|
|
||||||
cur.execute(
|
|
||||||
"""UPDATE job_triggers
|
|
||||||
SET status='error', finished_at=now(),
|
|
||||||
error='interrompu (redémarrage du crawler)'
|
|
||||||
WHERE status='running'
|
|
||||||
AND job_type = ANY(%s)""",
|
|
||||||
(CRAWLER_JOB_TYPES,),
|
|
||||||
)
|
|
||||||
n_jobs = cur.rowcount
|
|
||||||
if n_runs or n_jobs:
|
|
||||||
log.warning(f"[db] recover_stuck_runs: {n_runs} run(s) + {n_jobs} job(s) refermés")
|
|
||||||
except Exception as e:
|
|
||||||
log.warning(f"[db] recover_stuck_runs failed: {e}")
|
|
||||||
|
|
||||||
|
|
||||||
def finish_crawl_run(
|
def finish_crawl_run(
|
||||||
run_id: int,
|
run_id: int,
|
||||||
stats: dict[str, int],
|
stats: dict[str, int],
|
||||||
|
|
@ -411,19 +346,12 @@ def set_checkpoint(key: str, value: str):
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
def claim_job_trigger(job_type: str | None = None) -> int | None:
|
def claim_job_trigger(job_type: str | None = None) -> int | None:
|
||||||
"""Récupère et verrouille un job trigger en attente. Retourne son id ou None.
|
"""Récupère et verrouille un job trigger en attente. Retourne son id ou None."""
|
||||||
|
|
||||||
Sans `job_type` explicite, la recherche est bornée aux CRAWLER_JOB_TYPES :
|
|
||||||
réclamer un trigger qu'on ne sait pas exécuter revient à le détruire.
|
|
||||||
"""
|
|
||||||
where = "status = 'pending'"
|
where = "status = 'pending'"
|
||||||
params: list = []
|
params: list = []
|
||||||
if job_type:
|
if job_type:
|
||||||
where += " AND job_type = %s"
|
where += " AND job_type = %s"
|
||||||
params.append(job_type)
|
params.append(job_type)
|
||||||
else:
|
|
||||||
where += " AND job_type = ANY(%s)"
|
|
||||||
params.append(CRAWLER_JOB_TYPES)
|
|
||||||
params.append(1)
|
params.append(1)
|
||||||
try:
|
try:
|
||||||
with get_conn() as conn:
|
with get_conn() as conn:
|
||||||
|
|
@ -463,5 +391,5 @@ def finish_job_trigger(trigger_id: int, error: str | None = None):
|
||||||
def _parse_year(s: str | None) -> int | None:
|
def _parse_year(s: str | None) -> int | None:
|
||||||
if not s:
|
if not s:
|
||||||
return None
|
return None
|
||||||
m = re.search(r"\b(1[89]\d\d|20\d\d)\b", str(s))
|
m = __import__("re").search(r"\b(1[89]\d\d|20\d\d)\b", str(s))
|
||||||
return int(m.group(1)) if m else None
|
return int(m.group(1)) if m else None
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,5 @@
|
||||||
"""
|
"""
|
||||||
Contrôle de santé périodique des services de fond — module PARTAGÉ.
|
Contrôle de santé périodique des services de fond.
|
||||||
|
|
||||||
Vit dans libs/ et non dans une app : le crawler et le geocoder en avaient
|
|
||||||
deux copies rigoureusement identiques, dont une seule était couverte par des
|
|
||||||
tests. Rien ne signalait une divergence, et une correction appliquée d'un
|
|
||||||
seul côté serait passée inaperçue.
|
|
||||||
|
|
||||||
Embarqué dans les images par les Dockerfile (contexte de build = racine du
|
|
||||||
dépôt) et rendu importable en local par les conftest.py de chaque app.
|
|
||||||
|
|
||||||
Ces services ne sont pas exposés par Traefik : aucune sonde HTTP externe ne peut
|
Ces services ne sont pas exposés par Traefik : aucune sonde HTTP externe ne peut
|
||||||
les atteindre. Sans ce module, un crawler dont FlareSolverr est injoignable ou
|
les atteindre. Sans ce module, un crawler dont FlareSolverr est injoignable ou
|
||||||
|
|
@ -10,8 +10,6 @@ from .config import (
|
||||||
COOLDOWN_EVERY,
|
COOLDOWN_EVERY,
|
||||||
COOLDOWN_MAX,
|
COOLDOWN_MAX,
|
||||||
COOLDOWN_MIN,
|
COOLDOWN_MIN,
|
||||||
LIST_MAX_DELAY,
|
|
||||||
LIST_MIN_DELAY,
|
|
||||||
MA_BASE,
|
MA_BASE,
|
||||||
)
|
)
|
||||||
from .db import (
|
from .db import (
|
||||||
|
|
@ -74,12 +72,6 @@ def run_full_crawl(session: MASession, countries: list[str] = None):
|
||||||
stats["new"] += r["inserted"]
|
stats["new"] += r["inserted"]
|
||||||
stats["updated"] += r["updated"]
|
stats["updated"] += r["updated"]
|
||||||
|
|
||||||
update_crawl_run_progress(run_id, stats)
|
|
||||||
# Politesse entre pays : la dernière page d'un pays ne dort pas (le
|
|
||||||
# générateur sort sur un break), d'où cette pause explicite avant
|
|
||||||
# d'attaquer le suivant.
|
|
||||||
sleep_range(LIST_MIN_DELAY, LIST_MAX_DELAY)
|
|
||||||
|
|
||||||
set_checkpoint("last_full_crawl_at", _now_iso())
|
set_checkpoint("last_full_crawl_at", _now_iso())
|
||||||
log.info(f"[full] done: {stats}")
|
log.info(f"[full] done: {stats}")
|
||||||
log_event("info", f"full crawl done: {stats}", run_id=run_id)
|
log_event("info", f"full crawl done: {stats}", run_id=run_id)
|
||||||
|
|
@ -186,7 +178,6 @@ def run_enrich(session: MASession, limit: int = 100, country: str | None = None)
|
||||||
if not url.startswith("http"):
|
if not url.startswith("http"):
|
||||||
url = MA_BASE + url
|
url = MA_BASE + url
|
||||||
|
|
||||||
failed = False
|
|
||||||
try:
|
try:
|
||||||
html = session.get_html(url)
|
html = session.get_html(url)
|
||||||
data = parse_band_page(html)
|
data = parse_band_page(html)
|
||||||
|
|
@ -196,18 +187,11 @@ def run_enrich(session: MASession, limit: int = 100, country: str | None = None)
|
||||||
stats["enriched"] += 1
|
stats["enriched"] += 1
|
||||||
stats["seen"] += 1
|
stats["seen"] += 1
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
failed = True
|
|
||||||
log.warning(f"[enrich] failed ma_id={band['ma_id']}: {e}")
|
log.warning(f"[enrich] failed ma_id={band['ma_id']}: {e}")
|
||||||
log_event("warning", f"enrich failed: {e}", run_id=run_id, ma_id=band["ma_id"])
|
log_event("warning", f"enrich failed: {e}", run_id=run_id, ma_id=band["ma_id"])
|
||||||
|
continue
|
||||||
|
|
||||||
# La politesse s'applique TOUJOURS, y compris après un échec. Le
|
sleep_range(BAND_MIN_DELAY, BAND_MAX_DELAY)
|
||||||
# `continue` d'avant la sautait : or un échec de get_html est le plus
|
|
||||||
# souvent un 403/429/503 de Metal Archives, soit le pire moment pour
|
|
||||||
# enchaîner sans délai. On ralentit donc davantage dans ce cas.
|
|
||||||
if failed:
|
|
||||||
sleep_range(BAND_MAX_DELAY, BAND_MAX_DELAY * 3)
|
|
||||||
else:
|
|
||||||
sleep_range(BAND_MIN_DELAY, BAND_MAX_DELAY)
|
|
||||||
if COOLDOWN_EVERY and (i + 1) % COOLDOWN_EVERY == 0:
|
if COOLDOWN_EVERY and (i + 1) % COOLDOWN_EVERY == 0:
|
||||||
cooldown(COOLDOWN_MIN, COOLDOWN_MAX)
|
cooldown(COOLDOWN_MIN, COOLDOWN_MAX)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ from typing import Any
|
||||||
from urllib.parse import urlencode
|
from urllib.parse import urlencode
|
||||||
|
|
||||||
from .config import AJAX_PAGE_SIZE, LIST_MAX_DELAY, LIST_MIN_DELAY, MA_BASE
|
from .config import AJAX_PAGE_SIZE, LIST_MAX_DELAY, LIST_MIN_DELAY, MA_BASE
|
||||||
from .flaresolverr import FlareSolverr, FlareSolverrError
|
from .flaresolverr import FlareSolverr
|
||||||
from .polite import sleep_range
|
from .polite import sleep_range
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
@ -52,29 +52,13 @@ class MASession:
|
||||||
self.ensure_session()
|
self.ensure_session()
|
||||||
full_url = _build_url(url, params)
|
full_url = _build_url(url, params)
|
||||||
for attempt in range(retries + 1):
|
for attempt in range(retries + 1):
|
||||||
try:
|
status, body = self.fs.get(full_url, self._session_id)
|
||||||
status, body = self.fs.get(full_url, self._session_id)
|
|
||||||
except FlareSolverrError as e:
|
|
||||||
# Blip réseau / crash du Chrome headless : on retente avec une
|
|
||||||
# session neuve plutôt que de laisser l'erreur remonter et perdre
|
|
||||||
# tout le run en cours.
|
|
||||||
if attempt < retries:
|
|
||||||
log.warning(f"[ma_http] FlareSolverr KO sur {url} ({e}), refresh session…")
|
|
||||||
self.ensure_session(force_refresh=True)
|
|
||||||
sleep_range(LIST_MIN_DELAY * 2, LIST_MAX_DELAY * 2)
|
|
||||||
continue
|
|
||||||
raise RuntimeError(
|
|
||||||
f"FlareSolverr échec après {retries} tentatives sur {url}: {e}"
|
|
||||||
) from e
|
|
||||||
if status in (403, 429, 503) and attempt < retries:
|
if status in (403, 429, 503) and attempt < retries:
|
||||||
log.warning(f"[ma_http] {status} on {url}, refreshing session...")
|
log.warning(f"[ma_http] {status} on {url}, refreshing session...")
|
||||||
self.ensure_session(force_refresh=True)
|
self.ensure_session(force_refresh=True)
|
||||||
sleep_range(LIST_MIN_DELAY * 2, LIST_MAX_DELAY * 2)
|
sleep_range(LIST_MIN_DELAY * 2, LIST_MAX_DELAY * 2)
|
||||||
continue
|
continue
|
||||||
if status != 200:
|
if status != 200:
|
||||||
# Volontairement PAS de retry ici : un 500/502 n'est pas un
|
|
||||||
# blocage anti-bot, et réessayer ne ferait que marteler un
|
|
||||||
# serveur déjà en difficulté. Verrouillé par un test.
|
|
||||||
raise RuntimeError(f"HTTP {status} on {full_url}")
|
raise RuntimeError(f"HTTP {status} on {full_url}")
|
||||||
try:
|
try:
|
||||||
return _extract_json(body)
|
return _extract_json(body)
|
||||||
|
|
@ -93,20 +77,7 @@ class MASession:
|
||||||
"""GET HTML rendu (pages band, etc.)."""
|
"""GET HTML rendu (pages band, etc.)."""
|
||||||
self.ensure_session()
|
self.ensure_session()
|
||||||
for attempt in range(retries + 1):
|
for attempt in range(retries + 1):
|
||||||
try:
|
status, body = self.fs.get(url, self._session_id)
|
||||||
status, body = self.fs.get(url, self._session_id)
|
|
||||||
except FlareSolverrError as e:
|
|
||||||
# Blip réseau / crash du Chrome headless : on retente avec une
|
|
||||||
# session neuve plutôt que de laisser l'erreur remonter et perdre
|
|
||||||
# tout le run en cours.
|
|
||||||
if attempt < retries:
|
|
||||||
log.warning(f"[ma_http] FlareSolverr KO sur {url} ({e}), refresh session…")
|
|
||||||
self.ensure_session(force_refresh=True)
|
|
||||||
sleep_range(LIST_MIN_DELAY * 2, LIST_MAX_DELAY * 2)
|
|
||||||
continue
|
|
||||||
raise RuntimeError(
|
|
||||||
f"FlareSolverr échec après {retries} tentatives sur {url}: {e}"
|
|
||||||
) from e
|
|
||||||
if status in (403, 429, 503) and attempt < retries:
|
if status in (403, 429, 503) and attempt < retries:
|
||||||
log.warning(f"[ma_http] {status} on {url}, refreshing session...")
|
log.warning(f"[ma_http] {status} on {url}, refreshing session...")
|
||||||
self.ensure_session(force_refresh=True)
|
self.ensure_session(force_refresh=True)
|
||||||
|
|
@ -294,25 +265,7 @@ def _parse_archive_row(row, order_by: str) -> dict | None:
|
||||||
|
|
||||||
country_href, _ = _extract_link(row[2])
|
country_href, _ = _extract_link(row[2])
|
||||||
m = re.search(r"/lists/([A-Z]{1,3})", country_href or "")
|
m = re.search(r"/lists/([A-Z]{1,3})", country_href or "")
|
||||||
if m:
|
cc = m.group(1) if m else _clean(row[2])
|
||||||
cc = m.group(1)
|
|
||||||
else:
|
|
||||||
# Lien pays absent ou malformé : on retombe sur le texte brut (ex.
|
|
||||||
# "Germany"), qui ne matchera pas _EU_SET côté jobs.py et sera écarté.
|
|
||||||
# Le log rend visible un éventuel changement de structure HTML chez MA.
|
|
||||||
cc = _clean(row[2])
|
|
||||||
log.warning(
|
|
||||||
f"[ma_http] code pays introuvable dans '{country_href}', "
|
|
||||||
f"fallback texte='{cc}' (ma_id={ma_id})"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Horodatage "modified" que MA affiche lui-même dans sa liste (ex. "Jun 1st,
|
|
||||||
# 02:04"). Sert de signal de ré-enrichissement : quand ce texte change pour
|
|
||||||
# un groupe, on sait que MA l'a modifié même si aucun champ du listing ne
|
|
||||||
# bouge. Sans objet pour la liste des créations, où il est immuable.
|
|
||||||
ma_modified_seen = None
|
|
||||||
if order_by == "modified":
|
|
||||||
ma_modified_seen = _clean(row[4]) if len(row) > 4 else _clean(row[0])
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"ma_id": ma_id,
|
"ma_id": ma_id,
|
||||||
|
|
@ -320,7 +273,6 @@ def _parse_archive_row(row, order_by: str) -> dict | None:
|
||||||
"url": href,
|
"url": href,
|
||||||
"country": cc,
|
"country": cc,
|
||||||
"genre": _clean(row[3]),
|
"genre": _clean(row[3]),
|
||||||
"ma_modified_seen": ma_modified_seen,
|
|
||||||
"date_str": None, # pas d'année dans le format MA → pas de filtre date
|
"date_str": None, # pas d'année dans le format MA → pas de filtre date
|
||||||
"date_type": order_by,
|
"date_type": order_by,
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -16,13 +16,11 @@ Variables d'env :
|
||||||
CRAWLER_FULL_CRAWL_INTERVAL_DAYS (défaut: 60, mettre 0 pour désactiver)
|
CRAWLER_FULL_CRAWL_INTERVAL_DAYS (défaut: 60, mettre 0 pour désactiver)
|
||||||
"""
|
"""
|
||||||
import logging
|
import logging
|
||||||
import signal
|
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
import schedule
|
import schedule
|
||||||
from bm_health import database_probe, freshness_probe, http_probe, maybe_run
|
|
||||||
|
|
||||||
from .config import (
|
from .config import (
|
||||||
ENRICH_LIMIT,
|
ENRICH_LIMIT,
|
||||||
|
|
@ -32,8 +30,9 @@ from .config import (
|
||||||
SCHED_ENRICH_H,
|
SCHED_ENRICH_H,
|
||||||
SCHED_INCREMENTAL_H,
|
SCHED_INCREMENTAL_H,
|
||||||
)
|
)
|
||||||
from .db import claim_job_trigger, finish_job_trigger, get_checkpoint, recover_stuck_runs
|
from .db import claim_job_trigger, finish_job_trigger, get_checkpoint
|
||||||
from .flaresolverr import FlareSolverr
|
from .flaresolverr import FlareSolverr
|
||||||
|
from .health import database_probe, freshness_probe, http_probe, maybe_run
|
||||||
from .jobs import run_enrich, run_full_crawl, run_incremental
|
from .jobs import run_enrich, run_full_crawl, run_incremental
|
||||||
from .ma_http import MASession
|
from .ma_http import MASession
|
||||||
|
|
||||||
|
|
@ -56,53 +55,13 @@ def _wait_flaresolverr(fs: FlareSolverr, max_wait: int = 120):
|
||||||
raise RuntimeError("FlareSolverr not reachable after timeout")
|
raise RuntimeError("FlareSolverr not reachable after timeout")
|
||||||
|
|
||||||
|
|
||||||
# Arrêt propre sur SIGTERM/SIGINT. Sans lui, chaque redeploy abandonnait une
|
|
||||||
# session Chrome persistante côté FlareSolverr : elle n'est détruite qu'au
|
|
||||||
# moment d'en ouvrir une neuve, jamais à l'extinction du crawler.
|
|
||||||
_shutdown = False
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_shutdown(signum, frame):
|
|
||||||
global _shutdown
|
|
||||||
_shutdown = True
|
|
||||||
log.info(f"[main] signal {signum} reçu — arrêt propre demandé")
|
|
||||||
|
|
||||||
|
|
||||||
def _safe(label: str, fn):
|
|
||||||
"""Exécute un job en isolant toute exception.
|
|
||||||
|
|
||||||
Un job qui plante ne doit jamais tuer le process : `restart: unless-stopped`
|
|
||||||
le relancerait en boucle, et une panne DB transitoire suffirait à mettre le
|
|
||||||
crawler en crash-loop au lieu de le faire attendre le cycle suivant.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
fn()
|
|
||||||
except Exception as e:
|
|
||||||
log.error(f"[main] job '{label}' a échoué (ignoré, on continue): {e}", exc_info=True)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
signal.signal(signal.SIGTERM, _handle_shutdown)
|
|
||||||
signal.signal(signal.SIGINT, _handle_shutdown)
|
|
||||||
|
|
||||||
log.info("[main] crawler starting")
|
log.info("[main] crawler starting")
|
||||||
|
|
||||||
# Runs/jobs laissés en 'running' par une instance précédente tuée brutalement
|
|
||||||
# (OOM, SIGKILL) : ils ne peuvent pas appartenir à ce process, on les referme.
|
|
||||||
_safe("recover_stuck_runs", recover_stuck_runs)
|
|
||||||
|
|
||||||
fs = FlareSolverr(FLARESOLVERR_URL, timeout_ms=FS_TIMEOUT_MS)
|
fs = FlareSolverr(FLARESOLVERR_URL, timeout_ms=FS_TIMEOUT_MS)
|
||||||
_wait_flaresolverr(fs)
|
_wait_flaresolverr(fs)
|
||||||
|
|
||||||
ma = MASession(fs)
|
ma = MASession(fs)
|
||||||
|
|
||||||
def _liberer_session():
|
|
||||||
"""Rend sa session Chrome à FlareSolverr avant de sortir."""
|
|
||||||
sid = getattr(ma, "_session_id", None)
|
|
||||||
if sid:
|
|
||||||
fs.destroy_session(sid)
|
|
||||||
ma._session_id = None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Helpers pour lancer les jobs avec log
|
# Helpers pour lancer les jobs avec log
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
@ -151,24 +110,16 @@ def main():
|
||||||
|
|
||||||
# Premier run au démarrage (job_full rattrape un redémarrage qui aurait loupé la fenêtre)
|
# Premier run au démarrage (job_full rattrape un redémarrage qui aurait loupé la fenêtre)
|
||||||
log.info("[main] running initial jobs at startup")
|
log.info("[main] running initial jobs at startup")
|
||||||
_safe("incremental", job_incremental)
|
job_incremental()
|
||||||
_safe("enrich", job_enrich)
|
job_enrich()
|
||||||
_safe("full", job_full)
|
job_full()
|
||||||
|
|
||||||
log.info("[main] entering scheduler loop")
|
log.info("[main] entering scheduler loop")
|
||||||
try:
|
while True:
|
||||||
while not _shutdown:
|
schedule.run_pending()
|
||||||
_safe("run_pending", schedule.run_pending)
|
_check_job_triggers(ma)
|
||||||
_safe("check_job_triggers", lambda: _check_job_triggers(ma))
|
_check_health()
|
||||||
_safe("check_health", _check_health)
|
time.sleep(60)
|
||||||
# Sommeil fractionné : un SIGTERM ne doit pas attendre une minute.
|
|
||||||
for _ in range(60):
|
|
||||||
if _shutdown:
|
|
||||||
break
|
|
||||||
time.sleep(1)
|
|
||||||
finally:
|
|
||||||
_safe("liberer_session", _liberer_session)
|
|
||||||
log.info("[main] crawler arrêté")
|
|
||||||
|
|
||||||
|
|
||||||
def _check_health():
|
def _check_health():
|
||||||
|
|
|
||||||
|
|
@ -1,9 +1,6 @@
|
||||||
import logging
|
|
||||||
import random
|
import random
|
||||||
import time
|
import time
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
|
|
||||||
def sleep_range(min_s: float, max_s: float):
|
def sleep_range(min_s: float, max_s: float):
|
||||||
time.sleep(random.uniform(min_s, max_s))
|
time.sleep(random.uniform(min_s, max_s))
|
||||||
|
|
@ -11,5 +8,5 @@ def sleep_range(min_s: float, max_s: float):
|
||||||
|
|
||||||
def cooldown(min_s: float, max_s: float):
|
def cooldown(min_s: float, max_s: float):
|
||||||
secs = random.uniform(min_s, max_s)
|
secs = random.uniform(min_s, max_s)
|
||||||
log.info(f"[polite] cooldown {secs:.1f}s")
|
print(f"[polite] cooldown {secs:.1f}s")
|
||||||
time.sleep(secs)
|
time.sleep(secs)
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,3 @@
|
||||||
# Le module de sante est partage entre les apps (libs/bm_health.py) : en
|
|
||||||
# Docker il arrive via PYTHONPATH, ici on l'ajoute au chemin d'import.
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "libs"))
|
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
|
||||||
|
|
@ -8,9 +8,9 @@ de contrôle du tout — d'où le nombre de cas d'échec vérifiés ici.
|
||||||
|
|
||||||
import contextlib
|
import contextlib
|
||||||
|
|
||||||
import bm_health as health
|
|
||||||
import pytest
|
import pytest
|
||||||
from bm_health import (
|
from src import health
|
||||||
|
from src.health import (
|
||||||
database_probe,
|
database_probe,
|
||||||
freshness_probe,
|
freshness_probe,
|
||||||
http_probe,
|
http_probe,
|
||||||
|
|
|
||||||
|
|
@ -3,16 +3,10 @@ FROM python:3.12-slim
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
|
|
||||||
# Copier et installer les dépendances en premier (meilleur cache Docker)
|
# Copier et installer les dépendances en premier (meilleur cache Docker)
|
||||||
COPY apps/geocoder/requirements.txt .
|
COPY requirements.txt .
|
||||||
RUN pip install --no-cache-dir -r requirements.txt
|
RUN pip install --no-cache-dir -r requirements.txt
|
||||||
|
|
||||||
# Contexte de build = racine du depot, pour embarquer le module partage.
|
# Copier le code de l'application
|
||||||
COPY libs/ ./libs/
|
COPY . .
|
||||||
COPY apps/geocoder/src ./src
|
|
||||||
ENV PYTHONPATH=/app/libs
|
|
||||||
|
|
||||||
# Utilisateur non-root
|
# Le command sera spécifié dans docker-compose.yml
|
||||||
RUN useradd -m -u 1000 geocoder
|
|
||||||
USER geocoder
|
|
||||||
|
|
||||||
# Le command est spécifié dans docker-compose.yml (working_dir: /app)
|
|
||||||
|
|
|
||||||
|
|
@ -17,13 +17,10 @@ Tourne en continu (restart: unless-stopped), poll toutes les 15s.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import signal
|
|
||||||
import time
|
import time
|
||||||
|
|
||||||
import psycopg2
|
import psycopg2
|
||||||
from bm_health import database_probe, freshness_probe, maybe_run
|
|
||||||
from parser import (
|
from parser import (
|
||||||
COUNTRY_CENTROIDS,
|
|
||||||
COUNTRY_NAME_TO_ISO2,
|
COUNTRY_NAME_TO_ISO2,
|
||||||
country_centroid,
|
country_centroid,
|
||||||
parse_location_text,
|
parse_location_text,
|
||||||
|
|
@ -33,42 +30,16 @@ POLL_INTERVAL = int(os.environ.get("GEOCODE_ENQUEUE_POLL", "15"))
|
||||||
BATCH = int(os.environ.get("GEOCODE_ENQUEUE_BATCH", "100000"))
|
BATCH = int(os.environ.get("GEOCODE_ENQUEUE_BATCH", "100000"))
|
||||||
AUTO_INTERVAL_MIN = int(os.environ.get("ENQUEUE_AUTO_INTERVAL_MIN", "60"))
|
AUTO_INTERVAL_MIN = int(os.environ.get("ENQUEUE_AUTO_INTERVAL_MIN", "60"))
|
||||||
|
|
||||||
# Arrêt propre sur SIGTERM/SIGINT.
|
|
||||||
_shutdown = False
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_shutdown(signum, frame):
|
|
||||||
global _shutdown
|
|
||||||
_shutdown = True
|
|
||||||
print(f"[enqueue] signal {signum} reçu — arrêt propre")
|
|
||||||
|
|
||||||
|
|
||||||
def _sleep_interruptible(seconds: float, step: float = 1.0):
|
|
||||||
remaining = seconds
|
|
||||||
while remaining > 0 and not _shutdown:
|
|
||||||
time.sleep(min(step, remaining))
|
|
||||||
remaining -= step
|
|
||||||
|
|
||||||
|
|
||||||
def resolve_iso2(location_raw: str, band_country: str | None) -> str | None:
|
def resolve_iso2(location_raw: str, band_country: str | None) -> str | None:
|
||||||
t = location_raw.strip()
|
t = location_raw.strip()
|
||||||
# Un code à 2 lettres est validé contre la liste connue : sans ça, un faux
|
|
||||||
# code ("XX") était renvoyé tel quel, et country_centroid retournait None
|
|
||||||
# plus loin sans qu'on sache pourquoi. Les alias non standards ("UK") sont
|
|
||||||
# rattrapés par le mapping.
|
|
||||||
if len(t) == 2 and t.upper().isalpha():
|
if len(t) == 2 and t.upper().isalpha():
|
||||||
code = t.upper()
|
return t.upper()
|
||||||
if code in COUNTRY_CENTROIDS:
|
|
||||||
return code
|
|
||||||
alias = COUNTRY_NAME_TO_ISO2.get(t.lower())
|
|
||||||
if alias:
|
|
||||||
return alias
|
|
||||||
iso = COUNTRY_NAME_TO_ISO2.get(t.lower())
|
iso = COUNTRY_NAME_TO_ISO2.get(t.lower())
|
||||||
if iso:
|
if iso:
|
||||||
return iso
|
return iso
|
||||||
if band_country:
|
if band_country:
|
||||||
bc = band_country.strip().upper()
|
return band_country.strip().upper()
|
||||||
return bc if bc in COUNTRY_CENTROIDS else None
|
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -224,116 +195,46 @@ def _execute_run(cur, trigger_label: str, job_id: int | None = None) -> None:
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def _connect(dsn):
|
def main():
|
||||||
|
dsn = os.environ["DATABASE_URL"]
|
||||||
conn = psycopg2.connect(dsn)
|
conn = psycopg2.connect(dsn)
|
||||||
conn.autocommit = True
|
conn.autocommit = True
|
||||||
return conn
|
|
||||||
|
|
||||||
|
|
||||||
def _recover_stuck(cur):
|
|
||||||
"""Referme les runs/jobs geocoder_enqueue laissés 'running' par une instance
|
|
||||||
précédente tuée brutalement. Strictement bornés à ce type : les autres
|
|
||||||
appartiennent au crawler, qui fait sa propre récupération."""
|
|
||||||
cur.execute(
|
|
||||||
"""UPDATE crawl_run SET status='error', finished_at=now(),
|
|
||||||
error='interrompu (redémarrage enqueue)'
|
|
||||||
WHERE run_type='geocoder_enqueue' AND status='running'"""
|
|
||||||
)
|
|
||||||
cur.execute(
|
|
||||||
"""UPDATE job_triggers SET status='error', finished_at=now(),
|
|
||||||
error='interrompu (redémarrage enqueue)'
|
|
||||||
WHERE job_type='geocoder_enqueue' AND status='running'"""
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
signal.signal(signal.SIGTERM, _handle_shutdown)
|
|
||||||
signal.signal(signal.SIGINT, _handle_shutdown)
|
|
||||||
|
|
||||||
dsn = os.environ["DATABASE_URL"]
|
|
||||||
conn = _connect(dsn)
|
|
||||||
|
|
||||||
print(f"[enqueue] daemon démarré, poll {POLL_INTERVAL}s, auto toutes les {AUTO_INTERVAL_MIN}min")
|
print(f"[enqueue] daemon démarré, poll {POLL_INTERVAL}s, auto toutes les {AUTO_INTERVAL_MIN}min")
|
||||||
|
|
||||||
# La migration 015 prévoit un battement de cœur par service de fond ; celui
|
|
||||||
# de l'enqueue n'existait pas. Un daemon mort restait donc invisible dans le
|
|
||||||
# bandeau de santé de l'admin.
|
|
||||||
import contextlib
|
|
||||||
|
|
||||||
@contextlib.contextmanager
|
|
||||||
def get_conn():
|
|
||||||
yield conn
|
|
||||||
|
|
||||||
def health_check():
|
|
||||||
maybe_run(get_conn, "geocoder-enqueue", [
|
|
||||||
("database", database_probe(get_conn)),
|
|
||||||
# L'alimentation tourne au moins toutes les AUTO_INTERVAL_MIN
|
|
||||||
# minutes : au-delà du double, elle ne passe plus.
|
|
||||||
("progression", freshness_probe(
|
|
||||||
get_conn,
|
|
||||||
"SELECT max(started_at) FROM crawl_run WHERE run_type = 'geocoder_enqueue'",
|
|
||||||
max(2, (AUTO_INTERVAL_MIN * 2) // 60), "dernière alimentation")),
|
|
||||||
])
|
|
||||||
|
|
||||||
last_auto = time.monotonic()
|
last_auto = time.monotonic()
|
||||||
|
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
_recover_stuck(cur)
|
while True:
|
||||||
|
# Rattrapage automatique (bands rendus 'dirty' par le trigger DB)
|
||||||
|
if time.monotonic() - last_auto >= AUTO_INTERVAL_MIN * 60:
|
||||||
|
_execute_run(cur, "auto")
|
||||||
|
last_auto = time.monotonic()
|
||||||
|
|
||||||
while not _shutdown:
|
# Déclenchement manuel via l'admin
|
||||||
try:
|
cur.execute(
|
||||||
with conn.cursor() as cur:
|
"""
|
||||||
# Rattrapage automatique (bands rendus 'dirty' par le trigger DB)
|
SELECT id FROM job_triggers
|
||||||
if time.monotonic() - last_auto >= AUTO_INTERVAL_MIN * 60:
|
WHERE job_type = 'geocoder_enqueue' AND status = 'pending'
|
||||||
_execute_run(cur, "auto")
|
ORDER BY created_at ASC
|
||||||
last_auto = time.monotonic()
|
LIMIT 1
|
||||||
|
FOR UPDATE SKIP LOCKED
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
row = cur.fetchone()
|
||||||
|
|
||||||
# Déclenchement manuel via l'admin — claim ATOMIQUE : en
|
if not row:
|
||||||
# autocommit, le `SELECT … FOR UPDATE` relâchait son verrou avant
|
time.sleep(POLL_INTERVAL)
|
||||||
# l'UPDATE suivant, donc il ne protégeait rien.
|
continue
|
||||||
cur.execute(
|
|
||||||
"""
|
|
||||||
UPDATE job_triggers AS t
|
|
||||||
SET status='running', started_at=now()
|
|
||||||
WHERE t.id = (
|
|
||||||
SELECT id FROM job_triggers
|
|
||||||
WHERE job_type = 'geocoder_enqueue' AND status = 'pending'
|
|
||||||
ORDER BY created_at ASC
|
|
||||||
LIMIT 1
|
|
||||||
FOR UPDATE SKIP LOCKED
|
|
||||||
)
|
|
||||||
RETURNING t.id
|
|
||||||
"""
|
|
||||||
)
|
|
||||||
row = cur.fetchone()
|
|
||||||
|
|
||||||
if not row:
|
job_id = row[0]
|
||||||
health_check()
|
cur.execute(
|
||||||
_sleep_interruptible(POLL_INTERVAL)
|
"UPDATE job_triggers SET status='running', started_at=now() WHERE id=%s",
|
||||||
continue
|
(job_id,),
|
||||||
|
)
|
||||||
|
_execute_run(cur, "manuel", job_id=job_id)
|
||||||
|
|
||||||
_execute_run(cur, "manuel", job_id=row[0])
|
conn.close()
|
||||||
except psycopg2.Error as e:
|
|
||||||
# Perte de connexion en cours de route : on se reconnecte au lieu de
|
|
||||||
# laisser mourir le process (sinon crash-loop du conteneur).
|
|
||||||
print(f"[enqueue] erreur DB ({e}), reconnexion dans 5s")
|
|
||||||
try:
|
|
||||||
conn.close()
|
|
||||||
except Exception: # noqa: S110 - la connexion est déjà morte, c'est le cas nominal ici
|
|
||||||
pass
|
|
||||||
_sleep_interruptible(5)
|
|
||||||
if _shutdown:
|
|
||||||
break
|
|
||||||
try:
|
|
||||||
conn = _connect(dsn)
|
|
||||||
except Exception as e2:
|
|
||||||
print(f"[enqueue] reconnexion échouée ({e2})")
|
|
||||||
_sleep_interruptible(10)
|
|
||||||
|
|
||||||
try:
|
|
||||||
conn.close()
|
|
||||||
except Exception: # noqa: S110 - fermeture au mieux à l'arrêt, rien à journaliser
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|
|
||||||
|
|
@ -19,36 +19,16 @@ import hashlib
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
import signal
|
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
import psycopg2
|
import psycopg2
|
||||||
import requests
|
import requests
|
||||||
from bm_health import database_probe, freshness_probe, maybe_run
|
|
||||||
|
|
||||||
# sys.path[0] = src/ quand lancé comme "python src/groq_worker.py"
|
# sys.path[0] = src/ quand lancé comme "python src/groq_worker.py"
|
||||||
from parser import COUNTRY_NAMES
|
from parser import COUNTRY_NAMES
|
||||||
|
|
||||||
# Arrêt propre sur SIGTERM/SIGINT (les redeploys Coolify sont fréquents).
|
|
||||||
_shutdown = False
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_shutdown(signum, frame):
|
|
||||||
global _shutdown
|
|
||||||
_shutdown = True
|
|
||||||
print(f"[groq] signal {signum} reçu — arrêt propre après l'itération courante")
|
|
||||||
|
|
||||||
|
|
||||||
def _sleep_interruptible(seconds: float, step: float = 1.0):
|
|
||||||
"""Attend `seconds`, en se réveillant tôt si un arrêt a été demandé."""
|
|
||||||
remaining = seconds
|
|
||||||
while remaining > 0 and not _shutdown:
|
|
||||||
time.sleep(min(step, remaining))
|
|
||||||
remaining -= step
|
|
||||||
|
|
||||||
|
|
||||||
GROQ_API_KEY = os.environ.get("GROQ_API_KEY", "").strip()
|
GROQ_API_KEY = os.environ.get("GROQ_API_KEY", "").strip()
|
||||||
GROQ_BASE = "https://api.groq.com/openai/v1/chat/completions"
|
GROQ_BASE = "https://api.groq.com/openai/v1/chat/completions"
|
||||||
MAX_LLM_TRIES = int(os.environ.get("GROQ_MAX_TRIES", "5"))
|
MAX_LLM_TRIES = int(os.environ.get("GROQ_MAX_TRIES", "5"))
|
||||||
|
|
@ -117,7 +97,7 @@ def _call_groq(model: str, location_raw: str, country: str) -> tuple[str, int, i
|
||||||
{"role": "system", "content": _SYSTEM},
|
{"role": "system", "content": _SYSTEM},
|
||||||
{"role": "user", "content": prompt},
|
{"role": "user", "content": prompt},
|
||||||
],
|
],
|
||||||
"max_tokens": 120,
|
"max_tokens": 80,
|
||||||
"temperature": 0.0,
|
"temperature": 0.0,
|
||||||
"response_format": {"type": "json_object"},
|
"response_format": {"type": "json_object"},
|
||||||
},
|
},
|
||||||
|
|
@ -176,9 +156,6 @@ def main():
|
||||||
print("[groq] GROQ_API_KEY non configuré — arrêt")
|
print("[groq] GROQ_API_KEY non configuré — arrêt")
|
||||||
sys.exit(0)
|
sys.exit(0)
|
||||||
|
|
||||||
signal.signal(signal.SIGTERM, _handle_shutdown)
|
|
||||||
signal.signal(signal.SIGINT, _handle_shutdown)
|
|
||||||
|
|
||||||
dsn = os.environ["DATABASE_URL"]
|
dsn = os.environ["DATABASE_URL"]
|
||||||
conn = psycopg2.connect(dsn)
|
conn = psycopg2.connect(dsn)
|
||||||
conn.autocommit = True
|
conn.autocommit = True
|
||||||
|
|
@ -188,34 +165,8 @@ def main():
|
||||||
min_start = time.monotonic()
|
min_start = time.monotonic()
|
||||||
day_start = datetime.now(UTC).date()
|
day_start = datetime.now(UTC).date()
|
||||||
|
|
||||||
# La migration 015 annonce un battement de cœur pour 'groq-worker', mais
|
|
||||||
# aucun n'était jamais écrit : sa ligne n'existait pas, et le bandeau de
|
|
||||||
# santé de l'admin ne pouvait donc rien signaler — y compris quand le
|
|
||||||
# service était mort.
|
|
||||||
import contextlib
|
|
||||||
|
|
||||||
@contextlib.contextmanager
|
|
||||||
def get_conn():
|
|
||||||
yield conn
|
|
||||||
|
|
||||||
def health_check():
|
|
||||||
maybe_run(get_conn, "groq-worker", [
|
|
||||||
("database", database_probe(get_conn)),
|
|
||||||
("groq_cle", lambda: (bool(GROQ_API_KEY),
|
|
||||||
"clé absente" if not GROQ_API_KEY else "clé présente")),
|
|
||||||
# Le pipeline doit avancer : au-delà de 24 h sans lieu sorti de la
|
|
||||||
# file LLM alors qu'il en reste, quelque chose bloque.
|
|
||||||
("progression", freshness_probe(
|
|
||||||
get_conn,
|
|
||||||
"SELECT max(updated_at) FROM band_locations "
|
|
||||||
"WHERE geocode_status IN ('queued','manual') AND geocode_tries_llm > 0",
|
|
||||||
24, "dernier lieu traité par le LLM")),
|
|
||||||
])
|
|
||||||
|
|
||||||
health_check()
|
|
||||||
|
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
while not _shutdown:
|
while True:
|
||||||
# Reset compteurs si nouvelle minute / nouveau jour
|
# Reset compteurs si nouvelle minute / nouveau jour
|
||||||
if time.monotonic() - min_start >= 60:
|
if time.monotonic() - min_start >= 60:
|
||||||
min_count = {m[0]: 0 for m in MODELS}
|
min_count = {m[0]: 0 for m in MODELS}
|
||||||
|
|
@ -225,34 +176,23 @@ def main():
|
||||||
day_count = {m[0]: 0 for m in MODELS}
|
day_count = {m[0]: 0 for m in MODELS}
|
||||||
day_start = today
|
day_start = today
|
||||||
|
|
||||||
# Claim ATOMIQUE : on réserve la ligne en poussant geocode_next_at
|
|
||||||
# 5 min dans le futur, en un seul statement. En autocommit, un
|
|
||||||
# `SELECT … FOR UPDATE` séparé relâche son verrou immédiatement,
|
|
||||||
# laissant une 2e réplique sélectionner la même ligne et payer un
|
|
||||||
# appel LLM en double. Si le worker meurt ensuite, la ligne reste
|
|
||||||
# 'llm_needed' et sera reprise après 5 min.
|
|
||||||
cur.execute(
|
cur.execute(
|
||||||
"""
|
"""
|
||||||
UPDATE band_locations AS t
|
SELECT bl.id, bl.ma_id, bl.location_raw, bl.geocode_tries_llm,
|
||||||
SET geocode_next_at = now() + interval '5 minutes'
|
b.country
|
||||||
WHERE t.id = (
|
FROM band_locations bl
|
||||||
SELECT bl.id
|
JOIN bands b ON b.ma_id = bl.ma_id
|
||||||
FROM band_locations bl
|
WHERE bl.geocode_status = 'llm_needed'
|
||||||
WHERE bl.geocode_status = 'llm_needed'
|
AND bl.geocode_next_at <= now()
|
||||||
AND bl.geocode_next_at <= now()
|
ORDER BY bl.geocode_tries_llm ASC, bl.id ASC
|
||||||
ORDER BY bl.geocode_tries_llm ASC, bl.id ASC
|
LIMIT 1
|
||||||
LIMIT 1
|
FOR UPDATE OF bl SKIP LOCKED
|
||||||
FOR UPDATE SKIP LOCKED
|
|
||||||
)
|
|
||||||
RETURNING t.id, t.ma_id, t.location_raw, t.geocode_tries_llm,
|
|
||||||
(SELECT country FROM bands WHERE ma_id = t.ma_id) AS country
|
|
||||||
"""
|
"""
|
||||||
)
|
)
|
||||||
row = cur.fetchone()
|
row = cur.fetchone()
|
||||||
if not row:
|
if not row:
|
||||||
print("[groq] rien à traiter, attente 120s")
|
print("[groq] rien à traiter, attente 120s")
|
||||||
health_check()
|
time.sleep(120)
|
||||||
_sleep_interruptible(120)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
loc_id, ma_id, location_raw, tries_llm, country = row
|
loc_id, ma_id, location_raw, tries_llm, country = row
|
||||||
|
|
@ -267,14 +207,11 @@ def main():
|
||||||
if not chosen_model:
|
if not chosen_model:
|
||||||
wait = max(5, 60 - (time.monotonic() - min_start))
|
wait = max(5, 60 - (time.monotonic() - min_start))
|
||||||
print(f"[groq] quota atteint, attente {wait:.0f}s")
|
print(f"[groq] quota atteint, attente {wait:.0f}s")
|
||||||
# Relâcher la réservation posée par le claim : sinon la ligne
|
|
||||||
# resterait inutilement décalée de 5 min alors qu'on ne l'a pas
|
|
||||||
# traitée.
|
|
||||||
cur.execute(
|
cur.execute(
|
||||||
"UPDATE band_locations SET geocode_next_at=now(), updated_at=now() WHERE id=%s",
|
"UPDATE band_locations SET geocode_status='llm_needed', updated_at=now() WHERE id=%s",
|
||||||
(loc_id,),
|
(loc_id,),
|
||||||
)
|
)
|
||||||
_sleep_interruptible(wait)
|
time.sleep(wait)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Vérifier llm_cache
|
# Vérifier llm_cache
|
||||||
|
|
@ -340,7 +277,7 @@ def main():
|
||||||
min_count[hit_model] = 9999
|
min_count[hit_model] = 9999
|
||||||
print(f"[groq] rate limit {hit_model}")
|
print(f"[groq] rate limit {hit_model}")
|
||||||
cur.execute(
|
cur.execute(
|
||||||
"UPDATE band_locations SET geocode_next_at=now(), updated_at=now() WHERE id=%s",
|
"UPDATE band_locations SET geocode_status='llm_needed', updated_at=now() WHERE id=%s",
|
||||||
(loc_id,),
|
(loc_id,),
|
||||||
)
|
)
|
||||||
continue
|
continue
|
||||||
|
|
@ -359,28 +296,8 @@ def main():
|
||||||
(new_tries, str(exc)[:300], backoff, loc_id),
|
(new_tries, str(exc)[:300], backoff, loc_id),
|
||||||
)
|
)
|
||||||
print(f"[groq] id={loc_id} error: {exc}")
|
print(f"[groq] id={loc_id} error: {exc}")
|
||||||
except Exception as exc:
|
|
||||||
# requests lève HTTPError / Timeout / ConnectionError, qui
|
|
||||||
# héritent d'OSError et NON de RuntimeError : sans ce filet,
|
|
||||||
# elles s'échappaient de la boucle et tuaient le process — donc
|
|
||||||
# crash-loop du conteneur sur toute panne durable de l'API Groq.
|
|
||||||
# Idem pour un JSON invalide renvoyé par r.json().
|
|
||||||
new_tries = tries_llm + 1
|
|
||||||
backoff = 30 * new_tries
|
|
||||||
cur.execute(
|
|
||||||
"""
|
|
||||||
UPDATE band_locations
|
|
||||||
SET geocode_tries_llm=%s,
|
|
||||||
geocode_error=%s,
|
|
||||||
geocode_next_at=now() + (%s || ' minutes')::interval,
|
|
||||||
updated_at=now()
|
|
||||||
WHERE id=%s
|
|
||||||
""",
|
|
||||||
(new_tries, f"{type(exc).__name__}: {exc}"[:300], backoff, loc_id),
|
|
||||||
)
|
|
||||||
print(f"[groq] id={loc_id} erreur inattendue: {type(exc).__name__}: {exc}")
|
|
||||||
|
|
||||||
_sleep_interruptible(CALL_DELAY)
|
time.sleep(CALL_DELAY)
|
||||||
|
|
||||||
conn.close()
|
conn.close()
|
||||||
|
|
||||||
|
|
|
||||||
133
apps/geocoder/src/health.py
Normal file
133
apps/geocoder/src/health.py
Normal file
|
|
@ -0,0 +1,133 @@
|
||||||
|
"""
|
||||||
|
Contrôle de santé périodique des services de fond.
|
||||||
|
|
||||||
|
Ces services ne sont pas exposés par Traefik : aucune sonde HTTP externe ne peut
|
||||||
|
les atteindre. Sans ce module, un crawler dont FlareSolverr est injoignable ou
|
||||||
|
un worker à court de quota reste muet, et le seul symptôme est l'absence de
|
||||||
|
données nouvelles — qu'il faut remarquer soi-même.
|
||||||
|
|
||||||
|
Chaque sonde renvoie (nom, ok, détail). Le résultat agrégé est écrit dans
|
||||||
|
`service_health` (une ligne par service, écrasée), lu par le dashboard admin.
|
||||||
|
|
||||||
|
Aucune sonde ne peut interrompre le service : toute exception est convertie en
|
||||||
|
échec de sonde. Un contrôle de santé qui fait tomber ce qu'il surveille serait
|
||||||
|
pire que pas de contrôle du tout.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Intervalle minimal entre deux contrôles complets.
|
||||||
|
DEFAULT_INTERVAL_S = 3600
|
||||||
|
|
||||||
|
_last_run = 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def probe(name, fn):
|
||||||
|
"""Exécute une sonde et convertit toute exception en échec."""
|
||||||
|
try:
|
||||||
|
ok, detail = fn()
|
||||||
|
return name, bool(ok), (detail or "")
|
||||||
|
except Exception as e: # noqa: BLE001 - une sonde ne doit jamais propager
|
||||||
|
return name, False, f"{type(e).__name__}: {e}"
|
||||||
|
|
||||||
|
|
||||||
|
def database_probe(get_conn):
|
||||||
|
"""La base répond-elle ?"""
|
||||||
|
def _run():
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute("SELECT 1")
|
||||||
|
cur.fetchone()
|
||||||
|
return True, "connexion établie"
|
||||||
|
return _run
|
||||||
|
|
||||||
|
|
||||||
|
def http_probe(session_get, url, expect_below=500, timeout=10):
|
||||||
|
"""Une dépendance HTTP répond-elle sans erreur serveur ?"""
|
||||||
|
def _run():
|
||||||
|
r = session_get(url, timeout=timeout)
|
||||||
|
code = getattr(r, "status_code", 0)
|
||||||
|
return code < expect_below, f"HTTP {code}"
|
||||||
|
return _run
|
||||||
|
|
||||||
|
|
||||||
|
def freshness_probe(get_conn, sql, max_age_hours, label):
|
||||||
|
"""Les données progressent-elles ?
|
||||||
|
|
||||||
|
`sql` doit renvoyer un unique timestamp (le plus récent). Une base vide
|
||||||
|
(NULL) n'est pas un échec : c'est un état légitime au premier démarrage.
|
||||||
|
"""
|
||||||
|
def _run():
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(sql)
|
||||||
|
row = cur.fetchone()
|
||||||
|
latest = row[0] if row else None
|
||||||
|
if latest is None:
|
||||||
|
return True, f"{label}: aucune donnée pour l'instant"
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"SELECT EXTRACT(EPOCH FROM (now() - %s)) / 3600.0", (latest,)
|
||||||
|
)
|
||||||
|
age_h = float(cur.fetchone()[0])
|
||||||
|
return age_h <= max_age_hours, f"{label}: {age_h:.1f} h (seuil {max_age_hours} h)"
|
||||||
|
return _run
|
||||||
|
|
||||||
|
|
||||||
|
def write_health(get_conn, service, results):
|
||||||
|
"""Enregistre le résultat agrégé. N'échoue jamais bruyamment."""
|
||||||
|
import json
|
||||||
|
|
||||||
|
checks = {name: ok for name, ok, _ in results}
|
||||||
|
failed = [f"{name} — {detail}" for name, ok, detail in results if not ok]
|
||||||
|
ok = not failed
|
||||||
|
try:
|
||||||
|
with get_conn() as conn:
|
||||||
|
with conn.cursor() as cur:
|
||||||
|
cur.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO service_health (service, ok, checks, error, checked_at)
|
||||||
|
VALUES (%s, %s, %s::jsonb, %s, now())
|
||||||
|
ON CONFLICT (service) DO UPDATE SET
|
||||||
|
ok = EXCLUDED.ok,
|
||||||
|
checks = EXCLUDED.checks,
|
||||||
|
error = EXCLUDED.error,
|
||||||
|
checked_at = EXCLUDED.checked_at
|
||||||
|
""",
|
||||||
|
(service, ok, json.dumps(checks), failed[0] if failed else None),
|
||||||
|
)
|
||||||
|
except Exception as e: # noqa: BLE001
|
||||||
|
log.warning(f"[health] écriture impossible: {e}")
|
||||||
|
return ok, failed
|
||||||
|
|
||||||
|
|
||||||
|
def run_checks(get_conn, service, probes, log_event=None):
|
||||||
|
"""Exécute toutes les sondes, enregistre le résultat, journalise les échecs."""
|
||||||
|
results = [probe(name, fn) for name, fn in probes]
|
||||||
|
ok, failed = write_health(get_conn, service, results)
|
||||||
|
|
||||||
|
if failed:
|
||||||
|
msg = f"[health:{service}] DÉGRADÉ — " + " | ".join(failed)
|
||||||
|
log.warning(msg)
|
||||||
|
if log_event:
|
||||||
|
try:
|
||||||
|
log_event("warning", msg)
|
||||||
|
except Exception: # noqa: BLE001, S110 - journaliser un échec de
|
||||||
|
# journalisation n'apporte rien et risquerait une récursion.
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
log.info(f"[health:{service}] toutes les sondes sont au vert")
|
||||||
|
return ok
|
||||||
|
|
||||||
|
|
||||||
|
def maybe_run(get_conn, service, probes, interval_s=DEFAULT_INTERVAL_S, log_event=None, force=False):
|
||||||
|
"""Point d'entrée depuis une boucle de service : ne contrôle qu'une fois par intervalle."""
|
||||||
|
global _last_run
|
||||||
|
now = time.monotonic()
|
||||||
|
if not force and _last_run and (now - _last_run) < interval_s:
|
||||||
|
return None
|
||||||
|
_last_run = now
|
||||||
|
return run_checks(get_conn, service, probes, log_event=log_event)
|
||||||
|
|
@ -77,17 +77,13 @@ COUNTRY_NAME_TO_ISO2 = {
|
||||||
'north macedonia': 'MK', 'macedonia': 'MK', 'montenegro': 'ME',
|
'north macedonia': 'MK', 'macedonia': 'MK', 'montenegro': 'ME',
|
||||||
'kosovo': 'XK', 'san marino': 'SM', 'liechtenstein': 'LI',
|
'kosovo': 'XK', 'san marino': 'SM', 'liechtenstein': 'LI',
|
||||||
'monaco': 'MC', 'andorra': 'AD', 'vatican': 'VA',
|
'monaco': 'MC', 'andorra': 'AD', 'vatican': 'VA',
|
||||||
# Alias non standards fréquents sur Metal Archives
|
|
||||||
'uk': 'GB', 'great britain': 'GB', 'holland': 'NL', 'czechia': 'CZ',
|
|
||||||
}
|
}
|
||||||
|
|
||||||
COUNTRY_NAMES = {v: k.title() for k, v in COUNTRY_NAME_TO_ISO2.items() if len(v) == 2}
|
COUNTRY_NAMES = {v: k.title() for k, v in COUNTRY_NAME_TO_ISO2.items() if len(v) == 2}
|
||||||
# Plusieurs libellés pointent vers le même ISO2 ; l'inversion ci-dessus garde le
|
# Fix doublons (en → GB wins "United Kingdom")
|
||||||
# DERNIER rencontré, qui n'est pas forcément le nom canonique. On réimpose donc
|
|
||||||
# explicitement le nom à envoyer au géocodeur pour chaque code concerné.
|
|
||||||
COUNTRY_NAMES.update({
|
COUNTRY_NAMES.update({
|
||||||
'GB': 'United Kingdom', 'CZ': 'Czech Republic', 'BA': 'Bosnia and Herzegovina',
|
'GB': 'United Kingdom', 'CZ': 'Czech Republic', 'BA': 'Bosnia and Herzegovina',
|
||||||
'MK': 'North Macedonia', 'XK': 'Kosovo', 'NL': 'Netherlands',
|
'MK': 'North Macedonia', 'XK': 'Kosovo',
|
||||||
})
|
})
|
||||||
|
|
||||||
_ISO2_RE = re.compile(r'^[A-Z]{2}$')
|
_ISO2_RE = re.compile(r'^[A-Z]{2}$')
|
||||||
|
|
@ -197,18 +193,9 @@ def build_fallback_queries(location_raw: str, country_code: str | None) -> list[
|
||||||
seen.add(q)
|
seen.add(q)
|
||||||
result.append(q)
|
result.append(q)
|
||||||
|
|
||||||
# Le code pays doit être cherché comme MOT ENTIER, pas comme sous-chaîne.
|
|
||||||
# `"DE" in "DRESDEN"` est vrai : le pays était donc considéré comme déjà
|
|
||||||
# présent, et la variante « ville, pays » se retrouvait reléguée APRÈS la
|
|
||||||
# ville nue. Or le worker s'arrête au premier résultat fiable — il
|
|
||||||
# interrogeait donc Geoapify sans aucun contexte pays, précisément sur les
|
|
||||||
# noms ambigus (il existe un Dresden dans l'Ohio). Le cas touchait environ
|
|
||||||
# une ville européenne sur quatre : Dresden/Dessau/Detmold (DE),
|
|
||||||
# Fresnes/Fréjus (FR), Notodden/Nordfjord (NO), Cáceres/Torres (ES)…
|
|
||||||
tokens = set(re.findall(r"[A-Za-z]+", location_raw.upper()))
|
|
||||||
has_country = bool(country_name) and (
|
has_country = bool(country_name) and (
|
||||||
country_name.lower() in location_raw.lower()
|
country_name.lower() in location_raw.lower()
|
||||||
or (country_code and country_code.upper() in tokens)
|
or (country_code and country_code.upper() in location_raw.upper())
|
||||||
)
|
)
|
||||||
|
|
||||||
# Requête complète + pays (si pays pas déjà dans le texte)
|
# Requête complète + pays (si pays pas déjà dans le texte)
|
||||||
|
|
|
||||||
|
|
@ -14,61 +14,19 @@ Stratégie :
|
||||||
4. Succès → band_locations + sync bands.lat/lon (origine).
|
4. Succès → band_locations + sync bands.lat/lon (origine).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import contextlib
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import random
|
import random
|
||||||
import signal
|
|
||||||
import time
|
import time
|
||||||
|
|
||||||
import psycopg2
|
import psycopg2
|
||||||
import requests
|
import requests
|
||||||
from bm_health import database_probe, freshness_probe, http_probe, maybe_run
|
from health import database_probe, freshness_probe, http_probe, maybe_run
|
||||||
from parser import build_fallback_queries
|
from parser import build_fallback_queries
|
||||||
|
|
||||||
GEOAPIFY_API_KEY = os.environ.get("GEOAPIFY_API_KEY", "").strip()
|
GEOAPIFY_API_KEY = os.environ.get("GEOAPIFY_API_KEY", "").strip()
|
||||||
GEOAPIFY_BASE = "https://api.geoapify.com/v1/geocode/search"
|
GEOAPIFY_BASE = "https://api.geoapify.com/v1/geocode/search"
|
||||||
|
|
||||||
# Délai au-delà duquel une ligne restée 'processing' est considérée orpheline
|
|
||||||
# (worker tué en plein traitement) et remise en file.
|
|
||||||
STUCK_PROCESSING_MIN = int(os.environ.get("GEOCODE_STUCK_PROCESSING_MIN", "10"))
|
|
||||||
|
|
||||||
# Arrêt propre sur SIGTERM/SIGINT (les redeploys Coolify sont fréquents) : on
|
|
||||||
# termine l'itération en cours au lieu de laisser une ligne en 'processing'.
|
|
||||||
_shutdown = False
|
|
||||||
|
|
||||||
|
|
||||||
def _handle_shutdown(signum, frame):
|
|
||||||
global _shutdown
|
|
||||||
_shutdown = True
|
|
||||||
print(f"[worker] signal {signum} reçu — arrêt propre après l'itération courante")
|
|
||||||
|
|
||||||
|
|
||||||
def _scrub(msg: str) -> str:
|
|
||||||
"""Ne jamais laisser la clé API dans un message d'erreur.
|
|
||||||
|
|
||||||
Ces messages atterrissent dans band_locations.geocode_error, que le
|
|
||||||
dashboard admin affiche. Tronque aussi à 300 caractères.
|
|
||||||
"""
|
|
||||||
if GEOAPIFY_API_KEY and GEOAPIFY_API_KEY in msg:
|
|
||||||
msg = msg.replace(GEOAPIFY_API_KEY, "***")
|
|
||||||
return msg[:300]
|
|
||||||
|
|
||||||
|
|
||||||
@contextlib.contextmanager
|
|
||||||
def _tx(conn):
|
|
||||||
"""Transaction explicite ponctuelle malgré autocommit=True : rend atomique
|
|
||||||
un groupe d'UPDATE liés (band_locations + bands)."""
|
|
||||||
conn.autocommit = False
|
|
||||||
try:
|
|
||||||
yield
|
|
||||||
conn.commit()
|
|
||||||
except Exception:
|
|
||||||
conn.rollback()
|
|
||||||
raise
|
|
||||||
finally:
|
|
||||||
conn.autocommit = True
|
|
||||||
|
|
||||||
MIN_DELAY = float(os.environ.get("GEOCODER_MIN_DELAY", "0.22"))
|
MIN_DELAY = float(os.environ.get("GEOCODER_MIN_DELAY", "0.22"))
|
||||||
JITTER = float(os.environ.get("GEOCODER_JITTER", "0.10"))
|
JITTER = float(os.environ.get("GEOCODER_JITTER", "0.10"))
|
||||||
MAX_PER_RUN = int(os.environ.get("GEOCODE_MAX_PER_RUN", "100000"))
|
MAX_PER_RUN = int(os.environ.get("GEOCODE_MAX_PER_RUN", "100000"))
|
||||||
|
|
@ -95,17 +53,10 @@ def is_reliable(confidence, granularity, is_country_only) -> bool:
|
||||||
|
|
||||||
On exige STRICTEMENT plus que le seuil : 0.7 et en-dessous → rejeté (LLM).
|
On exige STRICTEMENT plus que le seuil : 0.7 et en-dessous → rejeté (LLM).
|
||||||
"""
|
"""
|
||||||
# La confiance est TOUJOURS exigée, y compris pour un lieu "pays seul". Une
|
if is_country_only:
|
||||||
# ligne is_country_only est censée avoir été résolue par centroïde dès
|
return True # résolu par centroïde à l'enqueue, granularité pays assumée
|
||||||
# l'enqueue ; si elle atterrit ici, c'est que le centroïde a échoué (code non
|
|
||||||
# standard type "UK"). On ne peut donc PAS lui faire confiance aveuglément,
|
|
||||||
# sous peine de poser un point aberrant sur la carte.
|
|
||||||
if confidence is None or confidence <= MIN_CONFIDENCE:
|
if confidence is None or confidence <= MIN_CONFIDENCE:
|
||||||
return False
|
return False
|
||||||
if is_country_only:
|
|
||||||
# Un résultat "pays" est légitimement grossier : on saute le rejet de
|
|
||||||
# granularité, mais la confiance vient bien d'être vérifiée.
|
|
||||||
return True
|
|
||||||
if granularity and granularity.lower() in COARSE_TYPES:
|
if granularity and granularity.lower() in COARSE_TYPES:
|
||||||
return False
|
return False
|
||||||
return True
|
return True
|
||||||
|
|
@ -116,18 +67,10 @@ def geoapify_search(query: str) -> dict | None:
|
||||||
raise RuntimeError("GEOAPIFY_API_KEY not set")
|
raise RuntimeError("GEOAPIFY_API_KEY not set")
|
||||||
params = {"text": query, "apiKey": GEOAPIFY_API_KEY, "limit": 1}
|
params = {"text": query, "apiKey": GEOAPIFY_API_KEY, "limit": 1}
|
||||||
_polite_sleep()
|
_polite_sleep()
|
||||||
# La clé part en query string : toute exception requests (raise_for_status,
|
r = requests.get(GEOAPIFY_BASE, params=params, timeout=30)
|
||||||
# ConnectionError, Timeout…) embarque l'URL COMPLÈTE, donc la clé. Ces
|
|
||||||
# messages finissent dans band_locations.geocode_error, affiché par l'admin.
|
|
||||||
# On les remplace donc par une erreur propre, sans URL.
|
|
||||||
try:
|
|
||||||
r = requests.get(GEOAPIFY_BASE, params=params, timeout=30)
|
|
||||||
except requests.RequestException as e:
|
|
||||||
raise RuntimeError(f"geoapify_network:{type(e).__name__}") from None
|
|
||||||
if r.status_code == 429:
|
if r.status_code == 429:
|
||||||
raise RuntimeError("geoapify_throttle:429")
|
raise RuntimeError("geoapify_throttle:429")
|
||||||
if r.status_code != 200:
|
r.raise_for_status()
|
||||||
raise RuntimeError(f"geoapify_http:{r.status_code}")
|
|
||||||
data = r.json()
|
data = r.json()
|
||||||
features = data.get("features") or []
|
features = data.get("features") or []
|
||||||
if not features:
|
if not features:
|
||||||
|
|
@ -228,30 +171,7 @@ def try_dedup(cur, loc_id, ma_id, location_raw, country) -> bool:
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
def recover_stuck_processing(cur) -> int:
|
|
||||||
"""Remet en file toute ligne 'processing' orpheline.
|
|
||||||
|
|
||||||
Sans ça, un worker tué en plein traitement (redeploy, OOM) laisse sa ligne
|
|
||||||
bloquée pour toujours : rien d'autre ne relit ce statut.
|
|
||||||
"""
|
|
||||||
cur.execute(
|
|
||||||
"""
|
|
||||||
UPDATE band_locations
|
|
||||||
SET geocode_status='queued', updated_at=now()
|
|
||||||
WHERE geocode_status='processing'
|
|
||||||
AND updated_at < now() - make_interval(mins => %s)
|
|
||||||
""",
|
|
||||||
(STUCK_PROCESSING_MIN,),
|
|
||||||
)
|
|
||||||
if cur.rowcount:
|
|
||||||
print(f"[worker] {cur.rowcount} ligne(s) 'processing' bloquée(s) remise(s) en file")
|
|
||||||
return cur.rowcount
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
signal.signal(signal.SIGTERM, _handle_shutdown)
|
|
||||||
signal.signal(signal.SIGINT, _handle_shutdown)
|
|
||||||
|
|
||||||
dsn = os.environ["DATABASE_URL"]
|
dsn = os.environ["DATABASE_URL"]
|
||||||
conn = psycopg2.connect(dsn)
|
conn = psycopg2.connect(dsn)
|
||||||
conn.autocommit = True
|
conn.autocommit = True
|
||||||
|
|
@ -289,52 +209,36 @@ def main():
|
||||||
|
|
||||||
processed = 0
|
processed = 0
|
||||||
with conn.cursor() as cur:
|
with conn.cursor() as cur:
|
||||||
# Au démarrage : récupérer les lignes laissées 'processing' par une
|
while processed < MAX_PER_RUN:
|
||||||
# instance précédente tuée brutalement.
|
|
||||||
recover_stuck_processing(cur)
|
|
||||||
|
|
||||||
while processed < MAX_PER_RUN and not _shutdown:
|
|
||||||
# Claim ATOMIQUE : la sélection et le passage en 'processing' sont un
|
|
||||||
# SEUL statement. En autocommit, un `SELECT … FOR UPDATE` séparé
|
|
||||||
# relâche son verrou avant l'UPDATE suivant — deux répliques
|
|
||||||
# pouvaient alors réclamer la même ligne et payer deux appels API.
|
|
||||||
cur.execute(
|
cur.execute(
|
||||||
"""
|
"""
|
||||||
UPDATE band_locations AS t
|
SELECT bl.id, bl.ma_id, bl.location_raw, bl.is_country_only,
|
||||||
SET geocode_status='processing', updated_at=now()
|
bl.geocode_tries_geo, bl.geocode_query, b.country
|
||||||
WHERE t.id = (
|
FROM band_locations bl
|
||||||
SELECT bl.id
|
JOIN bands b ON b.ma_id = bl.ma_id
|
||||||
FROM band_locations bl
|
WHERE bl.geocode_status = 'queued'
|
||||||
WHERE bl.geocode_status = 'queued'
|
AND bl.geocode_next_at <= now()
|
||||||
AND bl.geocode_next_at <= now()
|
ORDER BY bl.geocode_next_at ASC, bl.id ASC
|
||||||
ORDER BY bl.geocode_next_at ASC, bl.id ASC
|
LIMIT 1
|
||||||
LIMIT 1
|
FOR UPDATE OF bl SKIP LOCKED
|
||||||
FOR UPDATE SKIP LOCKED
|
|
||||||
)
|
|
||||||
RETURNING t.id, t.ma_id, t.location_raw, t.is_country_only,
|
|
||||||
t.geocode_tries_geo, t.geocode_query,
|
|
||||||
(SELECT country FROM bands WHERE ma_id = t.ma_id) AS country
|
|
||||||
"""
|
"""
|
||||||
)
|
)
|
||||||
row = cur.fetchone()
|
row = cur.fetchone()
|
||||||
if not row:
|
if not row:
|
||||||
# File vide : profiter de l'inactivité pour récupérer les lignes
|
|
||||||
# 'processing' bloquées, puis attendre.
|
|
||||||
recover_stuck_processing(cur)
|
|
||||||
print("[worker] rien à traiter, attente 60s")
|
print("[worker] rien à traiter, attente 60s")
|
||||||
health_check()
|
health_check()
|
||||||
for _ in range(60):
|
time.sleep(60)
|
||||||
if _shutdown:
|
|
||||||
break
|
|
||||||
time.sleep(1)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
loc_id, ma_id, location_raw, is_country_only, tries, llm_query, country = row
|
loc_id, ma_id, location_raw, is_country_only, tries, llm_query, country = row
|
||||||
|
|
||||||
# 0. Fast-path dedup (aucun appel API) — atomique (band_locations + bands)
|
cur.execute(
|
||||||
with _tx(conn):
|
"UPDATE band_locations SET geocode_status='processing', updated_at=now() WHERE id=%s",
|
||||||
deduped = try_dedup(cur, loc_id, ma_id, location_raw, country)
|
(loc_id,),
|
||||||
if deduped:
|
)
|
||||||
|
|
||||||
|
# 0. Fast-path dedup (aucun appel API)
|
||||||
|
if try_dedup(cur, loc_id, ma_id, location_raw, country):
|
||||||
processed += 1
|
processed += 1
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
|
@ -368,10 +272,6 @@ def main():
|
||||||
(query,),
|
(query,),
|
||||||
)
|
)
|
||||||
cached = cur.fetchone()
|
cached = cur.fetchone()
|
||||||
if cached and cached[0] is None:
|
|
||||||
# Connu, et connu SANS résultat : inutile de rappeler l'API.
|
|
||||||
last_err = f"no_result_cache:'{query}'"
|
|
||||||
continue
|
|
||||||
if cached and cached[0] is not None:
|
if cached and cached[0] is not None:
|
||||||
c_lat, c_lon, c_conf, c_gran = cached
|
c_lat, c_lon, c_conf, c_gran = cached
|
||||||
if is_reliable(c_conf, c_gran, is_country_only):
|
if is_reliable(c_conf, c_gran, is_country_only):
|
||||||
|
|
@ -422,23 +322,6 @@ def main():
|
||||||
break
|
break
|
||||||
last_err = f"low_conf:{r_conf}:{r_gran}:'{query}'"
|
last_err = f"low_conf:{r_conf}:{r_gran}:'{query}'"
|
||||||
else:
|
else:
|
||||||
# Résultat vide mémorisé lui aussi (lat/lon NULL). Le
|
|
||||||
# commentaire ci-dessus disait « toujours mettre en cache »
|
|
||||||
# mais l'insertion était à l'intérieur du `if res:` : une
|
|
||||||
# requête sans résultat était donc re-payée à chaque
|
|
||||||
# tentative — jusqu'à MAX_GEO_TRIES passages, multipliés
|
|
||||||
# par les requêtes de repli, puis de nouveau après chaque
|
|
||||||
# aller-retour LLM qui remet les compteurs à zéro.
|
|
||||||
cur.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO geocode_cache
|
|
||||||
(query, provider, lat, lon, raw, confidence, granularity, updated_at)
|
|
||||||
VALUES (%s,'geoapify',NULL,NULL,%s,NULL,NULL,now())
|
|
||||||
ON CONFLICT (query) DO UPDATE
|
|
||||||
SET raw = EXCLUDED.raw, updated_at = now()
|
|
||||||
""",
|
|
||||||
(query, json.dumps({"features": []})),
|
|
||||||
)
|
|
||||||
last_err = f"no_result:'{query}'"
|
last_err = f"no_result:'{query}'"
|
||||||
except RuntimeError as exc:
|
except RuntimeError as exc:
|
||||||
if "429" in str(exc):
|
if "429" in str(exc):
|
||||||
|
|
@ -456,17 +339,16 @@ def main():
|
||||||
rate_limit = True
|
rate_limit = True
|
||||||
time.sleep(60)
|
time.sleep(60)
|
||||||
break
|
break
|
||||||
last_err = _scrub(str(exc))
|
last_err = str(exc)[:300]
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
last_err = _scrub(str(exc))
|
last_err = str(exc)[:300]
|
||||||
|
|
||||||
if rate_limit:
|
if rate_limit:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if success:
|
if success:
|
||||||
with _tx(conn):
|
mark_done(cur, loc_id, ma_id, lat, lon, used_query,
|
||||||
mark_done(cur, loc_id, ma_id, lat, lon, used_query,
|
provider, confidence, granularity)
|
||||||
provider, confidence, granularity)
|
|
||||||
else:
|
else:
|
||||||
new_tries = tries + 1
|
new_tries = tries + 1
|
||||||
if new_tries >= MAX_GEO_TRIES:
|
if new_tries >= MAX_GEO_TRIES:
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,3 @@
|
||||||
# Le module de sante est partage entre les apps (libs/bm_health.py) : en
|
|
||||||
# Docker il arrive via PYTHONPATH, ici on l'ajoute au chemin d'import.
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "libs"))
|
|
||||||
|
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
|
||||||
|
|
@ -132,30 +132,6 @@ class TestBuildFallbackQueries:
|
||||||
queries = build_fallback_queries("Bergen, Norway", "NO")
|
queries = build_fallback_queries("Bergen, Norway", "NO")
|
||||||
assert not any(q.count("Norway") > 1 for q in queries)
|
assert not any(q.count("Norway") > 1 for q in queries)
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("ville", "code", "pays"),
|
|
||||||
[
|
|
||||||
("Dresden", "DE", "Germany"), # "DE" est une sous-chaîne de "DRESDEN"
|
|
||||||
("Dessau", "DE", "Germany"),
|
|
||||||
("Fresnes", "FR", "France"),
|
|
||||||
("Notodden", "NO", "Norway"),
|
|
||||||
("Torres", "ES", "Spain"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_le_contexte_pays_passe_en_premier(self, ville, code, pays):
|
|
||||||
# Le worker s'arrête au PREMIER résultat fiable : l'ordre est la seule
|
|
||||||
# chose qui compte. Un code pays contenu par hasard dans le nom de la
|
|
||||||
# ville faisait passer la requête nue devant, supprimant toute
|
|
||||||
# désambiguïsation là où elle était le plus nécessaire.
|
|
||||||
assert build_fallback_queries(ville, code)[0] == f"{ville}, {pays}"
|
|
||||||
|
|
||||||
def test_le_code_pays_reellement_present_reste_prioritaire(self):
|
|
||||||
# Contrepartie : quand le code EST un mot du texte, le texte d'origine
|
|
||||||
# passe en premier — on ne le double pas d'un contexte redondant. La
|
|
||||||
# forme « ville, pays » reste en dernier recours, ce qui est voulu.
|
|
||||||
queries = build_fallback_queries("Berlin, DE", "DE")
|
|
||||||
assert queries[0] == "Berlin, DE"
|
|
||||||
|
|
||||||
def test_pays_inconnu_ne_plante_pas(self):
|
def test_pays_inconnu_ne_plante_pas(self):
|
||||||
assert build_fallback_queries("Bergen", "ZZ") == ["Bergen"]
|
assert build_fallback_queries("Bergen", "ZZ") == ["Bergen"]
|
||||||
assert build_fallback_queries("Bergen", None) == ["Bergen"]
|
assert build_fallback_queries("Bergen", None) == ["Bergen"]
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,4 @@
|
||||||
FROM nginx:1.27-alpine
|
FROM nginx:alpine
|
||||||
ARG API_BASE=https://bm.nicolasfryder.ovh
|
ARG API_BASE=https://bm.nicolasfryder.ovh
|
||||||
COPY site/ /usr/share/nginx/html
|
COPY site/ /usr/share/nginx/html
|
||||||
COPY nginx.conf /etc/nginx/conf.d/default.conf
|
|
||||||
RUN sed -i "s|https://bm.nicolasfryder.ovh|${API_BASE}|g" /usr/share/nginx/html/app.js
|
RUN sed -i "s|https://bm.nicolasfryder.ovh|${API_BASE}|g" /usr/share/nginx/html/app.js
|
||||||
|
|
|
||||||
|
|
@ -1,28 +0,0 @@
|
||||||
server {
|
|
||||||
listen 80;
|
|
||||||
root /usr/share/nginx/html;
|
|
||||||
index index.html;
|
|
||||||
|
|
||||||
server_tokens off;
|
|
||||||
|
|
||||||
# En-têtes de sécurité (l'image nginx par défaut n'en pose aucun).
|
|
||||||
add_header X-Frame-Options "DENY" always;
|
|
||||||
add_header X-Content-Type-Options "nosniff" always;
|
|
||||||
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
|
|
||||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
|
|
||||||
|
|
||||||
# CSP du site cartographique public : permissive sur script/style/img/connect
|
|
||||||
# (Leaflet, tuiles CartoCDN, analytics) pour ne pas casser la carte, mais
|
|
||||||
# verrouille les sinks dangereux (object-src, base-uri, frame-ancestors). Les
|
|
||||||
# libs CDN sont en plus protégées par SRI (integrity=) dans index.html.
|
|
||||||
#
|
|
||||||
# ⚠ connect-src liste les hôtes d'API EN DUR (dev + prod), alors que l'URL
|
|
||||||
# utilisée par app.js est substituée au build via ARG API_BASE. Ajouter un
|
|
||||||
# nouvel environnement impose donc de compléter cette liste, sinon le
|
|
||||||
# navigateur bloquera silencieusement tous les appels API.
|
|
||||||
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' https://unpkg.com https://gc.zgo.at; style-src 'self' 'unsafe-inline' https://unpkg.com https://cdn.jsdelivr.net https://fonts.googleapis.com; font-src 'self' https://fonts.gstatic.com https://cdn.jsdelivr.net data:; img-src 'self' data: https:; connect-src 'self' https://bm.nicolasfryder.ovh https://dev-api.metalfrom.eu https://metalfromeurope.goatcounter.com; object-src 'none'; base-uri 'self'; frame-ancestors 'none'" always;
|
|
||||||
|
|
||||||
location / {
|
|
||||||
try_files $uri $uri/ /index.html;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -173,19 +173,11 @@ const MACRO_GENRES = [
|
||||||
];
|
];
|
||||||
|
|
||||||
// --- API ---
|
// --- API ---
|
||||||
async function apiGet(path, timeoutMs = 15000) {
|
async function apiGet(path) {
|
||||||
const url = `${API_BASE}${path}`;
|
const url = `${API_BASE}${path}`;
|
||||||
// Timeout explicite : sans lui, une API qui ne répond jamais laisse l'UI (et
|
const r = await fetch(url, { mode: "cors" });
|
||||||
// le rechargement du viewport de la carte) bloquée indéfiniment.
|
if (!r.ok) throw new Error(`API ${path} failed: ${r.status}`);
|
||||||
const ctrl = new AbortController();
|
return await r.json();
|
||||||
const t = setTimeout(() => ctrl.abort(), timeoutMs);
|
|
||||||
try {
|
|
||||||
const r = await fetch(url, { mode: "cors", signal: ctrl.signal });
|
|
||||||
if (!r.ok) throw new Error(`API ${path} failed: ${r.status}`);
|
|
||||||
return await r.json();
|
|
||||||
} finally {
|
|
||||||
clearTimeout(t);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
async function loadStats() {
|
async function loadStats() {
|
||||||
|
|
@ -343,15 +335,17 @@ async function loadViewportBands() {
|
||||||
const bounds = map.getBounds();
|
const bounds = map.getBounds();
|
||||||
const zoom = map.getZoom();
|
const zoom = map.getZoom();
|
||||||
|
|
||||||
// Élargie de 20 % pour précharger les bords, puis bornée : voir
|
// Expand bounds slightly to preload edges
|
||||||
// BMPure.viewportBbox pour le détail (une bbox hors bornes fait répondre
|
const expandFactor = 0.2;
|
||||||
// 400 à l'API, et la carte reste vide sans message).
|
const latDiff = (bounds.getNorth() - bounds.getSouth()) * expandFactor;
|
||||||
const { minLon, minLat, maxLon, maxLat } = BMPure.viewportBbox({
|
const lonDiff = (bounds.getEast() - bounds.getWest()) * expandFactor;
|
||||||
south: bounds.getSouth(), west: bounds.getWest(),
|
|
||||||
north: bounds.getNorth(), east: bounds.getEast(),
|
|
||||||
}, 0.2);
|
|
||||||
|
|
||||||
const bbox = [minLon, minLat, maxLon, maxLat].join(",");
|
const bbox = [
|
||||||
|
bounds.getWest() - lonDiff,
|
||||||
|
bounds.getSouth() - latDiff,
|
||||||
|
bounds.getEast() + lonDiff,
|
||||||
|
bounds.getNorth() + latDiff
|
||||||
|
].join(",");
|
||||||
|
|
||||||
// Build query params for filters
|
// Build query params for filters
|
||||||
const params = new URLSearchParams({
|
const params = new URLSearchParams({
|
||||||
|
|
@ -407,18 +401,6 @@ async function loadViewportBands() {
|
||||||
}
|
}
|
||||||
|
|
||||||
$("count").textContent = String(j.count || currentViewportData.length || 0);
|
$("count").textContent = String(j.count || currentViewportData.length || 0);
|
||||||
|
|
||||||
// Le serveur plafonne sa réponse (2000 groupes, 1000 clusters). Il le dit
|
|
||||||
// désormais ; sans cet indicateur, des groupes disparaissaient de la carte
|
|
||||||
// dans les zones denses sans que rien ne le signale.
|
|
||||||
const note = $("truncNote");
|
|
||||||
if (note) {
|
|
||||||
note.hidden = !j.truncated;
|
|
||||||
if (j.truncated) {
|
|
||||||
note.textContent = "⚠";
|
|
||||||
note.title = t("truncated_hint").replace("{n}", String(j.limit ?? ""));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
console.error("Failed to load viewport bands:", e);
|
console.error("Failed to load viewport bands:", e);
|
||||||
|
|
@ -1199,55 +1181,9 @@ async function loadNoLocationBands() {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ------------------------------------------------------------------
|
|
||||||
// Modales — focus
|
|
||||||
//
|
|
||||||
// Les deux modales déclarent aria-modal="true", ce qui affirme aux lecteurs
|
|
||||||
// d'écran que tout le reste de la page est inerte. Sans piège à focus, c'était
|
|
||||||
// faux : la tabulation ressortait derrière le voile, sur des commandes
|
|
||||||
// invisibles, et la fermeture ne rendait pas le focus à son point de départ.
|
|
||||||
// Même traitement que le dashboard admin, qui le faisait déjà correctement.
|
|
||||||
// ------------------------------------------------------------------
|
|
||||||
const FOCUSABLES = 'button, [href], input, select, textarea, summary, [tabindex]:not([tabindex="-1"])';
|
|
||||||
let lastFocusedBeforeModal = null;
|
|
||||||
|
|
||||||
function trapModalFocus(backdrop) {
|
|
||||||
if (!backdrop || backdrop.__trap) return;
|
|
||||||
const onKeydown = (e) => {
|
|
||||||
if (e.key !== "Tab") return;
|
|
||||||
const f = backdrop.querySelectorAll(FOCUSABLES);
|
|
||||||
if (!f.length) return;
|
|
||||||
const first = f[0];
|
|
||||||
const last = f[f.length - 1];
|
|
||||||
if (e.shiftKey && document.activeElement === first) { e.preventDefault(); last.focus(); }
|
|
||||||
else if (!e.shiftKey && document.activeElement === last) { e.preventDefault(); first.focus(); }
|
|
||||||
};
|
|
||||||
backdrop.addEventListener("keydown", onKeydown);
|
|
||||||
backdrop.__trap = onKeydown;
|
|
||||||
}
|
|
||||||
|
|
||||||
function openBackdrop(backdrop) {
|
|
||||||
if (!backdrop) return;
|
|
||||||
lastFocusedBeforeModal = document.activeElement;
|
|
||||||
backdrop.classList.add("on");
|
|
||||||
backdrop.setAttribute("aria-hidden", "false");
|
|
||||||
trapModalFocus(backdrop);
|
|
||||||
const f = backdrop.querySelectorAll(FOCUSABLES);
|
|
||||||
if (f.length) f[0].focus();
|
|
||||||
}
|
|
||||||
|
|
||||||
function closeBackdrop(backdrop) {
|
|
||||||
if (!backdrop || !backdrop.classList.contains("on")) return;
|
|
||||||
backdrop.classList.remove("on");
|
|
||||||
backdrop.setAttribute("aria-hidden", "true");
|
|
||||||
if (lastFocusedBeforeModal && document.contains(lastFocusedBeforeModal)) {
|
|
||||||
lastFocusedBeforeModal.focus();
|
|
||||||
lastFocusedBeforeModal = null;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function closeModal() {
|
function closeModal() {
|
||||||
closeBackdrop(modalBackdrop);
|
modalBackdrop.classList.remove("on");
|
||||||
|
modalBackdrop.setAttribute("aria-hidden", "true");
|
||||||
}
|
}
|
||||||
|
|
||||||
modalClose?.addEventListener("click", closeModal);
|
modalClose?.addEventListener("click", closeModal);
|
||||||
|
|
@ -1267,11 +1203,14 @@ function openInfoModal(title, html) {
|
||||||
if (!infoBackdrop || !infoBody || !infoTitle) return;
|
if (!infoBackdrop || !infoBody || !infoTitle) return;
|
||||||
infoTitle.textContent = title;
|
infoTitle.textContent = title;
|
||||||
infoBody.innerHTML = html;
|
infoBody.innerHTML = html;
|
||||||
openBackdrop(infoBackdrop);
|
infoBackdrop.classList.add("on");
|
||||||
|
infoBackdrop.setAttribute("aria-hidden", "false");
|
||||||
}
|
}
|
||||||
|
|
||||||
function closeInfoModal() {
|
function closeInfoModal() {
|
||||||
closeBackdrop(infoBackdrop);
|
if (!infoBackdrop) return;
|
||||||
|
infoBackdrop.classList.remove("on");
|
||||||
|
infoBackdrop.setAttribute("aria-hidden", "true");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (infoClose) infoClose.addEventListener("click", closeInfoModal);
|
if (infoClose) infoClose.addEventListener("click", closeInfoModal);
|
||||||
|
|
@ -1387,7 +1326,8 @@ setupDropdown("themeToggle", "themeSelect");
|
||||||
// --- Buttons / controls ---
|
// --- Buttons / controls ---
|
||||||
$("btnNoLocation")?.addEventListener("click", () => {
|
$("btnNoLocation")?.addEventListener("click", () => {
|
||||||
modalTitle.textContent = t("without_coords_modal");
|
modalTitle.textContent = t("without_coords_modal");
|
||||||
openBackdrop(modalBackdrop);
|
modalBackdrop.classList.add("on");
|
||||||
|
modalBackdrop.setAttribute("aria-hidden", "false");
|
||||||
loadNoLocationBands();
|
loadNoLocationBands();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -11,35 +11,26 @@
|
||||||
<link href="https://fonts.googleapis.com/css2?family=UnifrakturCook:wght@700&family=Inter:wght@300;400;600;700;900&display=swap" rel="stylesheet">
|
<link href="https://fonts.googleapis.com/css2?family=UnifrakturCook:wght@700&family=Inter:wght@300;400;600;700;900&display=swap" rel="stylesheet">
|
||||||
|
|
||||||
<!-- Leaflet -->
|
<!-- Leaflet -->
|
||||||
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css"
|
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" />
|
||||||
integrity="sha384-sHL9NAb7lN7rfvG5lfHpm643Xkcjzp4jFvuavGOndn6pjVqS6ny56CAt3nsEVT4H" crossorigin="anonymous" />
|
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"></script>
|
||||||
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js"
|
|
||||||
integrity="sha384-cxOPjt7s7Iz04uaHJceBmS+qpjv2JkIHNVcuOrM+YHwZOmJGBXI00mdUXEq65HTH" crossorigin="anonymous"></script>
|
|
||||||
|
|
||||||
<!-- MarkerCluster -->
|
<!-- MarkerCluster -->
|
||||||
<link rel="stylesheet" href="https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css"
|
<link rel="stylesheet" href="https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.css"/>
|
||||||
integrity="sha384-pmjIAcz2bAn0xukfxADbZIb3t8oRT9Sv0rvO+BR5Csr6Dhqq+nZs59P0pPKQJkEV" crossorigin="anonymous"/>
|
<link rel="stylesheet" href="https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.Default.css"/>
|
||||||
<link rel="stylesheet" href="https://unpkg.com/leaflet.markercluster@1.5.3/dist/MarkerCluster.Default.css"
|
<script src="https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js"></script>
|
||||||
integrity="sha384-wgw+aLYNQ7dlhK47ZPK7FRACiq7ROZwgFNg0m04avm4CaXS+Z9Y7nMu8yNjBKYC+" crossorigin="anonymous"/>
|
|
||||||
<script src="https://unpkg.com/leaflet.markercluster@1.5.3/dist/leaflet.markercluster.js"
|
|
||||||
integrity="sha384-eXVCORTRlv4FUUgS/xmOyr66XBVraen8ATNLMESp92FKXLAMiKkerixTiBvXriZr" crossorigin="anonymous"></script>
|
|
||||||
|
|
||||||
<!-- Heatmap -->
|
<!-- Heatmap -->
|
||||||
<script src="https://unpkg.com/leaflet.heat@0.2.0/dist/leaflet-heat.js"
|
<script src="https://unpkg.com/leaflet.heat@0.2.0/dist/leaflet-heat.js"></script>
|
||||||
integrity="sha384-mFKkGiGvT5vo1fEyGCD3hshDdKmW3wzXW/x+fWriYJArD0R3gawT6lMvLboM22c0" crossorigin="anonymous"></script>
|
|
||||||
|
|
||||||
<!-- noUiSlider (double slider timeline) -->
|
<!-- noUiSlider (double slider timeline) -->
|
||||||
<link rel="stylesheet" href="https://unpkg.com/nouislider@15.7.1/dist/nouislider.min.css"
|
<link rel="stylesheet" href="https://unpkg.com/nouislider@15.7.1/dist/nouislider.min.css">
|
||||||
integrity="sha384-PSZaVsyG9jDu8hFaSJev5s/9poIJlX7cuxSGdqCgXRHpo2DzIaZAyCd2rG/DJJmV" crossorigin="anonymous">
|
<script src="https://unpkg.com/nouislider@15.7.1/dist/nouislider.min.js"></script>
|
||||||
<script src="https://unpkg.com/nouislider@15.7.1/dist/nouislider.min.js"
|
|
||||||
integrity="sha384-/gBUOLHADjY2rp6bHB0IyW9AC28q4OsnirJScje4l1crgYW7Qarx3dH8zcqcUgmy" crossorigin="anonymous"></script>
|
|
||||||
|
|
||||||
<!-- GoatCounter (script d'analytics non versionné → pas de SRI possible) -->
|
<!-- GoatCounter -->
|
||||||
<script data-goatcounter="https://metalfromeurope.goatcounter.com/count"
|
<script data-goatcounter="https://metalfromeurope.goatcounter.com/count"
|
||||||
async src="//gc.zgo.at/count.js"></script>
|
async src="//gc.zgo.at/count.js"></script>
|
||||||
|
|
||||||
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/flag-icons@7.2.3/css/flag-icons.min.css"
|
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/flag-icons@7.2.3/css/flag-icons.min.css">
|
||||||
integrity="sha384-aQuvIWWIbpu/mSqULLDiUveyYiPoJzPKAWjUmGJ+Elm+N/LJhzfZqsutsfw870JS" crossorigin="anonymous">
|
|
||||||
<link rel="stylesheet" href="./styles.css" />
|
<link rel="stylesheet" href="./styles.css" />
|
||||||
</head>
|
</head>
|
||||||
|
|
||||||
|
|
@ -113,7 +104,7 @@
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="stats">
|
<div class="stats">
|
||||||
<span class="chip"><span class="dot"></span><b id="count">0</b> <span data-i18n="displayed">affichés</span><span id="truncNote" class="trunc-note" hidden></span></span>
|
<span class="chip"><span class="dot"></span><b id="count">0</b> <span data-i18n="displayed">affichés</span></span>
|
||||||
<span class="chip"><b id="total">0</b> <span data-i18n="total_bands">total</span></span>
|
<span class="chip"><b id="total">0</b> <span data-i18n="total_bands">total</span></span>
|
||||||
<span class="chip"><b id="unique">0</b> <span data-i18n="localities">localités</span></span>
|
<span class="chip"><b id="unique">0</b> <span data-i18n="localities">localités</span></span>
|
||||||
<span class="chip"><b id="geocoded">0</b> <span data-i18n="located">localisés</span></span>
|
<span class="chip"><b id="geocoded">0</b> <span data-i18n="located">localisés</span></span>
|
||||||
|
|
|
||||||
|
|
@ -57,7 +57,6 @@
|
||||||
|
|
||||||
fr: {
|
fr: {
|
||||||
search_placeholder: "Rechercher (nom, genre, ville, pays)…",
|
search_placeholder: "Rechercher (nom, genre, ville, pays)…",
|
||||||
truncated_hint: "Seuls les {n} premiers résultats de cette zone sont affichés — zoomez pour tout voir.",
|
|
||||||
displayed: "affichés", total_bands: "total", localities: "localités", located: "localisés",
|
displayed: "affichés", total_bands: "total", localities: "localités", located: "localisés",
|
||||||
view_section: "Vue", reset_view: "Reset vue", heatmap_label: "Heatmap",
|
view_section: "Vue", reset_view: "Reset vue", heatmap_label: "Heatmap",
|
||||||
filters_section: "Filtres", without_coords_btn: "Sans coords",
|
filters_section: "Filtres", without_coords_btn: "Sans coords",
|
||||||
|
|
@ -96,7 +95,6 @@
|
||||||
|
|
||||||
en: {
|
en: {
|
||||||
search_placeholder: "Search (name, genre, city, country)…",
|
search_placeholder: "Search (name, genre, city, country)…",
|
||||||
truncated_hint: "Only the first {n} results in this area are shown — zoom in to see them all.",
|
|
||||||
displayed: "displayed", total_bands: "total", localities: "localities", located: "located",
|
displayed: "displayed", total_bands: "total", localities: "localities", located: "located",
|
||||||
view_section: "View", reset_view: "Reset view", heatmap_label: "Heatmap",
|
view_section: "View", reset_view: "Reset view", heatmap_label: "Heatmap",
|
||||||
filters_section: "Filters", without_coords_btn: "Without coords",
|
filters_section: "Filters", without_coords_btn: "Without coords",
|
||||||
|
|
@ -134,7 +132,6 @@
|
||||||
|
|
||||||
de: {
|
de: {
|
||||||
search_placeholder: "Suchen (Name, Genre, Stadt, Land)…",
|
search_placeholder: "Suchen (Name, Genre, Stadt, Land)…",
|
||||||
truncated_hint: "Es werden nur die ersten {n} Ergebnisse in diesem Bereich angezeigt — zoomen Sie hinein, um alle zu sehen.",
|
|
||||||
displayed: "angezeigt", total_bands: "gesamt", localities: "Orte", located: "geortet",
|
displayed: "angezeigt", total_bands: "gesamt", localities: "Orte", located: "geortet",
|
||||||
view_section: "Ansicht", reset_view: "Ansicht zurücksetzen", heatmap_label: "Heatmap",
|
view_section: "Ansicht", reset_view: "Ansicht zurücksetzen", heatmap_label: "Heatmap",
|
||||||
filters_section: "Filter", without_coords_btn: "Ohne Koordinaten",
|
filters_section: "Filter", without_coords_btn: "Ohne Koordinaten",
|
||||||
|
|
@ -172,7 +169,6 @@
|
||||||
|
|
||||||
es: {
|
es: {
|
||||||
search_placeholder: "Buscar (nombre, género, ciudad, país)…",
|
search_placeholder: "Buscar (nombre, género, ciudad, país)…",
|
||||||
truncated_hint: "Solo se muestran los primeros {n} resultados de esta zona — amplía para verlos todos.",
|
|
||||||
displayed: "mostrados", total_bands: "total", localities: "localidades", located: "localizados",
|
displayed: "mostrados", total_bands: "total", localities: "localidades", located: "localizados",
|
||||||
view_section: "Vista", reset_view: "Restablecer vista", heatmap_label: "Mapa de calor",
|
view_section: "Vista", reset_view: "Restablecer vista", heatmap_label: "Mapa de calor",
|
||||||
filters_section: "Filtros", without_coords_btn: "Sin coordenadas",
|
filters_section: "Filtros", without_coords_btn: "Sin coordenadas",
|
||||||
|
|
@ -210,7 +206,6 @@
|
||||||
|
|
||||||
it: {
|
it: {
|
||||||
search_placeholder: "Cerca (nome, genere, città, paese)…",
|
search_placeholder: "Cerca (nome, genere, città, paese)…",
|
||||||
truncated_hint: "Vengono mostrati solo i primi {n} risultati di quest'area — ingrandisci per vederli tutti.",
|
|
||||||
displayed: "visualizzati", total_bands: "totale", localities: "località", located: "geolocalizzati",
|
displayed: "visualizzati", total_bands: "totale", localities: "località", located: "geolocalizzati",
|
||||||
view_section: "Vista", reset_view: "Reimposta vista", heatmap_label: "Mappa di calore",
|
view_section: "Vista", reset_view: "Reimposta vista", heatmap_label: "Mappa di calore",
|
||||||
filters_section: "Filtri", without_coords_btn: "Senza coordinate",
|
filters_section: "Filtri", without_coords_btn: "Senza coordinate",
|
||||||
|
|
@ -248,7 +243,6 @@
|
||||||
|
|
||||||
pl: {
|
pl: {
|
||||||
search_placeholder: "Szukaj (nazwa, gatunek, miasto, kraj)…",
|
search_placeholder: "Szukaj (nazwa, gatunek, miasto, kraj)…",
|
||||||
truncated_hint: "Wyświetlono tylko pierwsze {n} wyników w tym obszarze — przybliż, aby zobaczyć wszystkie.",
|
|
||||||
displayed: "wyświetlono", total_bands: "łącznie", localities: "miejscowości", located: "zlokalizowano",
|
displayed: "wyświetlono", total_bands: "łącznie", localities: "miejscowości", located: "zlokalizowano",
|
||||||
view_section: "Widok", reset_view: "Resetuj widok", heatmap_label: "Mapa cieplna",
|
view_section: "Widok", reset_view: "Resetuj widok", heatmap_label: "Mapa cieplna",
|
||||||
filters_section: "Filtry", without_coords_btn: "Bez współrzędnych",
|
filters_section: "Filtry", without_coords_btn: "Bez współrzędnych",
|
||||||
|
|
@ -286,7 +280,6 @@
|
||||||
|
|
||||||
nl: {
|
nl: {
|
||||||
search_placeholder: "Zoeken (naam, genre, stad, land)…",
|
search_placeholder: "Zoeken (naam, genre, stad, land)…",
|
||||||
truncated_hint: "Alleen de eerste {n} resultaten in dit gebied worden getoond — zoom in om ze allemaal te zien.",
|
|
||||||
displayed: "weergegeven", total_bands: "totaal", localities: "locaties", located: "gelokaliseerd",
|
displayed: "weergegeven", total_bands: "totaal", localities: "locaties", located: "gelokaliseerd",
|
||||||
view_section: "Weergave", reset_view: "Weergave herstellen", heatmap_label: "Warmtekaart",
|
view_section: "Weergave", reset_view: "Weergave herstellen", heatmap_label: "Warmtekaart",
|
||||||
filters_section: "Filters", without_coords_btn: "Zonder coördinaten",
|
filters_section: "Filters", without_coords_btn: "Zonder coördinaten",
|
||||||
|
|
@ -324,7 +317,6 @@
|
||||||
|
|
||||||
ro: {
|
ro: {
|
||||||
search_placeholder: "Căutare (nume, gen, oraş, țară)…",
|
search_placeholder: "Căutare (nume, gen, oraş, țară)…",
|
||||||
truncated_hint: "Sunt afișate doar primele {n} rezultate din această zonă — apropie pentru a le vedea pe toate.",
|
|
||||||
displayed: "afişate", total_bands: "total", localities: "localități", located: "localizate",
|
displayed: "afişate", total_bands: "total", localities: "localități", located: "localizate",
|
||||||
view_section: "Vedere", reset_view: "Resetare vedere", heatmap_label: "Hartă termică",
|
view_section: "Vedere", reset_view: "Resetare vedere", heatmap_label: "Hartă termică",
|
||||||
filters_section: "Filtre", without_coords_btn: "Fără coordonate",
|
filters_section: "Filtre", without_coords_btn: "Fără coordonate",
|
||||||
|
|
@ -362,7 +354,6 @@
|
||||||
|
|
||||||
pt: {
|
pt: {
|
||||||
search_placeholder: "Pesquisar (nome, género, cidade, país)…",
|
search_placeholder: "Pesquisar (nome, género, cidade, país)…",
|
||||||
truncated_hint: "Apenas os primeiros {n} resultados desta área são mostrados — amplie para ver todos.",
|
|
||||||
displayed: "exibidos", total_bands: "total", localities: "localidades", located: "localizados",
|
displayed: "exibidos", total_bands: "total", localities: "localidades", located: "localizados",
|
||||||
view_section: "Vista", reset_view: "Repor vista", heatmap_label: "Mapa de calor",
|
view_section: "Vista", reset_view: "Repor vista", heatmap_label: "Mapa de calor",
|
||||||
filters_section: "Filtros", without_coords_btn: "Sem coordenadas",
|
filters_section: "Filtros", without_coords_btn: "Sem coordenadas",
|
||||||
|
|
@ -400,7 +391,6 @@
|
||||||
|
|
||||||
cs: {
|
cs: {
|
||||||
search_placeholder: "Hledat (název, žánr, město, země)…",
|
search_placeholder: "Hledat (název, žánr, město, země)…",
|
||||||
truncated_hint: "Zobrazeno je pouze prvních {n} výsledků v této oblasti — přibližte pro zobrazení všech.",
|
|
||||||
displayed: "zobrazeno", total_bands: "celkem", localities: "míst", located: "lokalizováno",
|
displayed: "zobrazeno", total_bands: "celkem", localities: "míst", located: "lokalizováno",
|
||||||
view_section: "Zobrazení", reset_view: "Obnovit zobrazení", heatmap_label: "Tepelná mapa",
|
view_section: "Zobrazení", reset_view: "Obnovit zobrazení", heatmap_label: "Tepelná mapa",
|
||||||
filters_section: "Filtry", without_coords_btn: "Bez souřadnic",
|
filters_section: "Filtry", without_coords_btn: "Bez souřadnic",
|
||||||
|
|
@ -438,7 +428,6 @@
|
||||||
|
|
||||||
sv: {
|
sv: {
|
||||||
search_placeholder: "Sök (namn, genre, stad, land)…",
|
search_placeholder: "Sök (namn, genre, stad, land)…",
|
||||||
truncated_hint: "Endast de första {n} resultaten i det här området visas — zooma in för att se alla.",
|
|
||||||
displayed: "visade", total_bands: "totalt", localities: "orter", located: "lokaliserade",
|
displayed: "visade", total_bands: "totalt", localities: "orter", located: "lokaliserade",
|
||||||
view_section: "Vy", reset_view: "Återställ vy", heatmap_label: "Värmekarta",
|
view_section: "Vy", reset_view: "Återställ vy", heatmap_label: "Värmekarta",
|
||||||
filters_section: "Filter", without_coords_btn: "Utan koordinater",
|
filters_section: "Filter", without_coords_btn: "Utan koordinater",
|
||||||
|
|
@ -476,7 +465,6 @@
|
||||||
|
|
||||||
hu: {
|
hu: {
|
||||||
search_placeholder: "Keresés (név, műfaj, város, ország)…",
|
search_placeholder: "Keresés (név, műfaj, város, ország)…",
|
||||||
truncated_hint: "Csak az első {n} találat látható ezen a területen — nagyíts rá az összes megtekintéséhez.",
|
|
||||||
displayed: "megjelenített", total_bands: "összesen", localities: "helységek", located: "lokalizált",
|
displayed: "megjelenített", total_bands: "összesen", localities: "helységek", located: "lokalizált",
|
||||||
view_section: "Nézet", reset_view: "Nézet alaphelyzetbe", heatmap_label: "Hőtérkép",
|
view_section: "Nézet", reset_view: "Nézet alaphelyzetbe", heatmap_label: "Hőtérkép",
|
||||||
filters_section: "Szűrők", without_coords_btn: "Koordináták nélkül",
|
filters_section: "Szűrők", without_coords_btn: "Koordináták nélkül",
|
||||||
|
|
@ -514,7 +502,6 @@
|
||||||
|
|
||||||
da: {
|
da: {
|
||||||
search_placeholder: "Søg (navn, genre, by, land)…",
|
search_placeholder: "Søg (navn, genre, by, land)…",
|
||||||
truncated_hint: "Kun de første {n} resultater i dette område vises — zoom ind for at se dem alle.",
|
|
||||||
displayed: "vist", total_bands: "i alt", localities: "lokaliteter", located: "lokaliserede",
|
displayed: "vist", total_bands: "i alt", localities: "lokaliteter", located: "lokaliserede",
|
||||||
view_section: "Visning", reset_view: "Nulstil visning", heatmap_label: "Varmekort",
|
view_section: "Visning", reset_view: "Nulstil visning", heatmap_label: "Varmekort",
|
||||||
filters_section: "Filtre", without_coords_btn: "Uden koordinater",
|
filters_section: "Filtre", without_coords_btn: "Uden koordinater",
|
||||||
|
|
@ -552,7 +539,6 @@
|
||||||
|
|
||||||
fi: {
|
fi: {
|
||||||
search_placeholder: "Haku (nimi, genre, kaupunki, maa)…",
|
search_placeholder: "Haku (nimi, genre, kaupunki, maa)…",
|
||||||
truncated_hint: "Vain tämän alueen {n} ensimmäistä tulosta näytetään — lähennä nähdäksesi kaikki.",
|
|
||||||
displayed: "näytetty", total_bands: "yhteensä", localities: "paikkakunnat", located: "paikannettu",
|
displayed: "näytetty", total_bands: "yhteensä", localities: "paikkakunnat", located: "paikannettu",
|
||||||
view_section: "Näkymä", reset_view: "Palauta näkymä", heatmap_label: "Lämpökartta",
|
view_section: "Näkymä", reset_view: "Palauta näkymä", heatmap_label: "Lämpökartta",
|
||||||
filters_section: "Suodattimet", without_coords_btn: "Ilman koordinaatteja",
|
filters_section: "Suodattimet", without_coords_btn: "Ilman koordinaatteja",
|
||||||
|
|
@ -590,7 +576,6 @@
|
||||||
|
|
||||||
sk: {
|
sk: {
|
||||||
search_placeholder: "Hľadať (názov, žáner, mesto, krajina)…",
|
search_placeholder: "Hľadať (názov, žáner, mesto, krajina)…",
|
||||||
truncated_hint: "Zobrazuje sa len prvých {n} výsledkov v tejto oblasti — priblížte pre zobrazenie všetkých.",
|
|
||||||
displayed: "zobrazené", total_bands: "celkom", localities: "miest", located: "lokalizované",
|
displayed: "zobrazené", total_bands: "celkom", localities: "miest", located: "lokalizované",
|
||||||
view_section: "Zobrazenie", reset_view: "Obnoviť zobrazenie", heatmap_label: "Tepelná mapa",
|
view_section: "Zobrazenie", reset_view: "Obnoviť zobrazenie", heatmap_label: "Tepelná mapa",
|
||||||
filters_section: "Filtre", without_coords_btn: "Bez súradníc",
|
filters_section: "Filtre", without_coords_btn: "Bez súradníc",
|
||||||
|
|
@ -628,7 +613,6 @@
|
||||||
|
|
||||||
hr: {
|
hr: {
|
||||||
search_placeholder: "Pretraži (ime, žanr, grad, država)…",
|
search_placeholder: "Pretraži (ime, žanr, grad, država)…",
|
||||||
truncated_hint: "Prikazano je samo prvih {n} rezultata u ovom području — približite za prikaz svih.",
|
|
||||||
displayed: "prikazano", total_bands: "ukupno", localities: "lokaliteta", located: "locirano",
|
displayed: "prikazano", total_bands: "ukupno", localities: "lokaliteta", located: "locirano",
|
||||||
view_section: "Prikaz", reset_view: "Resetiraj prikaz", heatmap_label: "Toplinska karta",
|
view_section: "Prikaz", reset_view: "Resetiraj prikaz", heatmap_label: "Toplinska karta",
|
||||||
filters_section: "Filtri", without_coords_btn: "Bez koordinata",
|
filters_section: "Filtri", without_coords_btn: "Bez koordinata",
|
||||||
|
|
@ -666,7 +650,6 @@
|
||||||
|
|
||||||
sl: {
|
sl: {
|
||||||
search_placeholder: "Iskanje (ime, zvrst, mesto, država)…",
|
search_placeholder: "Iskanje (ime, zvrst, mesto, država)…",
|
||||||
truncated_hint: "Prikazanih je le prvih {n} rezultatov na tem območju — približajte za ogled vseh.",
|
|
||||||
displayed: "prikazano", total_bands: "skupaj", localities: "krajev", located: "lokalizirano",
|
displayed: "prikazano", total_bands: "skupaj", localities: "krajev", located: "lokalizirano",
|
||||||
view_section: "Pogled", reset_view: "Ponastavi pogled", heatmap_label: "Toplotna karta",
|
view_section: "Pogled", reset_view: "Ponastavi pogled", heatmap_label: "Toplotna karta",
|
||||||
filters_section: "Filtri", without_coords_btn: "Brez koordinat",
|
filters_section: "Filtri", without_coords_btn: "Brez koordinat",
|
||||||
|
|
@ -704,7 +687,6 @@
|
||||||
|
|
||||||
lt: {
|
lt: {
|
||||||
search_placeholder: "Ieškoti (pavadinimas, žanras, miestas, šalis)…",
|
search_placeholder: "Ieškoti (pavadinimas, žanras, miestas, šalis)…",
|
||||||
truncated_hint: "Rodomi tik pirmieji {n} šios srities rezultatai — priartinkite, kad matytumėte visus.",
|
|
||||||
displayed: "rodoma", total_bands: "iš viso", localities: "vietovių", located: "lokalizuota",
|
displayed: "rodoma", total_bands: "iš viso", localities: "vietovių", located: "lokalizuota",
|
||||||
view_section: "Vaizdas", reset_view: "Atstatyti vaizdą", heatmap_label: "Šilumos žemėlapis",
|
view_section: "Vaizdas", reset_view: "Atstatyti vaizdą", heatmap_label: "Šilumos žemėlapis",
|
||||||
filters_section: "Filtrai", without_coords_btn: "Be koordinačių",
|
filters_section: "Filtrai", without_coords_btn: "Be koordinačių",
|
||||||
|
|
@ -742,7 +724,6 @@
|
||||||
|
|
||||||
lv: {
|
lv: {
|
||||||
search_placeholder: "Meklēt (nosaukums, žanrs, pilsēta, valsts)…",
|
search_placeholder: "Meklēt (nosaukums, žanrs, pilsēta, valsts)…",
|
||||||
truncated_hint: "Tiek rādīti tikai pirmie {n} rezultāti šajā apgabalā — pietuviniet, lai redzētu visus.",
|
|
||||||
displayed: "rādīts", total_bands: "kopā", localities: "vietu", located: "lokalizēts",
|
displayed: "rādīts", total_bands: "kopā", localities: "vietu", located: "lokalizēts",
|
||||||
view_section: "Skats", reset_view: "Atiestatīt skatu", heatmap_label: "Siltuma karte",
|
view_section: "Skats", reset_view: "Atiestatīt skatu", heatmap_label: "Siltuma karte",
|
||||||
filters_section: "Filtri", without_coords_btn: "Bez koordinātām",
|
filters_section: "Filtri", without_coords_btn: "Bez koordinātām",
|
||||||
|
|
@ -780,7 +761,6 @@
|
||||||
|
|
||||||
et: {
|
et: {
|
||||||
search_placeholder: "Otsi (nimi, žanr, linn, riik)…",
|
search_placeholder: "Otsi (nimi, žanr, linn, riik)…",
|
||||||
truncated_hint: "Kuvatakse ainult selle piirkonna esimesed {n} tulemust — suumi sisse, et näha kõiki.",
|
|
||||||
displayed: "kuvatud", total_bands: "kokku", localities: "asulatest", located: "lokaliseeritud",
|
displayed: "kuvatud", total_bands: "kokku", localities: "asulatest", located: "lokaliseeritud",
|
||||||
view_section: "Vaade", reset_view: "Lähtesta vaade", heatmap_label: "Soojuskaart",
|
view_section: "Vaade", reset_view: "Lähtesta vaade", heatmap_label: "Soojuskaart",
|
||||||
filters_section: "Filtrid", without_coords_btn: "Ilma koordinaatideta",
|
filters_section: "Filtrid", without_coords_btn: "Ilma koordinaatideta",
|
||||||
|
|
@ -818,7 +798,6 @@
|
||||||
|
|
||||||
nb: {
|
nb: {
|
||||||
search_placeholder: "Søk (navn, sjanger, by, land)…",
|
search_placeholder: "Søk (navn, sjanger, by, land)…",
|
||||||
truncated_hint: "Bare de første {n} resultatene i dette området vises — zoom inn for å se alle.",
|
|
||||||
displayed: "vist", total_bands: "totalt", localities: "steder", located: "lokaliserte",
|
displayed: "vist", total_bands: "totalt", localities: "steder", located: "lokaliserte",
|
||||||
view_section: "Visning", reset_view: "Tilbakestill visning", heatmap_label: "Varmekart",
|
view_section: "Visning", reset_view: "Tilbakestill visning", heatmap_label: "Varmekart",
|
||||||
filters_section: "Filtre", without_coords_btn: "Uten koordinater",
|
filters_section: "Filtre", without_coords_btn: "Uten koordinater",
|
||||||
|
|
|
||||||
|
|
@ -102,38 +102,6 @@ var BMPure = (function () {
|
||||||
return y >= yearFilter.min && y <= yearFilter.max;
|
return y >= yearFilter.min && y <= yearFilter.max;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* Bbox du viewport, élargie puis BORNÉE aux limites géographiques.
|
|
||||||
*
|
|
||||||
* /api/clusters rejette en 400 toute coordonnée hors [-90,90] / [-180,180],
|
|
||||||
* ainsi que tout intervalle dégénéré. Sans bornage, l'élargissement de 20 %
|
|
||||||
* destiné à précharger les bords produisait au niveau monde une bbox du type
|
|
||||||
* -164,-88.4,144,110.4 : refusée, donc plus aucun point sur la carte et aucun
|
|
||||||
* message. Leaflet peut de surcroît renvoyer des longitudes au-delà de ±180
|
|
||||||
* quand la vue chevauche plusieurs copies du monde.
|
|
||||||
*
|
|
||||||
* @param {{south:number,west:number,north:number,east:number}} b
|
|
||||||
* @param {number} expandFactor
|
|
||||||
* @returns {{minLon:number,minLat:number,maxLon:number,maxLat:number}}
|
|
||||||
*/
|
|
||||||
function viewportBbox(b, expandFactor = 0.2) {
|
|
||||||
const clamp = (v, lo, hi) => Math.min(hi, Math.max(lo, v));
|
|
||||||
const latDiff = (b.north - b.south) * expandFactor;
|
|
||||||
const lonDiff = (b.east - b.west) * expandFactor;
|
|
||||||
|
|
||||||
const minLat = clamp(b.south - latDiff, -90, 90);
|
|
||||||
const maxLat = clamp(b.north + latDiff, -90, 90);
|
|
||||||
let minLon = clamp(b.west - lonDiff, -180, 180);
|
|
||||||
let maxLon = clamp(b.east + lonDiff, -180, 180);
|
|
||||||
|
|
||||||
// Vue à cheval sur plusieurs copies du monde : après bornage l'intervalle
|
|
||||||
// peut être vide. On demande alors la bande complète, que la vue couvre
|
|
||||||
// de toute façon.
|
|
||||||
if (minLon >= maxLon) { minLon = -180; maxLon = 180; }
|
|
||||||
|
|
||||||
return { minLon, minLat, maxLon, maxLat };
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Clé d'agrégation des groupes partageant exactement les mêmes coordonnées. */
|
/** Clé d'agrégation des groupes partageant exactement les mêmes coordonnées. */
|
||||||
function keyFromLatLon(lat, lon) {
|
function keyFromLatLon(lat, lon) {
|
||||||
return `${Number(lat).toFixed(6)},${Number(lon).toFixed(6)}`;
|
return `${Number(lat).toFixed(6)},${Number(lon).toFixed(6)}`;
|
||||||
|
|
@ -173,7 +141,6 @@ var BMPure = (function () {
|
||||||
matchesMacroGenre,
|
matchesMacroGenre,
|
||||||
matchesTheme,
|
matchesTheme,
|
||||||
matchesYear,
|
matchesYear,
|
||||||
viewportBbox,
|
|
||||||
keyFromLatLon,
|
keyFromLatLon,
|
||||||
sortBands,
|
sortBands,
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -1399,13 +1399,3 @@ a:hover {
|
||||||
min-height: 300px;
|
min-height: 300px;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/* Signale que la carte ne montre qu'une partie des résultats de la zone.
|
|
||||||
Sans ça, la troncature du serveur était totalement invisible. */
|
|
||||||
.trunc-note {
|
|
||||||
margin-left: 6px;
|
|
||||||
color: #e0a144;
|
|
||||||
cursor: help;
|
|
||||||
font-weight: 700;
|
|
||||||
}
|
|
||||||
.trunc-note[hidden] { display: none; }
|
|
||||||
|
|
|
||||||
|
|
@ -11,17 +11,10 @@ import axe from "axe-core";
|
||||||
* On charge le HTML livré tel quel dans jsdom et on passe axe-core dessus.
|
* On charge le HTML livré tel quel dans jsdom et on passe axe-core dessus.
|
||||||
* Pas de navigateur, pas de conteneur : ~1s pour les deux pages.
|
* Pas de navigateur, pas de conteneur : ~1s pour les deux pages.
|
||||||
*
|
*
|
||||||
* Limite RÉELLEMENT non couverte : les règles qui exigent un moteur de rendu
|
* Limite assumée : les règles qui nécessitent un vrai moteur de rendu
|
||||||
* (contraste de couleurs, taille des cibles tactiles) ne s'évaluent pas sous
|
* (contraste de couleurs, cibles tactiles) ne peuvent pas s'évaluer sous jsdom
|
||||||
* jsdom et sont désactivées ci-dessous.
|
* et sont désactivées. Elles ne sont donc PAS couvertes ici — voir le README
|
||||||
*
|
* de la CI pour l'audit Lighthouse manuel qui les complète.
|
||||||
* Ce commentaire renvoyait auparavant à « l'audit Lighthouse manuel du README
|
|
||||||
* de la CI ». Cet audit n'existe pas, et n'a jamais existé : README-CI.md
|
|
||||||
* documente apps/admin/test/e2e/a11y.spec.js, qui couvre bien le contraste
|
|
||||||
* dans un vrai navigateur — mais UNIQUEMENT pour le dashboard admin. Pour le
|
|
||||||
* site public, ces deux règles ne sont vérifiées nulle part. Le dire
|
|
||||||
* franchement vaut mieux que de renvoyer vers un filet imaginaire : la
|
|
||||||
* couverture n'a pas changé, seule sa description était fausse.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||||
|
|
|
||||||
|
|
@ -193,38 +193,3 @@ describe("clé de coordonnées", () => {
|
||||||
), { numRuns: 400 });
|
), { numRuns: 400 });
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("viewportBbox — propriétés", () => {
|
|
||||||
// Les contraintes exactes de /api/clusters. Toute vue Leaflet imaginable doit
|
|
||||||
// produire une bbox que l'API accepte : sinon la carte se vide sans message.
|
|
||||||
const acceptable = ({ minLon, minLat, maxLon, maxLat }) =>
|
|
||||||
minLat >= -90 && maxLat <= 90 &&
|
|
||||||
minLon >= -180 && maxLon <= 180 &&
|
|
||||||
minLon < maxLon && minLat < maxLat;
|
|
||||||
|
|
||||||
it("aucune vue ne produit une bbox rejetée par l'API", () => {
|
|
||||||
fc.assert(
|
|
||||||
fc.property(
|
|
||||||
// Latitudes dans la plage Web Mercator, longitudes volontairement
|
|
||||||
// au-delà de ±180 : Leaflet les renvoie ainsi quand la vue chevauche
|
|
||||||
// plusieurs copies du monde.
|
|
||||||
fc.double({ min: -85, max: 85, noNaN: true }),
|
|
||||||
fc.double({ min: -85, max: 85, noNaN: true }),
|
|
||||||
fc.double({ min: -540, max: 540, noNaN: true }),
|
|
||||||
fc.double({ min: -540, max: 540, noNaN: true }),
|
|
||||||
fc.double({ min: 0, max: 1, noNaN: true }),
|
|
||||||
(a, b, c, d, f) => {
|
|
||||||
const south = Math.min(a, b);
|
|
||||||
const north = Math.max(a, b);
|
|
||||||
const west = Math.min(c, d);
|
|
||||||
const east = Math.max(c, d);
|
|
||||||
// Une vue dégénérée (hauteur ou largeur nulle) n'existe pas sur une
|
|
||||||
// carte réellement affichée.
|
|
||||||
fc.pre(north > south && east > west);
|
|
||||||
return acceptable(P.viewportBbox({ south, west, north, east }, f));
|
|
||||||
}
|
|
||||||
),
|
|
||||||
{ numRuns: 500 }
|
|
||||||
);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
|
||||||
|
|
@ -337,46 +337,3 @@ describe("sortBands", () => {
|
||||||
expect(P.sortBands([], "az")).toEqual([]);
|
expect(P.sortBands([], "az")).toEqual([]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("viewportBbox", () => {
|
|
||||||
const vue = (south, west, north, east) => ({ south, west, north, east });
|
|
||||||
// Contraintes reprises telles quelles de /api/clusters (app.js) : toute
|
|
||||||
// violation y renvoie un 400, et la carte reste vide sans message.
|
|
||||||
const estAcceptableParLAPI = ({ minLon, minLat, maxLon, maxLat }) =>
|
|
||||||
minLat >= -90 && maxLat <= 90 && minLon >= -180 && maxLon <= 180 &&
|
|
||||||
minLon < maxLon && minLat < maxLat;
|
|
||||||
|
|
||||||
it("élargit la vue pour précharger les bords", () => {
|
|
||||||
const b = P.viewportBbox(vue(40, 0, 50, 10), 0.2);
|
|
||||||
expect(b).toEqual({ minLon: -2, minLat: 38, maxLon: 12, maxLat: 52 });
|
|
||||||
});
|
|
||||||
|
|
||||||
it("borne la latitude au niveau monde", () => {
|
|
||||||
// Sans bornage : maxLat = 110.4 -> 400 « Coordinates out of range ».
|
|
||||||
const b = P.viewportBbox(vue(-60, -120, 82, 100), 0.2);
|
|
||||||
expect(b.maxLat).toBe(90);
|
|
||||||
expect(b.minLat).toBeGreaterThanOrEqual(-90);
|
|
||||||
expect(estAcceptableParLAPI(b)).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("retombe sur la bande complète quand la vue chevauche plusieurs mondes", () => {
|
|
||||||
const b = P.viewportBbox(vue(40, 340, 60, 400), 0.2);
|
|
||||||
expect(b.minLon).toBe(-180);
|
|
||||||
expect(b.maxLon).toBe(180);
|
|
||||||
expect(estAcceptableParLAPI(b)).toBe(true);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("produit toujours une bbox que l'API accepte", () => {
|
|
||||||
const vues = [
|
|
||||||
vue(38, -10, 60.5, 30), // Europe, vue initiale
|
|
||||||
vue(25, -25, 70, 45), // dézoom intermédiaire
|
|
||||||
vue(-60, -120, 82, 100), // monde
|
|
||||||
vue(-85, -180, 85, 180), // dézoom maximal
|
|
||||||
vue(-85, -400, 85, 400), // plusieurs copies du monde
|
|
||||||
vue(0, 0, 0.001, 0.001), // zoom extrême
|
|
||||||
];
|
|
||||||
for (const v of vues) {
|
|
||||||
expect(estAcceptableParLAPI(P.viewportBbox(v, 0.2))).toBe(true);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
});
|
|
||||||
|
|
|
||||||
|
|
@ -12,8 +12,7 @@ services:
|
||||||
|
|
||||||
crawler:
|
crawler:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/crawler
|
||||||
dockerfile: apps/crawler/Dockerfile
|
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
FLARESOLVERR_URL: http://flaresolverr:8191
|
FLARESOLVERR_URL: http://flaresolverr:8191
|
||||||
|
|
@ -30,8 +29,7 @@ services:
|
||||||
|
|
||||||
geocoder-enqueue:
|
geocoder-enqueue:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -47,8 +45,7 @@ services:
|
||||||
|
|
||||||
geocoder-worker:
|
geocoder-worker:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -67,8 +64,7 @@ services:
|
||||||
|
|
||||||
groq-worker:
|
groq-worker:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -82,6 +78,25 @@ services:
|
||||||
- traefik.enable=false
|
- traefik.enable=false
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
pgadmin:
|
||||||
|
image: dpage/pgadmin4:8
|
||||||
|
environment:
|
||||||
|
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_EMAIL}
|
||||||
|
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_PASSWORD}
|
||||||
|
PGADMIN_CONFIG_SERVER_MODE: "True"
|
||||||
|
volumes:
|
||||||
|
- bm_dev_pgadmin_data:/var/lib/pgadmin
|
||||||
|
networks:
|
||||||
|
- coolify
|
||||||
|
labels:
|
||||||
|
- traefik.enable=true
|
||||||
|
- traefik.docker.network=coolify
|
||||||
|
- traefik.http.routers.dev-pgadmin.rule=Host(`dev-pgadmin.metalfrom.eu`)
|
||||||
|
- traefik.http.routers.dev-pgadmin.entrypoints=https
|
||||||
|
- traefik.http.routers.dev-pgadmin.tls=true
|
||||||
|
- traefik.http.routers.dev-pgadmin.tls.certresolver=letsencrypt
|
||||||
|
- traefik.http.services.dev-pgadmin.loadbalancer.server.port=80
|
||||||
|
|
||||||
api:
|
api:
|
||||||
build:
|
build:
|
||||||
context: apps/api
|
context: apps/api
|
||||||
|
|
@ -142,3 +157,6 @@ networks:
|
||||||
coolify:
|
coolify:
|
||||||
external: true
|
external: true
|
||||||
name: coolify
|
name: coolify
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
bm_dev_pgadmin_data:
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,7 @@
|
||||||
services:
|
services:
|
||||||
geocoder-enqueue:
|
geocoder-enqueue:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -18,8 +17,7 @@ services:
|
||||||
|
|
||||||
geocoder-worker:
|
geocoder-worker:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -38,8 +36,7 @@ services:
|
||||||
|
|
||||||
groq-worker:
|
groq-worker:
|
||||||
build:
|
build:
|
||||||
context: .
|
context: apps/geocoder
|
||||||
dockerfile: apps/geocoder/Dockerfile
|
|
||||||
working_dir: /app
|
working_dir: /app
|
||||||
environment:
|
environment:
|
||||||
DATABASE_URL: ${DATABASE_URL}
|
DATABASE_URL: ${DATABASE_URL}
|
||||||
|
|
@ -53,6 +50,25 @@ services:
|
||||||
- traefik.enable=false
|
- traefik.enable=false
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|
||||||
|
pgadmin:
|
||||||
|
image: dpage/pgadmin4:8
|
||||||
|
environment:
|
||||||
|
PGADMIN_DEFAULT_EMAIL: ${PGADMIN_EMAIL}
|
||||||
|
PGADMIN_DEFAULT_PASSWORD: ${PGADMIN_PASSWORD}
|
||||||
|
PGADMIN_CONFIG_SERVER_MODE: "True"
|
||||||
|
volumes:
|
||||||
|
- pgadmin_data:/var/lib/pgadmin
|
||||||
|
networks:
|
||||||
|
- coolify
|
||||||
|
labels:
|
||||||
|
- traefik.enable=true
|
||||||
|
- traefik.docker.network=coolify
|
||||||
|
- traefik.http.routers.bm-pgadmin.rule=Host(`pgadmin.bm.nicolasfryder.ovh`)
|
||||||
|
- traefik.http.routers.bm-pgadmin.entrypoints=https
|
||||||
|
- traefik.http.routers.bm-pgadmin.tls=true
|
||||||
|
- traefik.http.routers.bm-pgadmin.tls.certresolver=letsencrypt
|
||||||
|
- traefik.http.services.bm-pgadmin.loadbalancer.server.port=80
|
||||||
|
|
||||||
api:
|
api:
|
||||||
build:
|
build:
|
||||||
context: apps/api
|
context: apps/api
|
||||||
|
|
@ -113,3 +129,6 @@ networks:
|
||||||
coolify:
|
coolify:
|
||||||
external: true
|
external: true
|
||||||
name: coolify
|
name: coolify
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
pgadmin_data:
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,12 @@ GEOAPIFY_API_KEY=
|
||||||
# Désambiguïsation LLM des lieux que Geoapify ne résout pas.
|
# Désambiguïsation LLM des lieux que Geoapify ne résout pas.
|
||||||
GROQ_API_KEY=
|
GROQ_API_KEY=
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# pgAdmin
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
PGADMIN_EMAIL=
|
||||||
|
PGADMIN_PASSWORD=
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Divers
|
# Divers
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|
|
||||||
357
package-lock.json
generated
357
package-lock.json
generated
|
|
@ -31,7 +31,6 @@
|
||||||
"name": "bm-api",
|
"name": "bm-api",
|
||||||
"version": "0.0.1",
|
"version": "0.0.1",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@fastify/compress": "^7.0.3",
|
|
||||||
"@fastify/cookie": "^9.4.0",
|
"@fastify/cookie": "^9.4.0",
|
||||||
"@fastify/helmet": "^11.1.1",
|
"@fastify/helmet": "^11.1.1",
|
||||||
"@fastify/rate-limit": "^9.0.0",
|
"@fastify/rate-limit": "^9.0.0",
|
||||||
|
|
@ -41,31 +40,6 @@
|
||||||
"pg": "^8.12.0"
|
"pg": "^8.12.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"apps/api/node_modules/@fastify/accept-negotiator": {
|
|
||||||
"version": "1.1.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/@fastify/accept-negotiator/-/accept-negotiator-1.1.0.tgz",
|
|
||||||
"integrity": "sha512-OIHZrb2ImZ7XG85HXOONLcJWGosv7sIvM2ifAPQVhg9Lv7qdmMBNVaai4QTdyuaqbKM5eO6sLSQOYI7wEQeCJQ==",
|
|
||||||
"license": "MIT",
|
|
||||||
"engines": {
|
|
||||||
"node": ">=14"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"apps/api/node_modules/@fastify/compress": {
|
|
||||||
"version": "7.0.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/@fastify/compress/-/compress-7.0.3.tgz",
|
|
||||||
"integrity": "sha512-xa9fo5/DgK1s0bkS6xrYgNn8HmofO5tJvbCDk8QuXshSgLd2cFZANv1ox/Qv7zswS7JroHwTlCVv/XGTVO98tg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"@fastify/accept-negotiator": "^1.1.0",
|
|
||||||
"fastify-plugin": "^4.5.0",
|
|
||||||
"mime-db": "^1.52.0",
|
|
||||||
"minipass": "^7.0.2",
|
|
||||||
"peek-stream": "^1.1.3",
|
|
||||||
"pump": "^3.0.0",
|
|
||||||
"pumpify": "^2.0.1",
|
|
||||||
"readable-stream": "^4.5.2"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/@ampproject/remapping": {
|
"node_modules/@ampproject/remapping": {
|
||||||
"version": "2.3.0",
|
"version": "2.3.0",
|
||||||
"resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz",
|
"resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.3.0.tgz",
|
||||||
|
|
@ -2663,18 +2637,6 @@
|
||||||
"url": "https://opencollective.com/vitest"
|
"url": "https://opencollective.com/vitest"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/abort-controller": {
|
|
||||||
"version": "3.0.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz",
|
|
||||||
"integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"event-target-shim": "^5.0.0"
|
|
||||||
},
|
|
||||||
"engines": {
|
|
||||||
"node": ">=6.5"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/abstract-logging": {
|
"node_modules/abstract-logging": {
|
||||||
"version": "2.0.1",
|
"version": "2.0.1",
|
||||||
"resolved": "https://registry.npmjs.org/abstract-logging/-/abstract-logging-2.0.1.tgz",
|
"resolved": "https://registry.npmjs.org/abstract-logging/-/abstract-logging-2.0.1.tgz",
|
||||||
|
|
@ -2862,26 +2824,6 @@
|
||||||
"dev": true,
|
"dev": true,
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
"node_modules/base64-js": {
|
|
||||||
"version": "1.5.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
|
|
||||||
"integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==",
|
|
||||||
"funding": [
|
|
||||||
{
|
|
||||||
"type": "github",
|
|
||||||
"url": "https://github.com/sponsors/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "patreon",
|
|
||||||
"url": "https://www.patreon.com/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "consulting",
|
|
||||||
"url": "https://feross.org/support"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/baseline-browser-mapping": {
|
"node_modules/baseline-browser-mapping": {
|
||||||
"version": "2.11.15",
|
"version": "2.11.15",
|
||||||
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.15.tgz",
|
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.15.tgz",
|
||||||
|
|
@ -2959,42 +2901,12 @@
|
||||||
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
|
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/buffer": {
|
|
||||||
"version": "6.0.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz",
|
|
||||||
"integrity": "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA==",
|
|
||||||
"funding": [
|
|
||||||
{
|
|
||||||
"type": "github",
|
|
||||||
"url": "https://github.com/sponsors/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "patreon",
|
|
||||||
"url": "https://www.patreon.com/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "consulting",
|
|
||||||
"url": "https://feross.org/support"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"base64-js": "^1.3.1",
|
|
||||||
"ieee754": "^1.2.1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/buffer-equal-constant-time": {
|
"node_modules/buffer-equal-constant-time": {
|
||||||
"version": "1.0.1",
|
"version": "1.0.1",
|
||||||
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
|
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
|
||||||
"integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==",
|
"integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==",
|
||||||
"license": "BSD-3-Clause"
|
"license": "BSD-3-Clause"
|
||||||
},
|
},
|
||||||
"node_modules/buffer-from": {
|
|
||||||
"version": "1.1.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
|
|
||||||
"integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/cac": {
|
"node_modules/cac": {
|
||||||
"version": "6.7.14",
|
"version": "6.7.14",
|
||||||
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
|
"resolved": "https://registry.npmjs.org/cac/-/cac-6.7.14.tgz",
|
||||||
|
|
@ -3199,12 +3111,6 @@
|
||||||
"node": ">=6.6.0"
|
"node": ">=6.6.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/core-util-is": {
|
|
||||||
"version": "1.0.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.3.tgz",
|
|
||||||
"integrity": "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/cross-spawn": {
|
"node_modules/cross-spawn": {
|
||||||
"version": "7.0.6",
|
"version": "7.0.6",
|
||||||
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
|
"resolved": "https://registry.npmjs.org/cross-spawn/-/cross-spawn-7.0.6.tgz",
|
||||||
|
|
@ -3340,48 +3246,6 @@
|
||||||
"node": ">= 0.4"
|
"node": ">= 0.4"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/duplexify": {
|
|
||||||
"version": "3.7.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/duplexify/-/duplexify-3.7.1.tgz",
|
|
||||||
"integrity": "sha512-07z8uv2wMyS51kKhD1KsdXJg5WQ6t93RneqRxUHnskXVtlYYkLqM0gqStQZ3pj073g687jPCHrqNfCzawLYh5g==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"end-of-stream": "^1.0.0",
|
|
||||||
"inherits": "^2.0.1",
|
|
||||||
"readable-stream": "^2.0.0",
|
|
||||||
"stream-shift": "^1.0.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/duplexify/node_modules/readable-stream": {
|
|
||||||
"version": "2.3.8",
|
|
||||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz",
|
|
||||||
"integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"core-util-is": "~1.0.0",
|
|
||||||
"inherits": "~2.0.3",
|
|
||||||
"isarray": "~1.0.0",
|
|
||||||
"process-nextick-args": "~2.0.0",
|
|
||||||
"safe-buffer": "~5.1.1",
|
|
||||||
"string_decoder": "~1.1.1",
|
|
||||||
"util-deprecate": "~1.0.1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/duplexify/node_modules/safe-buffer": {
|
|
||||||
"version": "5.1.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
|
|
||||||
"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/duplexify/node_modules/string_decoder": {
|
|
||||||
"version": "1.1.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz",
|
|
||||||
"integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"safe-buffer": "~5.1.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/eastasianwidth": {
|
"node_modules/eastasianwidth": {
|
||||||
"version": "0.2.0",
|
"version": "0.2.0",
|
||||||
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz",
|
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz",
|
||||||
|
|
@ -3412,15 +3276,6 @@
|
||||||
"dev": true,
|
"dev": true,
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
"node_modules/end-of-stream": {
|
|
||||||
"version": "1.4.5",
|
|
||||||
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz",
|
|
||||||
"integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"once": "^1.4.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/entities": {
|
"node_modules/entities": {
|
||||||
"version": "6.0.1",
|
"version": "6.0.1",
|
||||||
"resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
|
"resolved": "https://registry.npmjs.org/entities/-/entities-6.0.1.tgz",
|
||||||
|
|
@ -3781,24 +3636,6 @@
|
||||||
"node": ">=0.10.0"
|
"node": ">=0.10.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/event-target-shim": {
|
|
||||||
"version": "5.0.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz",
|
|
||||||
"integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==",
|
|
||||||
"license": "MIT",
|
|
||||||
"engines": {
|
|
||||||
"node": ">=6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/events": {
|
|
||||||
"version": "3.3.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz",
|
|
||||||
"integrity": "sha512-mQw+2fkQbALzQ7V0MY0IqdnXNOeTtP4r0lN9z7AAawCXgqea7bDii20AYrIBrFd/Hx0M2Ocz6S111CaFkUcb0Q==",
|
|
||||||
"license": "MIT",
|
|
||||||
"engines": {
|
|
||||||
"node": ">=0.8.x"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/execa": {
|
"node_modules/execa": {
|
||||||
"version": "9.4.1",
|
"version": "9.4.1",
|
||||||
"resolved": "https://registry.npmjs.org/execa/-/execa-9.4.1.tgz",
|
"resolved": "https://registry.npmjs.org/execa/-/execa-9.4.1.tgz",
|
||||||
|
|
@ -4439,26 +4276,6 @@
|
||||||
"node": ">=0.10.0"
|
"node": ">=0.10.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/ieee754": {
|
|
||||||
"version": "1.2.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
|
|
||||||
"integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==",
|
|
||||||
"funding": [
|
|
||||||
{
|
|
||||||
"type": "github",
|
|
||||||
"url": "https://github.com/sponsors/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "patreon",
|
|
||||||
"url": "https://www.patreon.com/feross"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"type": "consulting",
|
|
||||||
"url": "https://feross.org/support"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"license": "BSD-3-Clause"
|
|
||||||
},
|
|
||||||
"node_modules/ignore": {
|
"node_modules/ignore": {
|
||||||
"version": "5.3.2",
|
"version": "5.3.2",
|
||||||
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz",
|
"resolved": "https://registry.npmjs.org/ignore/-/ignore-5.3.2.tgz",
|
||||||
|
|
@ -4500,6 +4317,7 @@
|
||||||
"version": "2.0.4",
|
"version": "2.0.4",
|
||||||
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
||||||
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
|
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
|
||||||
|
"dev": true,
|
||||||
"license": "ISC"
|
"license": "ISC"
|
||||||
},
|
},
|
||||||
"node_modules/ipaddr.js": {
|
"node_modules/ipaddr.js": {
|
||||||
|
|
@ -4590,12 +4408,6 @@
|
||||||
"url": "https://github.com/sponsors/sindresorhus"
|
"url": "https://github.com/sponsors/sindresorhus"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/isarray": {
|
|
||||||
"version": "1.0.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
|
|
||||||
"integrity": "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/isexe": {
|
"node_modules/isexe": {
|
||||||
"version": "2.0.0",
|
"version": "2.0.0",
|
||||||
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
|
"resolved": "https://registry.npmjs.org/isexe/-/isexe-2.0.0.tgz",
|
||||||
|
|
@ -5025,6 +4837,7 @@
|
||||||
"version": "1.52.0",
|
"version": "1.52.0",
|
||||||
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
|
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz",
|
||||||
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
|
"integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==",
|
||||||
|
"dev": true,
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">= 0.6"
|
"node": ">= 0.6"
|
||||||
|
|
@ -5070,6 +4883,7 @@
|
||||||
"version": "7.1.3",
|
"version": "7.1.3",
|
||||||
"resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz",
|
"resolved": "https://registry.npmjs.org/minipass/-/minipass-7.1.3.tgz",
|
||||||
"integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==",
|
"integrity": "sha512-tEBHqDnIoM/1rXME1zgka9g6Q2lcoCkxHLuc7ODJ5BxbP5d4c2Z5cGgtXAku59200Cx7diuHTOYfSBD8n6mm8A==",
|
||||||
|
"dev": true,
|
||||||
"license": "BlueOak-1.0.0",
|
"license": "BlueOak-1.0.0",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=16 || 14 >=14.17"
|
"node": ">=16 || 14 >=14.17"
|
||||||
|
|
@ -5224,15 +5038,6 @@
|
||||||
"node": ">=14.0.0"
|
"node": ">=14.0.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/once": {
|
|
||||||
"version": "1.4.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
|
|
||||||
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
|
|
||||||
"license": "ISC",
|
|
||||||
"dependencies": {
|
|
||||||
"wrappy": "1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/optionator": {
|
"node_modules/optionator": {
|
||||||
"version": "0.9.4",
|
"version": "0.9.4",
|
||||||
"resolved": "https://registry.npmjs.org/optionator/-/optionator-0.9.4.tgz",
|
"resolved": "https://registry.npmjs.org/optionator/-/optionator-0.9.4.tgz",
|
||||||
|
|
@ -5400,17 +5205,6 @@
|
||||||
"node": ">= 14.16"
|
"node": ">= 14.16"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/peek-stream": {
|
|
||||||
"version": "1.1.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/peek-stream/-/peek-stream-1.1.3.tgz",
|
|
||||||
"integrity": "sha512-FhJ+YbOSBb9/rIl2ZeE/QHEsWn7PqNYt8ARAY3kIgNGOk13g9FGyIY6JIl/xB/3TFRVoTv5as0l11weORrTekA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"buffer-from": "^1.0.0",
|
|
||||||
"duplexify": "^3.5.0",
|
|
||||||
"through2": "^2.0.3"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/pg": {
|
"node_modules/pg": {
|
||||||
"version": "8.23.0",
|
"version": "8.23.0",
|
||||||
"resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
|
"resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
|
||||||
|
|
@ -5701,21 +5495,6 @@
|
||||||
"url": "https://github.com/sponsors/sindresorhus"
|
"url": "https://github.com/sponsors/sindresorhus"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/process": {
|
|
||||||
"version": "0.11.10",
|
|
||||||
"resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
|
|
||||||
"integrity": "sha512-cdGef/drWFoydD1JsMzuFf8100nZl+GT+yacc2bEced5f9Rjk4z+WtFUTBu9PhOi9j/jfmBPu0mMEY4wIdAF8A==",
|
|
||||||
"license": "MIT",
|
|
||||||
"engines": {
|
|
||||||
"node": ">= 0.6.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/process-nextick-args": {
|
|
||||||
"version": "2.0.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.1.tgz",
|
|
||||||
"integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/process-warning": {
|
"node_modules/process-warning": {
|
||||||
"version": "3.0.0",
|
"version": "3.0.0",
|
||||||
"resolved": "https://registry.npmjs.org/process-warning/-/process-warning-3.0.0.tgz",
|
"resolved": "https://registry.npmjs.org/process-warning/-/process-warning-3.0.0.tgz",
|
||||||
|
|
@ -5745,53 +5524,6 @@
|
||||||
"node": ">= 0.10"
|
"node": ">= 0.10"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/pump": {
|
|
||||||
"version": "3.0.4",
|
|
||||||
"resolved": "https://registry.npmjs.org/pump/-/pump-3.0.4.tgz",
|
|
||||||
"integrity": "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"end-of-stream": "^1.1.0",
|
|
||||||
"once": "^1.3.1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/pumpify": {
|
|
||||||
"version": "2.0.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/pumpify/-/pumpify-2.0.1.tgz",
|
|
||||||
"integrity": "sha512-m7KOje7jZxrmutanlkS1daj1dS6z6BgslzOXmcSEpIlCxM3VJH7lG5QLeck/6hgF6F4crFf01UtQmNsJfweTAw==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"duplexify": "^4.1.1",
|
|
||||||
"inherits": "^2.0.3",
|
|
||||||
"pump": "^3.0.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/pumpify/node_modules/duplexify": {
|
|
||||||
"version": "4.1.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.3.tgz",
|
|
||||||
"integrity": "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"end-of-stream": "^1.4.1",
|
|
||||||
"inherits": "^2.0.3",
|
|
||||||
"readable-stream": "^3.1.1",
|
|
||||||
"stream-shift": "^1.0.2"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/pumpify/node_modules/readable-stream": {
|
|
||||||
"version": "3.6.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz",
|
|
||||||
"integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"inherits": "^2.0.3",
|
|
||||||
"string_decoder": "^1.1.1",
|
|
||||||
"util-deprecate": "^1.0.1"
|
|
||||||
},
|
|
||||||
"engines": {
|
|
||||||
"node": ">= 6"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/punycode": {
|
"node_modules/punycode": {
|
||||||
"version": "2.3.1",
|
"version": "2.3.1",
|
||||||
"resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz",
|
"resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz",
|
||||||
|
|
@ -5842,22 +5574,6 @@
|
||||||
"integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==",
|
"integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==",
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
"node_modules/readable-stream": {
|
|
||||||
"version": "4.7.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz",
|
|
||||||
"integrity": "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"abort-controller": "^3.0.0",
|
|
||||||
"buffer": "^6.0.3",
|
|
||||||
"events": "^3.3.0",
|
|
||||||
"process": "^0.11.10",
|
|
||||||
"string_decoder": "^1.3.0"
|
|
||||||
},
|
|
||||||
"engines": {
|
|
||||||
"node": "^12.22.0 || ^14.17.0 || >=16.0.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/real-require": {
|
"node_modules/real-require": {
|
||||||
"version": "0.2.0",
|
"version": "0.2.0",
|
||||||
"resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz",
|
"resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz",
|
||||||
|
|
@ -6227,21 +5943,6 @@
|
||||||
"dev": true,
|
"dev": true,
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
"node_modules/stream-shift": {
|
|
||||||
"version": "1.0.3",
|
|
||||||
"resolved": "https://registry.npmjs.org/stream-shift/-/stream-shift-1.0.3.tgz",
|
|
||||||
"integrity": "sha512-76ORR0DO1o1hlKwTbi/DM3EXWGf3ZJYO8cXX5RJwnul2DEg2oyoZyjLNoQM8WsvZiFKCRfC1O0J7iCvie3RZmQ==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/string_decoder": {
|
|
||||||
"version": "1.3.0",
|
|
||||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
|
|
||||||
"integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"safe-buffer": "~5.2.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/string-width": {
|
"node_modules/string-width": {
|
||||||
"version": "5.1.2",
|
"version": "5.1.2",
|
||||||
"resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz",
|
"resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz",
|
||||||
|
|
@ -6455,46 +6156,6 @@
|
||||||
"real-require": "^0.2.0"
|
"real-require": "^0.2.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/through2": {
|
|
||||||
"version": "2.0.5",
|
|
||||||
"resolved": "https://registry.npmjs.org/through2/-/through2-2.0.5.tgz",
|
|
||||||
"integrity": "sha512-/mrRod8xqpA+IHSLyGCQ2s8SPHiCDEeQJSep1jqLYeEUClOFG2Qsh+4FU6G9VeqpZnGW/Su8LQGc4YKni5rYSQ==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"readable-stream": "~2.3.6",
|
|
||||||
"xtend": "~4.0.1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/through2/node_modules/readable-stream": {
|
|
||||||
"version": "2.3.8",
|
|
||||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.8.tgz",
|
|
||||||
"integrity": "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"core-util-is": "~1.0.0",
|
|
||||||
"inherits": "~2.0.3",
|
|
||||||
"isarray": "~1.0.0",
|
|
||||||
"process-nextick-args": "~2.0.0",
|
|
||||||
"safe-buffer": "~5.1.1",
|
|
||||||
"string_decoder": "~1.1.1",
|
|
||||||
"util-deprecate": "~1.0.1"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/through2/node_modules/safe-buffer": {
|
|
||||||
"version": "5.1.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
|
|
||||||
"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/through2/node_modules/string_decoder": {
|
|
||||||
"version": "1.1.1",
|
|
||||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.1.1.tgz",
|
|
||||||
"integrity": "sha512-n/ShnvDi6FHbbVfviro+WojiFzv+s8MPMHBczVePfUpDJLwoLT0ht1l4YwBCbi8pJAveEEdnkHyPyTP/mzRfwg==",
|
|
||||||
"license": "MIT",
|
|
||||||
"dependencies": {
|
|
||||||
"safe-buffer": "~5.1.0"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"node_modules/tinybench": {
|
"node_modules/tinybench": {
|
||||||
"version": "2.9.0",
|
"version": "2.9.0",
|
||||||
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
|
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
|
||||||
|
|
@ -6769,12 +6430,6 @@
|
||||||
"punycode": "^2.1.0"
|
"punycode": "^2.1.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/util-deprecate": {
|
|
||||||
"version": "1.0.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
|
|
||||||
"integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==",
|
|
||||||
"license": "MIT"
|
|
||||||
},
|
|
||||||
"node_modules/vite": {
|
"node_modules/vite": {
|
||||||
"version": "5.4.21",
|
"version": "5.4.21",
|
||||||
"resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz",
|
"resolved": "https://registry.npmjs.org/vite/-/vite-5.4.21.tgz",
|
||||||
|
|
@ -7126,12 +6781,6 @@
|
||||||
"node": ">=8"
|
"node": ">=8"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"node_modules/wrappy": {
|
|
||||||
"version": "1.0.2",
|
|
||||||
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
|
|
||||||
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
|
|
||||||
"license": "ISC"
|
|
||||||
},
|
|
||||||
"node_modules/ws": {
|
"node_modules/ws": {
|
||||||
"version": "8.21.3",
|
"version": "8.21.3",
|
||||||
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
|
"resolved": "https://registry.npmjs.org/ws/-/ws-8.21.3.tgz",
|
||||||
|
|
|
||||||
|
|
@ -1,16 +1,12 @@
|
||||||
# Outillage qualité Python (hors runtime des apps).
|
# Outillage qualité Python (hors runtime des apps).
|
||||||
# pip install -r requirements-dev.txt
|
# pip install -r requirements-dev.txt
|
||||||
ruff>=0.8.4
|
ruff==0.8.4
|
||||||
pytest>=8.3.4
|
pytest==8.3.4
|
||||||
pip-audit>=2.7.3
|
pip-audit==2.7.3
|
||||||
|
|
||||||
# Les tests importent src.db, qui importe psycopg2 au chargement du module.
|
# Les tests importent src.db, qui importe psycopg2 au chargement du module.
|
||||||
# Aucune connexion n'est ouverte : get_conn est remplacé dans les tests.
|
# Aucune connexion n'est ouverte : get_conn est remplacé dans les tests.
|
||||||
#
|
psycopg2-binary==2.9.9
|
||||||
# Épinglage souple volontaire : 2.9.9 n'a pas de roue pour Python >= 3.13, et
|
beautifulsoup4==4.12.3
|
||||||
# la compilation depuis les sources échoue sans en-têtes PostgreSQL. La suite
|
lxml==5.3.0
|
||||||
# entière s'arrêtait alors dès la collecte, sur un poste par ailleurs sain.
|
hypothesis==6.122.3
|
||||||
psycopg2-binary>=2.9.10
|
|
||||||
beautifulsoup4>=4.12.3
|
|
||||||
lxml>=5.3.0
|
|
||||||
hypothesis>=6.122.3
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue