Error log, retry notice, a waiting counter that keeps counting
- generation_errors: every failed attempt (model, attempt, outcome, duration); admin Hibanapló page and GET /v1/admin/errors; kept 90 days - The waiting page counts from when the text was asked for (requested_at), not from the page load, so a reload no longer restarts it - It says when an attempt failed and is retried, and when a failed text's unit was given back; admins see the error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012vLHzh5qUhFtmJhzkcTbXL
This commit is contained in:
@@ -0,0 +1,26 @@
|
|||||||
|
-- Every failed attempt of an analysis, an answer or a prompt test run, for the
|
||||||
|
-- admin error log. No foreign keys: the log outlives deleted items. Kept for
|
||||||
|
-- 90 days (worker cleanup).
|
||||||
|
CREATE TABLE generation_errors (
|
||||||
|
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||||
|
created_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
source text NOT NULL CHECK (source IN ('analysis', 'question', 'preview')),
|
||||||
|
item_id uuid NOT NULL,
|
||||||
|
kind text NOT NULL,
|
||||||
|
model text NOT NULL,
|
||||||
|
attempt integer NOT NULL,
|
||||||
|
-- What happened next: tried again, the default model took over, or gave up.
|
||||||
|
outcome text NOT NULL CHECK (outcome IN ('retry', 'fallback', 'failed')),
|
||||||
|
error text NOT NULL,
|
||||||
|
seconds integer
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX generation_errors_created_idx ON generation_errors (created_at DESC);
|
||||||
|
CREATE INDEX generation_errors_item_idx ON generation_errors (item_id);
|
||||||
|
|
||||||
|
-- When the current text was asked for (a regeneration or an admin restart
|
||||||
|
-- sets it again): the waiting page counts from here.
|
||||||
|
ALTER TABLE analyses ADD COLUMN requested_at timestamptz NOT NULL DEFAULT now();
|
||||||
|
UPDATE analyses SET requested_at = created_at;
|
||||||
|
ALTER TABLE questions ADD COLUMN requested_at timestamptz NOT NULL DEFAULT now();
|
||||||
|
UPDATE questions SET requested_at = created_at;
|
||||||
@@ -127,6 +127,7 @@ Hibák formátuma: `{"error": {"code", "message"}}`. A születési adatok hibái
|
|||||||
- **Fizetős rész:** a részletes elemzés darabra megy. A képletelemzés egy `natal`, a szinasztria-elemzés egy `synastry` egységet von le (`credit_ledger.unit`). Az egységek csomagvásárlásból (Stripe, később) vagy admin jóváírásból jönnek. Havi ingyenes keret nincs.
|
- **Fizetős rész:** a részletes elemzés darabra megy. A képletelemzés egy `natal`, a szinasztria-elemzés egy `synastry` egységet von le (`credit_ledger.unit`). Az egységek csomagvásárlásból (Stripe, később) vagy admin jóváírásból jönnek. Havi ingyenes keret nincs.
|
||||||
- **Modell:** három beállítás az admin felületen, az OpenRouter listájából (`model_settings` tábla): az elemzések és az éves előrejelzés (képlet-, kapcsolati elemzés, előrejelzés) közös modellje, a kérdés-válasz modellje és a rövid összefoglalóé. Ha nincs beállítva, az `ANALYSIS_MODEL` / `SUMMARY_MODEL` (alapból mindkettő Opus 5.5, a 2026-11-es összevetés legjobbja, lásd `services/worker/README.md`). Csak kötött JSON-választ adó modell választható. A kliens nem választ modellt, és a modell neve felhasználónak sehol nem jelenik meg.
|
- **Modell:** három beállítás az admin felületen, az OpenRouter listájából (`model_settings` tábla): az elemzések és az éves előrejelzés (képlet-, kapcsolati elemzés, előrejelzés) közös modellje, a kérdés-válasz modellje és a rövid összefoglalóé. Ha nincs beállítva, az `ANALYSIS_MODEL` / `SUMMARY_MODEL` (alapból mindkettő Opus 5.5, a 2026-11-es összevetés legjobbja, lásd `services/worker/README.md`). Csak kötött JSON-választ adó modell választható. A kliens nem választ modellt, és a modell neve felhasználónak sehol nem jelenik meg.
|
||||||
- **Tartalék:** ha a választott modell kétszer hibára fut (végleges hibánál egyszer), a worker az alapmodellel próbálja tovább (`fallback_model`, `on_fallback`); csak ha az is elbukik, jár vissza az egység. A Modellek oldal mutatja, hány szövegnél kellett az elmúlt 30 napban.
|
- **Tartalék:** ha a választott modell kétszer hibára fut (végleges hibánál egyszer), a worker az alapmodellel próbálja tovább (`fallback_model`, `on_fallback`); csak ha az is elbukik, jár vissza az egység. A Modellek oldal mutatja, hány szövegnél kellett az elmúlt 30 napban.
|
||||||
|
- **Hibanapló:** a worker minden sikertelen próbálkozást a `generation_errors` táblába ír (mi, melyik modell, hányadik kísérlet, mi lett: újrapróbálás, tartalék vagy végleges hiba, mennyi ideig tartott); az admin Hibanapló oldala mutatja (`GET /v1/admin/errors?source=&outcome=`), 90 napig őrizzük. A felhasználó a várakozó oldalon látja, ha az előző próbálkozás nem sikerült és újrapróbáljuk; a számláló a kérés idejétől (`requested_at`) mér.
|
||||||
- **Futó generálások:** a Modellek oldalon látszik minden készülő és sorban álló elemzés és válasz, a modelljével és azzal, amivel most indulna; újraindításkor a jelenlegi beállítással indul újra (`GET /v1/admin/jobs`, `POST /v1/admin/jobs/restart`). Minden kísérlet `started_at` értéke a „jegye”: az újraindítás törli, így a workerben még futó régi kísérlet nem írhatja felül az újat, és a worker 10 másodpercen belül eldobja.
|
- **Futó generálások:** a Modellek oldalon látszik minden készülő és sorban álló elemzés és válasz, a modelljével és azzal, amivel most indulna; újraindításkor a jelenlegi beállítással indul újra (`GET /v1/admin/jobs`, `POST /v1/admin/jobs/restart`). Minden kísérlet `started_at` értéke a „jegye”: az újraindítás törli, így a workerben még futó régi kísérlet nem írhatja felül az újat, és a worker 10 másodpercen belül eldobja.
|
||||||
- **Újragenerálás:** admin minden elemzést egyenként újraírathat (`POST /v1/admin/analyses/{id}/regenerate`): ugyanaz a sor (az azonosító és a megosztások maradnak), a jelenlegi prompttal és modellel, egységlevonás nélkül.
|
- **Újragenerálás:** admin minden elemzést egyenként újraírathat (`POST /v1/admin/analyses/{id}/regenerate`): ugyanaz a sor (az azonosító és a megosztások maradnak), a jelenlegi prompttal és modellel, egységlevonás nélkül.
|
||||||
- **Újraszámolás:** számítási változás után a `backend recalculate` parancs minden mentett képletet, összevetést és előrejelzést újraszámol, majd minden elemzést újra sorba állít (egységlevonás nélkül).
|
- **Újraszámolás:** számítási változás után a `backend recalculate` parancs minden mentett képletet, összevetést és előrejelzést újraszámol, majd minden elemzést újra sorba állít (egységlevonás nélkül).
|
||||||
|
|||||||
@@ -49,11 +49,16 @@ pub struct AnalysisRow {
|
|||||||
pub credits_charged: i32,
|
pub credits_charged: i32,
|
||||||
pub created_at: OffsetDateTime,
|
pub created_at: OffsetDateTime,
|
||||||
pub completed_at: Option<OffsetDateTime>,
|
pub completed_at: Option<OffsetDateTime>,
|
||||||
|
/// Attempts so far; above 1 the page says it is being tried again.
|
||||||
|
pub attempts: i32,
|
||||||
|
pub next_attempt_at: OffsetDateTime,
|
||||||
|
/// When the current text was asked for (set again by a regeneration).
|
||||||
|
pub requested_at: OffsetDateTime,
|
||||||
}
|
}
|
||||||
|
|
||||||
const ANALYSIS_COLUMNS: &str = "id, kind, chart_id, synastry_id, forecast_id, language, prompt_version, status, \
|
const ANALYSIS_COLUMNS: &str = "id, kind, chart_id, synastry_id, forecast_id, language, prompt_version, status, \
|
||||||
content, error, \
|
content, error, \
|
||||||
credits_charged, created_at, completed_at";
|
credits_charged, created_at, completed_at, attempts, next_attempt_at, requested_at";
|
||||||
|
|
||||||
impl AnalysisRow {
|
impl AnalysisRow {
|
||||||
pub fn view(&self) -> Value {
|
pub fn view(&self) -> Value {
|
||||||
@@ -71,6 +76,9 @@ impl AnalysisRow {
|
|||||||
"credits_charged": self.credits_charged,
|
"credits_charged": self.credits_charged,
|
||||||
"created_at": rfc3339(self.created_at),
|
"created_at": rfc3339(self.created_at),
|
||||||
"completed_at": rfc3339_opt(self.completed_at),
|
"completed_at": rfc3339_opt(self.completed_at),
|
||||||
|
"attempts": self.attempts,
|
||||||
|
"next_attempt_at": rfc3339(self.next_attempt_at),
|
||||||
|
"requested_at": rfc3339(self.requested_at),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -391,7 +399,7 @@ pub async fn regenerate(
|
|||||||
"UPDATE analyses SET status = 'pending', content = NULL, error = NULL, model_used = NULL, \
|
"UPDATE analyses SET status = 'pending', content = NULL, error = NULL, model_used = NULL, \
|
||||||
input_tokens = NULL, output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), \
|
input_tokens = NULL, output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), \
|
||||||
started_at = NULL, completed_at = NULL, model = $2, prompt_version = $3, cache_key = $4, \
|
started_at = NULL, completed_at = NULL, model = $2, prompt_version = $3, cache_key = $4, \
|
||||||
fallback_model = $5, on_fallback = false \
|
fallback_model = $5, on_fallback = false, requested_at = now() \
|
||||||
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {ANALYSIS_COLUMNS}"
|
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {ANALYSIS_COLUMNS}"
|
||||||
))
|
))
|
||||||
.bind(id)
|
.bind(id)
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
//! Admin: the error log of the analyses, answers and prompt test runs (every
|
||||||
|
//! failed attempt, written by the worker), newest first.
|
||||||
|
|
||||||
|
use axum::Json;
|
||||||
|
use axum::extract::{Query, State};
|
||||||
|
use axum_extra::extract::WithRejection;
|
||||||
|
use serde::Deserialize;
|
||||||
|
use serde_json::{Value, json};
|
||||||
|
use time::OffsetDateTime;
|
||||||
|
use uuid::Uuid;
|
||||||
|
|
||||||
|
use super::St;
|
||||||
|
use crate::auth::AdminUser;
|
||||||
|
use crate::error::{ApiError, ApiResult};
|
||||||
|
use crate::views::rfc3339;
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
#[serde(deny_unknown_fields)]
|
||||||
|
pub struct ErrorQuery {
|
||||||
|
limit: Option<i64>,
|
||||||
|
offset: Option<i64>,
|
||||||
|
/// `analysis`, `question` or `preview`.
|
||||||
|
source: Option<String>,
|
||||||
|
/// `retry`, `fallback` or `failed`.
|
||||||
|
outcome: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
type Row = (i64, OffsetDateTime, String, Uuid, String, String, i32, String, String, Option<i32>, Option<String>);
|
||||||
|
|
||||||
|
pub async fn list(
|
||||||
|
State(state): St,
|
||||||
|
_admin: AdminUser,
|
||||||
|
WithRejection(Query(query), _): WithRejection<Query<ErrorQuery>, ApiError>,
|
||||||
|
) -> ApiResult<Json<Value>> {
|
||||||
|
let limit = query.limit.unwrap_or(100).clamp(1, 500);
|
||||||
|
let offset = query.offset.unwrap_or(0).max(0);
|
||||||
|
// Whose text it was: the requester of the analysis or the asker.
|
||||||
|
let rows: Vec<Row> = sqlx::query_as(
|
||||||
|
"SELECT e.id, e.created_at, e.source, e.item_id, e.kind, e.model, e.attempt, e.outcome, e.error, e.seconds, \
|
||||||
|
u.email \
|
||||||
|
FROM generation_errors e \
|
||||||
|
LEFT JOIN analyses a ON e.source = 'analysis' AND a.id = e.item_id \
|
||||||
|
LEFT JOIN questions q ON e.source = 'question' AND q.id = e.item_id \
|
||||||
|
LEFT JOIN prompt_previews p ON e.source = 'preview' AND p.id = e.item_id \
|
||||||
|
LEFT JOIN users u ON u.id = coalesce(a.requested_by, q.asked_by, p.created_by) \
|
||||||
|
WHERE ($1::text IS NULL OR e.source = $1) AND ($2::text IS NULL OR e.outcome = $2) \
|
||||||
|
ORDER BY e.id DESC LIMIT $3 OFFSET $4",
|
||||||
|
)
|
||||||
|
.bind(&query.source)
|
||||||
|
.bind(&query.outcome)
|
||||||
|
.bind(limit)
|
||||||
|
.bind(offset)
|
||||||
|
.fetch_all(&state.db)
|
||||||
|
.await?;
|
||||||
|
let (total, last_day): (i64, i64) = sqlx::query_as(
|
||||||
|
"SELECT count(*), count(*) FILTER (WHERE created_at > now() - interval '1 day') FROM generation_errors \
|
||||||
|
WHERE ($1::text IS NULL OR source = $1) AND ($2::text IS NULL OR outcome = $2)",
|
||||||
|
)
|
||||||
|
.bind(&query.source)
|
||||||
|
.bind(&query.outcome)
|
||||||
|
.fetch_one(&state.db)
|
||||||
|
.await?;
|
||||||
|
let errors: Vec<Value> = rows
|
||||||
|
.into_iter()
|
||||||
|
.map(|(id, at, source, item, kind, model, attempt, outcome, error, seconds, email)| {
|
||||||
|
json!({
|
||||||
|
"id": id, "created_at": rfc3339(at), "source": source, "item_id": item, "kind": kind, "model": model,
|
||||||
|
"attempt": attempt, "outcome": outcome, "error": error, "seconds": seconds, "user": email,
|
||||||
|
})
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
Ok(Json(json!({ "errors": errors, "total": total, "last_day": last_day, "limit": limit, "offset": offset })))
|
||||||
|
}
|
||||||
@@ -109,7 +109,8 @@ pub async fn restart(
|
|||||||
let (model, fallback, prompt_version) = super::questions::model_and_version(&state, &language).await?;
|
let (model, fallback, prompt_version) = super::questions::model_and_version(&state, &language).await?;
|
||||||
sqlx::query(
|
sqlx::query(
|
||||||
"UPDATE questions SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
"UPDATE questions SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
||||||
prompt_version = $4, attempts = 0, next_attempt_at = now(), started_at = NULL, error = NULL \
|
prompt_version = $4, attempts = 0, next_attempt_at = now(), started_at = NULL, error = NULL, \
|
||||||
|
requested_at = now() \
|
||||||
WHERE id = $1 AND status IN ('pending', 'running')",
|
WHERE id = $1 AND status IN ('pending', 'running')",
|
||||||
)
|
)
|
||||||
.bind(id)
|
.bind(id)
|
||||||
@@ -127,7 +128,7 @@ pub async fn restart(
|
|||||||
sqlx::query(
|
sqlx::query(
|
||||||
"UPDATE analyses SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
"UPDATE analyses SET status = 'pending', model = $2, fallback_model = $3, on_fallback = false, \
|
||||||
cache_key = $4, prompt_version = $5, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
cache_key = $4, prompt_version = $5, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
||||||
error = NULL \
|
error = NULL, requested_at = now() \
|
||||||
WHERE id = $1 AND status IN ('pending', 'running')",
|
WHERE id = $1 AND status IN ('pending', 'running')",
|
||||||
)
|
)
|
||||||
.bind(id)
|
.bind(id)
|
||||||
|
|||||||
@@ -10,6 +10,7 @@ mod admin;
|
|||||||
mod analyses;
|
mod analyses;
|
||||||
mod auth;
|
mod auth;
|
||||||
mod charts;
|
mod charts;
|
||||||
|
mod errors;
|
||||||
mod forecasts;
|
mod forecasts;
|
||||||
mod invitations;
|
mod invitations;
|
||||||
mod jobs;
|
mod jobs;
|
||||||
@@ -117,6 +118,7 @@ pub fn router(state: Arc<AppState>) -> Router {
|
|||||||
.route("/v1/admin/prompts/{kind}/{language}", axum::routing::put(prompts::save))
|
.route("/v1/admin/prompts/{kind}/{language}", axum::routing::put(prompts::save))
|
||||||
.route("/v1/admin/prompts/{kind}/{language}/preview", post(prompts::start_preview))
|
.route("/v1/admin/prompts/{kind}/{language}/preview", post(prompts::start_preview))
|
||||||
.route("/v1/admin/previews/{id}", get(prompts::get_preview))
|
.route("/v1/admin/previews/{id}", get(prompts::get_preview))
|
||||||
|
.route("/v1/admin/errors", get(errors::list))
|
||||||
.route("/v1/admin/jobs", get(jobs::list))
|
.route("/v1/admin/jobs", get(jobs::list))
|
||||||
.route("/v1/admin/jobs/restart", post(jobs::restart))
|
.route("/v1/admin/jobs/restart", post(jobs::restart))
|
||||||
.route("/v1/admin/models", get(models::list))
|
.route("/v1/admin/models", get(models::list))
|
||||||
|
|||||||
@@ -39,10 +39,13 @@ pub struct QuestionRow {
|
|||||||
pub credits_charged: i32,
|
pub credits_charged: i32,
|
||||||
pub created_at: OffsetDateTime,
|
pub created_at: OffsetDateTime,
|
||||||
pub completed_at: Option<OffsetDateTime>,
|
pub completed_at: Option<OffsetDateTime>,
|
||||||
|
pub attempts: i32,
|
||||||
|
pub next_attempt_at: OffsetDateTime,
|
||||||
|
pub requested_at: OffsetDateTime,
|
||||||
}
|
}
|
||||||
|
|
||||||
const QUESTION_COLUMNS: &str =
|
const QUESTION_COLUMNS: &str = "id, analysis_id, question, status, answer, error, credits_charged, created_at, \
|
||||||
"id, analysis_id, question, status, answer, error, credits_charged, created_at, completed_at";
|
completed_at, attempts, next_attempt_at, requested_at";
|
||||||
|
|
||||||
impl QuestionRow {
|
impl QuestionRow {
|
||||||
pub fn view(&self) -> Value {
|
pub fn view(&self) -> Value {
|
||||||
@@ -56,6 +59,9 @@ impl QuestionRow {
|
|||||||
"credits_charged": self.credits_charged,
|
"credits_charged": self.credits_charged,
|
||||||
"created_at": rfc3339(self.created_at),
|
"created_at": rfc3339(self.created_at),
|
||||||
"completed_at": rfc3339_opt(self.completed_at),
|
"completed_at": rfc3339_opt(self.completed_at),
|
||||||
|
"attempts": self.attempts,
|
||||||
|
"next_attempt_at": rfc3339(self.next_attempt_at),
|
||||||
|
"requested_at": rfc3339(self.requested_at),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -191,7 +197,8 @@ pub async fn regenerate(
|
|||||||
let row: QuestionRow = sqlx::query_as(&format!(
|
let row: QuestionRow = sqlx::query_as(&format!(
|
||||||
"UPDATE questions SET status = 'pending', answer = NULL, error = NULL, model_used = NULL, input_tokens = NULL, \
|
"UPDATE questions SET status = 'pending', answer = NULL, error = NULL, model_used = NULL, input_tokens = NULL, \
|
||||||
output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
output_tokens = NULL, cost_usd = NULL, attempts = 0, next_attempt_at = now(), started_at = NULL, \
|
||||||
completed_at = NULL, model = $2, prompt_version = $3, fallback_model = $4, on_fallback = false \
|
completed_at = NULL, model = $2, prompt_version = $3, fallback_model = $4, on_fallback = false, \
|
||||||
|
requested_at = now() \
|
||||||
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {QUESTION_COLUMNS}"
|
WHERE id = $1 AND status IN ('done', 'failed') RETURNING {QUESTION_COLUMNS}"
|
||||||
))
|
))
|
||||||
.bind(id)
|
.bind(id)
|
||||||
|
|||||||
@@ -1695,3 +1695,44 @@ async fn admins_restart_open_generations_with_the_current_models(pool: PgPool) {
|
|||||||
sqlx::query("UPDATE analyses SET status = 'done', content = '{}'").execute(&pool).await.unwrap();
|
sqlx::query("UPDATE analyses SET status = 'done', content = '{}'").execute(&pool).await.unwrap();
|
||||||
assert_eq!(app.post("/v1/admin/jobs/restart", Some(&admin), json!({})).await.body["restarted"], 0);
|
assert_eq!(app.post("/v1/admin/jobs/restart", Some(&admin), json!({})).await.body["restarted"], 0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[sqlx::test(migrator = "db::MIGRATOR")]
|
||||||
|
async fn admins_read_the_error_log(pool: PgPool) {
|
||||||
|
let app = TestApp::new(pool.clone()).await;
|
||||||
|
let admin = app.register("admin@example.com", "Admin").await;
|
||||||
|
let anna = app.register("anna@example.com", "Anna").await;
|
||||||
|
let chart = app.chart(&anna, "Saját", true, "1990-03-15").await;
|
||||||
|
let chart_id = chart["id"].as_str().unwrap();
|
||||||
|
let analysis: uuid::Uuid = sqlx::query_scalar("SELECT id FROM analyses WHERE chart_id = $1::uuid")
|
||||||
|
.bind(chart_id)
|
||||||
|
.fetch_one(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
for (attempt, outcome) in [(1, "retry"), (2, "failed")] {
|
||||||
|
sqlx::query(
|
||||||
|
"INSERT INTO generation_errors (source, item_id, kind, model, attempt, outcome, error, seconds) \
|
||||||
|
VALUES ('analysis', $1, 'summary', 'cheap/model', $2, $3, 'answer cut off at the token limit', 301)",
|
||||||
|
)
|
||||||
|
.bind(analysis)
|
||||||
|
.bind(attempt)
|
||||||
|
.bind(outcome)
|
||||||
|
.execute(&pool)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
}
|
||||||
|
assert_eq!(app.get("/v1/admin/errors", Some(&anna)).await.status, StatusCode::FORBIDDEN);
|
||||||
|
let r = app.get("/v1/admin/errors", Some(&admin)).await;
|
||||||
|
assert_eq!((r.status, &r.body["total"], &r.body["last_day"]), (StatusCode::OK, &json!(2), &json!(2)), "{}", r.body);
|
||||||
|
let first = &r.body["errors"][0];
|
||||||
|
assert_eq!(
|
||||||
|
(&first["outcome"], &first["attempt"], &first["user"], &first["seconds"]),
|
||||||
|
(&json!("failed"), &json!(2), &json!("anna@example.com"), &json!(301))
|
||||||
|
);
|
||||||
|
let r = app.get("/v1/admin/errors?outcome=retry", Some(&admin)).await;
|
||||||
|
assert_eq!(r.body["errors"].as_array().unwrap().len(), 1);
|
||||||
|
assert_eq!(app.get("/v1/admin/errors?source=question", Some(&admin)).await.body["total"], 0);
|
||||||
|
// users see the attempts of their analysis (the page says it is tried again)
|
||||||
|
let r = app.get(&format!("/v1/analyses/{analysis}"), Some(&anna)).await;
|
||||||
|
assert_eq!(r.body["attempts"], 0);
|
||||||
|
assert!(r.body["next_attempt_at"].is_string());
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,155 @@
|
|||||||
|
"use client";
|
||||||
|
|
||||||
|
import { useCallback, useEffect, useState } from "react";
|
||||||
|
|
||||||
|
import { api, errorText } from "@/lib/api";
|
||||||
|
import { formatDate } from "@/lib/astro";
|
||||||
|
import type { GenerationError } from "@/lib/types";
|
||||||
|
|
||||||
|
const PAGE = 100;
|
||||||
|
|
||||||
|
const SOURCES: Record<string, string> = { analysis: "Elemzés", question: "Kérdés-válasz", preview: "Prompt-próba" };
|
||||||
|
const KINDS: Record<string, string> = {
|
||||||
|
summary: "összefoglaló",
|
||||||
|
natal: "képletelemzés",
|
||||||
|
synastry: "kapcsolati",
|
||||||
|
forecast: "előrejelzés",
|
||||||
|
question: "válasz",
|
||||||
|
};
|
||||||
|
const OUTCOMES: Record<GenerationError["outcome"], { label: string; tone: string }> = {
|
||||||
|
retry: { label: "újrapróbálva", tone: "neutral" },
|
||||||
|
fallback: { label: "tartalék modell vette át", tone: "neutral" },
|
||||||
|
failed: { label: "végleg sikertelen", tone: "tense" },
|
||||||
|
};
|
||||||
|
|
||||||
|
interface Page {
|
||||||
|
errors: GenerationError[];
|
||||||
|
total: number;
|
||||||
|
last_day: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The admin error log with filters and paging. */
|
||||||
|
export function ErrorLog() {
|
||||||
|
const [source, setSource] = useState("");
|
||||||
|
const [outcome, setOutcome] = useState("");
|
||||||
|
const [errors, setErrors] = useState<GenerationError[] | null>(null);
|
||||||
|
const [total, setTotal] = useState(0);
|
||||||
|
const [lastDay, setLastDay] = useState(0);
|
||||||
|
const [busy, setBusy] = useState(false);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
const load = useCallback(
|
||||||
|
async (offset: number) => {
|
||||||
|
setBusy(true);
|
||||||
|
setError(null);
|
||||||
|
const params = new URLSearchParams({ limit: String(PAGE), offset: String(offset) });
|
||||||
|
if (source) params.set("source", source);
|
||||||
|
if (outcome) params.set("outcome", outcome);
|
||||||
|
try {
|
||||||
|
const page = await api<Page>(`/admin/errors?${params}`);
|
||||||
|
setErrors((all) => (offset === 0 ? page.errors : [...(all ?? []), ...page.errors]));
|
||||||
|
setTotal(page.total);
|
||||||
|
setLastDay(page.last_day);
|
||||||
|
} catch (e) {
|
||||||
|
setError(errorText(e));
|
||||||
|
} finally {
|
||||||
|
setBusy(false);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
[source, outcome],
|
||||||
|
);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
load(0);
|
||||||
|
}, [load]);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section className="card" style={{ gap: 12 }}>
|
||||||
|
<div className="row" style={{ gap: 12, flexWrap: "wrap", alignItems: "flex-end" }}>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="error-source">Mi</label>
|
||||||
|
<select id="error-source" className="inp" value={source} onChange={(e) => setSource(e.target.value)}>
|
||||||
|
<option value="">mind</option>
|
||||||
|
{Object.entries(SOURCES).map(([v, l]) => (
|
||||||
|
<option key={v} value={v}>
|
||||||
|
{l}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<div className="field">
|
||||||
|
<label htmlFor="error-outcome">Következmény</label>
|
||||||
|
<select id="error-outcome" className="inp" value={outcome} onChange={(e) => setOutcome(e.target.value)}>
|
||||||
|
<option value="">mind</option>
|
||||||
|
{Object.entries(OUTCOMES).map(([v, o]) => (
|
||||||
|
<option key={v} value={v}>
|
||||||
|
{o.label}
|
||||||
|
</option>
|
||||||
|
))}
|
||||||
|
</select>
|
||||||
|
</div>
|
||||||
|
<p className="hint" style={{ margin: 0 }}>
|
||||||
|
{total} bejegyzés, ebből {lastDay} az elmúlt 24 órában
|
||||||
|
</p>
|
||||||
|
<button type="button" className="btn btn-sm btn-ghost" onClick={() => load(0)} disabled={busy}>
|
||||||
|
Frissítés
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
{error && (
|
||||||
|
<p className="msg msg-error" role="alert">
|
||||||
|
{error}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
{errors === null ? (
|
||||||
|
<p className="muted">Betöltés…</p>
|
||||||
|
) : errors.length === 0 ? (
|
||||||
|
<p className="muted">Nincs hiba. 🎉</p>
|
||||||
|
) : (
|
||||||
|
<div className="table-wrap">
|
||||||
|
<table className="error-log">
|
||||||
|
<thead>
|
||||||
|
<tr>
|
||||||
|
<th>Időpont</th>
|
||||||
|
<th>Mi</th>
|
||||||
|
<th>Modell</th>
|
||||||
|
<th>Kísérlet</th>
|
||||||
|
<th>Következmény</th>
|
||||||
|
<th>Hiba</th>
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{errors.map((e) => (
|
||||||
|
<tr key={e.id}>
|
||||||
|
<td className="num">{formatDate(e.created_at, true)}</td>
|
||||||
|
<td>
|
||||||
|
{SOURCES[e.source] ?? e.source}
|
||||||
|
{e.source !== "question" && ` (${KINDS[e.kind] ?? e.kind})`}
|
||||||
|
{e.user && <span className="hint">{e.user}</span>}
|
||||||
|
<code className="hint" title={e.item_id}>
|
||||||
|
{e.item_id.slice(0, 8)}
|
||||||
|
</code>
|
||||||
|
</td>
|
||||||
|
<td>
|
||||||
|
<code>{e.model}</code>
|
||||||
|
</td>
|
||||||
|
<td className="num">
|
||||||
|
{e.attempt}.{e.seconds !== null && <span className="hint">{e.seconds} mp</span>}
|
||||||
|
</td>
|
||||||
|
<td>
|
||||||
|
<span className={`outcome tone-${OUTCOMES[e.outcome].tone}`}>{OUTCOMES[e.outcome].label}</span>
|
||||||
|
</td>
|
||||||
|
<td className="error-text">{e.error}</td>
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{errors && errors.length < total && (
|
||||||
|
<button type="button" className="btn btn-sm" onClick={() => load(errors.length)} disabled={busy}>
|
||||||
|
{busy ? "Betöltés…" : "Régebbiek"}
|
||||||
|
</button>
|
||||||
|
)}
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
import type { Metadata } from "next";
|
||||||
|
import { notFound } from "next/navigation";
|
||||||
|
|
||||||
|
import { AdminNav } from "@/components/AdminNav";
|
||||||
|
import { requireMe } from "@/lib/server";
|
||||||
|
|
||||||
|
import { ErrorLog } from "./ErrorLog";
|
||||||
|
|
||||||
|
export const metadata: Metadata = { title: "Hibanapló" };
|
||||||
|
|
||||||
|
export default async function AdminErrorsPage() {
|
||||||
|
const me = await requireMe("/admin/hibak");
|
||||||
|
if (!me.user.roles.includes("admin")) notFound();
|
||||||
|
return (
|
||||||
|
<main className="page" style={{ maxWidth: 1180 }}>
|
||||||
|
<AdminNav current="/admin/hibak" />
|
||||||
|
<div className="page-head">
|
||||||
|
<h1>Hibanapló</h1>
|
||||||
|
<p className="muted">
|
||||||
|
Az elemzések, válaszok és prompt-próbák minden sikertelen próbálkozása, a legújabb elöl. A worker írja; 90 napig
|
||||||
|
őrizzük.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
<ErrorLog />
|
||||||
|
</main>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -647,3 +647,13 @@ a.pair-row:hover strong { color: var(--gold); }
|
|||||||
.job-row { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 12px; align-items: center; padding: 8px 12px; border: 1px solid var(--line); border-radius: 8px; }
|
.job-row { display: grid; grid-template-columns: minmax(0, 1fr) auto auto; gap: 12px; align-items: center; padding: 8px 12px; border: 1px solid var(--line); border-radius: 8px; }
|
||||||
.job-row > span:first-child { display: flex; flex-direction: column; min-width: 0; }
|
.job-row > span:first-child { display: flex; flex-direction: column; min-width: 0; }
|
||||||
@media (max-width: 640px) { .job-row { grid-template-columns: minmax(0, 1fr) auto; } .job-row .model-price { grid-column: 1 / -1; align-items: flex-start; } }
|
@media (max-width: 640px) { .job-row { grid-template-columns: minmax(0, 1fr) auto; } .job-row .model-price { grid-column: 1 / -1; align-items: flex-start; } }
|
||||||
|
.retry-notice { display: flex; flex-direction: column; gap: 4px; }
|
||||||
|
.retry-notice p { margin: 0; }
|
||||||
|
|
||||||
|
/* admin: error log */
|
||||||
|
.error-log { width: 100%; border-collapse: collapse; font-size: 14px; }
|
||||||
|
.error-log th, .error-log td { text-align: left; vertical-align: top; padding: 8px 10px; border-bottom: 1px solid var(--line); }
|
||||||
|
.error-log td > .hint, .error-log td > code { display: block; }
|
||||||
|
.error-log .error-text { overflow-wrap: anywhere; max-width: 460px; }
|
||||||
|
.error-log .outcome { white-space: nowrap; }
|
||||||
|
.error-log .outcome.tone-tense { color: var(--tense); font-weight: 600; }
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ const PAGES = [
|
|||||||
{ href: "/admin/meghivok", label: "Meghívók" },
|
{ href: "/admin/meghivok", label: "Meghívók" },
|
||||||
{ href: "/admin/promptok", label: "Promptok" },
|
{ href: "/admin/promptok", label: "Promptok" },
|
||||||
{ href: "/admin/modellek", label: "Modellek" },
|
{ href: "/admin/modellek", label: "Modellek" },
|
||||||
|
{ href: "/admin/hibak", label: "Hibanapló" },
|
||||||
];
|
];
|
||||||
|
|
||||||
/** Switches between the admin pages. */
|
/** Switches between the admin pages. */
|
||||||
|
|||||||
@@ -4,13 +4,31 @@ import { useEffect, useState } from "react";
|
|||||||
|
|
||||||
import { ZodiacWheel } from "./ZodiacWheel";
|
import { ZodiacWheel } from "./ZodiacWheel";
|
||||||
|
|
||||||
/** Slowly turning wheel with the elapsed time, while the LLM writes. */
|
/**
|
||||||
export function AnalysisLoading({ title, text, compact = false }: { title?: string; text: string; compact?: boolean }) {
|
* Slowly turning wheel with the elapsed time, while the LLM writes. The time
|
||||||
const [seconds, setSeconds] = useState(0);
|
* counts from `since` (when the text was asked for), so it does not start
|
||||||
|
* again from zero after a reload; without it, from when the wheel appeared.
|
||||||
|
*/
|
||||||
|
export function AnalysisLoading({
|
||||||
|
title,
|
||||||
|
text,
|
||||||
|
compact = false,
|
||||||
|
since,
|
||||||
|
}: {
|
||||||
|
title?: string;
|
||||||
|
text: string;
|
||||||
|
compact?: boolean;
|
||||||
|
since?: string;
|
||||||
|
}) {
|
||||||
|
const [start] = useState(() => {
|
||||||
|
const t = since ? Date.parse(since) : NaN;
|
||||||
|
return Number.isNaN(t) ? Date.now() : Math.min(t, Date.now());
|
||||||
|
});
|
||||||
|
const [seconds, setSeconds] = useState(() => Math.floor((Date.now() - start) / 1000));
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
const t = setInterval(() => setSeconds((s) => s + 1), 1000);
|
const t = setInterval(() => setSeconds(Math.floor((Date.now() - start) / 1000)), 1000);
|
||||||
return () => clearInterval(t);
|
return () => clearInterval(t);
|
||||||
}, []);
|
}, [start]);
|
||||||
const elapsed = `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, "0")}`;
|
const elapsed = `${Math.floor(seconds / 60)}:${String(seconds % 60).padStart(2, "0")}`;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -21,7 +39,7 @@ export function AnalysisLoading({ title, text, compact = false }: { title?: stri
|
|||||||
<p className="muted" role="status">
|
<p className="muted" role="status">
|
||||||
{text}
|
{text}
|
||||||
</p>
|
</p>
|
||||||
<p className="hint" aria-hidden="true">
|
<p className="hint" aria-hidden="true" suppressHydrationWarning>
|
||||||
{elapsed}
|
{elapsed}
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import type { Question, UnitBalance } from "@/lib/types";
|
|||||||
import { AnalysisLoading } from "./AnalysisLoading";
|
import { AnalysisLoading } from "./AnalysisLoading";
|
||||||
import { PackageChooser } from "./PackageCards";
|
import { PackageChooser } from "./PackageCards";
|
||||||
import { RegenerateButton } from "./RegenerateButton";
|
import { RegenerateButton } from "./RegenerateButton";
|
||||||
|
import { RetryNotice } from "./RetryNotice";
|
||||||
import { Paragraphs } from "./Rich";
|
import { Paragraphs } from "./Rich";
|
||||||
|
|
||||||
const MAX_CHARS = 600;
|
const MAX_CHARS = 600;
|
||||||
@@ -139,12 +140,21 @@ export function AnalysisQuestions({
|
|||||||
<RegenerateButton<Question> url={`/admin/questions/${q.id}/regenerate`} onStarted={replace} />
|
<RegenerateButton<Question> url={`/admin/questions/${q.id}/regenerate`} onStarted={replace} />
|
||||||
)}
|
)}
|
||||||
<p className="msg msg-error" role="alert">
|
<p className="msg msg-error" role="alert">
|
||||||
Erre most nem sikerült válaszolni{q.credits_charged > 0 ? ", a kérdést visszaadtuk" : ""}. Kérdezz újra
|
Erre többszöri próbálkozás után sem sikerült válaszolni
|
||||||
egy kicsit később.
|
{q.credits_charged > 0 ? ". A kérdést visszaadtuk, így ingyen újra felteheted" : ""}. Kérdezz újra egy
|
||||||
|
kicsit később.
|
||||||
</p>
|
</p>
|
||||||
|
{isAdmin && q.error && <p className="hint">Hiba: {q.error}</p>}
|
||||||
</div>
|
</div>
|
||||||
) : (
|
) : (
|
||||||
<AnalysisLoading compact text="Készül a válasz, ez általában fél perc…" />
|
<>
|
||||||
|
<AnalysisLoading
|
||||||
|
compact
|
||||||
|
text="Készül a válasz, ez általában fél perc…"
|
||||||
|
since={q.requested_at ?? q.created_at}
|
||||||
|
/>
|
||||||
|
<RetryNotice status={q.status} error={q.error} nextAttemptAt={q.next_attempt_at} isAdmin={isAdmin} />
|
||||||
|
</>
|
||||||
)}
|
)}
|
||||||
</article>
|
</article>
|
||||||
))}
|
))}
|
||||||
|
|||||||
@@ -7,6 +7,7 @@ import type { Analysis } from "@/lib/types";
|
|||||||
|
|
||||||
import { AnalysisLoading } from "./AnalysisLoading";
|
import { AnalysisLoading } from "./AnalysisLoading";
|
||||||
import { RegenerateButton } from "./RegenerateButton";
|
import { RegenerateButton } from "./RegenerateButton";
|
||||||
|
import { RetryNotice } from "./RetryNotice";
|
||||||
import { Rich } from "./Rich";
|
import { Rich } from "./Rich";
|
||||||
import { openTab } from "./Tabs";
|
import { openTab } from "./Tabs";
|
||||||
|
|
||||||
@@ -102,7 +103,21 @@ export function ChartSummary({
|
|||||||
Az összefoglaló most nem készült el. Töltsd újra az oldalt egy kicsit később.
|
Az összefoglaló most nem készült el. Töltsd újra az oldalt egy kicsit később.
|
||||||
</p>
|
</p>
|
||||||
) : (
|
) : (
|
||||||
<AnalysisLoading compact text="Az összefoglalód készül, ez néhány másodperc…" />
|
<>
|
||||||
|
<AnalysisLoading
|
||||||
|
compact
|
||||||
|
text="Az összefoglalód készül, ez néhány másodperc…"
|
||||||
|
since={summary?.requested_at ?? summary?.created_at}
|
||||||
|
/>
|
||||||
|
{summary && (
|
||||||
|
<RetryNotice
|
||||||
|
status={summary.status}
|
||||||
|
error={summary.error}
|
||||||
|
nextAttemptAt={summary.next_attempt_at}
|
||||||
|
isAdmin={isAdmin}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
</>
|
||||||
)}
|
)}
|
||||||
</>
|
</>
|
||||||
)}
|
)}
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import { AnalysisLoading } from "./AnalysisLoading";
|
|||||||
import { AnalysisQuestions } from "./AnalysisQuestions";
|
import { AnalysisQuestions } from "./AnalysisQuestions";
|
||||||
import { PackageChooser } from "./PackageCards";
|
import { PackageChooser } from "./PackageCards";
|
||||||
import { RegenerateButton } from "./RegenerateButton";
|
import { RegenerateButton } from "./RegenerateButton";
|
||||||
|
import { RetryNotice } from "./RetryNotice";
|
||||||
import { AnalysisView } from "./AnalysisView";
|
import { AnalysisView } from "./AnalysisView";
|
||||||
|
|
||||||
const POLL_MS = 5000;
|
const POLL_MS = 5000;
|
||||||
@@ -172,7 +173,15 @@ export function DetailedAnalysis({
|
|||||||
return (
|
return (
|
||||||
<section className="analysis-card" aria-label={t.eyebrow} aria-busy="true">
|
<section className="analysis-card" aria-label={t.eyebrow} aria-busy="true">
|
||||||
<span className="eyebrow">{t.eyebrow}</span>
|
<span className="eyebrow">{t.eyebrow}</span>
|
||||||
<AnalysisLoading title={t.loadingTitle} text={t.loading} />
|
<AnalysisLoading title={t.loadingTitle} text={t.loading} since={analysis?.requested_at ?? analysis?.created_at} />
|
||||||
|
{analysis && (
|
||||||
|
<RetryNotice
|
||||||
|
status={analysis.status}
|
||||||
|
error={analysis.error}
|
||||||
|
nextAttemptAt={analysis.next_attempt_at}
|
||||||
|
isAdmin={isAdmin}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
</section>
|
</section>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
@@ -186,9 +195,15 @@ export function DetailedAnalysis({
|
|||||||
<p className="muted">{t.about}</p>
|
<p className="muted">{t.about}</p>
|
||||||
<p className="ai-line">{AI_SHORT}</p>
|
<p className="ai-line">{AI_SHORT}</p>
|
||||||
{analysis?.status === "failed" && (
|
{analysis?.status === "failed" && (
|
||||||
<p className="msg msg-error" role="alert">
|
<div className="msg msg-error" role="alert">
|
||||||
Az előző kérés nem sikerült{isAdmin ? "" : ", az elemzést visszaadtuk a keretedbe"}. Próbáld újra.
|
<p style={{ margin: 0 }}>
|
||||||
|
Az elemzés többszöri próbálkozás után sem készült el.
|
||||||
|
{isAdmin || analysis.credits_charged === 0
|
||||||
|
? " Próbáld újra."
|
||||||
|
: " A felhasznált egységet visszaadtuk, így ingyen újra kérheted."}
|
||||||
</p>
|
</p>
|
||||||
|
{isAdmin && analysis.error && <p className="hint">Hiba: {analysis.error}</p>}
|
||||||
|
</div>
|
||||||
)}
|
)}
|
||||||
{error && (
|
{error && (
|
||||||
<p className="msg msg-error" role="alert">
|
<p className="msg msg-error" role="alert">
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
/**
|
||||||
|
* Shown while a text is being written again after a failed attempt: it is
|
||||||
|
* retried on its own, nothing to do. Admins also see the error.
|
||||||
|
*/
|
||||||
|
export function RetryNotice({
|
||||||
|
status,
|
||||||
|
error,
|
||||||
|
nextAttemptAt,
|
||||||
|
isAdmin = false,
|
||||||
|
}: {
|
||||||
|
status: string;
|
||||||
|
error?: string | null;
|
||||||
|
nextAttemptAt?: string;
|
||||||
|
isAdmin?: boolean;
|
||||||
|
}) {
|
||||||
|
if (!error || (status !== "pending" && status !== "running")) return null;
|
||||||
|
const wait = nextAttemptAt ? Math.round((Date.parse(nextAttemptAt) - Date.now()) / 60_000) : 0;
|
||||||
|
return (
|
||||||
|
<div className="msg retry-notice" role="status">
|
||||||
|
<p>
|
||||||
|
Az előző próbálkozás nem sikerült, ezért automatikusan újrapróbáljuk. Nem kell újra kérned, és nem kerül
|
||||||
|
többe.
|
||||||
|
{status === "pending" && wait >= 1 ? ` A következő próbálkozás kb. ${wait} perc múlva indul.` : ""}
|
||||||
|
</p>
|
||||||
|
{isAdmin && <p className="hint">Hiba: {error}</p>}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -77,6 +77,9 @@ export interface Question {
|
|||||||
credits_charged: number;
|
credits_charged: number;
|
||||||
created_at: string;
|
created_at: string;
|
||||||
completed_at: string | null;
|
completed_at: string | null;
|
||||||
|
attempts?: number;
|
||||||
|
next_attempt_at?: string;
|
||||||
|
requested_at?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface PublicConfig {
|
export interface PublicConfig {
|
||||||
@@ -232,6 +235,26 @@ export interface Analysis {
|
|||||||
credits_charged: number;
|
credits_charged: number;
|
||||||
created_at: string;
|
created_at: string;
|
||||||
completed_at: string | null;
|
completed_at: string | null;
|
||||||
|
/** Attempts so far, and when the next one starts (while it is retried). */
|
||||||
|
attempts?: number;
|
||||||
|
next_attempt_at?: string;
|
||||||
|
/** When the current text was asked for: the waiting page counts from here. */
|
||||||
|
requested_at?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Admin: one failed attempt in the error log. */
|
||||||
|
export interface GenerationError {
|
||||||
|
id: number;
|
||||||
|
created_at: string;
|
||||||
|
source: "analysis" | "question" | "preview";
|
||||||
|
item_id: string;
|
||||||
|
kind: string;
|
||||||
|
model: string;
|
||||||
|
attempt: number;
|
||||||
|
outcome: "retry" | "fallback" | "failed";
|
||||||
|
error: string;
|
||||||
|
seconds: number | null;
|
||||||
|
user: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface AdminUser {
|
export interface AdminUser {
|
||||||
|
|||||||
@@ -253,6 +253,8 @@ impl Analyzer {
|
|||||||
.await
|
.await
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
|
self.log_error("preview", job.id, &job.kind, &job.model, 1, "failed", e.message(), started.elapsed())
|
||||||
|
.await;
|
||||||
sqlx::query(
|
sqlx::query(
|
||||||
"UPDATE prompt_previews SET status = 'failed', error = $2, completed_at = now() \
|
"UPDATE prompt_previews SET status = 'failed', error = $2, completed_at = now() \
|
||||||
WHERE id = $1 AND status = 'running'",
|
WHERE id = $1 AND status = 'running'",
|
||||||
@@ -322,7 +324,24 @@ impl Analyzer {
|
|||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let can_fall_back = !job.on_fallback && job.fallback_model.is_some();
|
let can_fall_back = !job.on_fallback && job.fallback_model.is_some();
|
||||||
match next_step(&e, job.attempts, self.max_attempts, can_fall_back) {
|
let next = next_step(&e, job.attempts, self.max_attempts, can_fall_back);
|
||||||
|
let outcome = match next {
|
||||||
|
Next::Retry(_) => "retry",
|
||||||
|
Next::Fallback => "fallback",
|
||||||
|
Next::Fail => "failed",
|
||||||
|
};
|
||||||
|
self.log_error(
|
||||||
|
"analysis",
|
||||||
|
job.id,
|
||||||
|
&job.kind,
|
||||||
|
job.active_model(),
|
||||||
|
job.attempts,
|
||||||
|
outcome,
|
||||||
|
e.message(),
|
||||||
|
took,
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
match next {
|
||||||
Next::Retry(wait) => {
|
Next::Retry(wait) => {
|
||||||
sqlx::query(
|
sqlx::query(
|
||||||
"UPDATE analyses SET status = 'pending', error = $2, \
|
"UPDATE analyses SET status = 'pending', error = $2, \
|
||||||
@@ -424,6 +443,38 @@ impl Analyzer {
|
|||||||
Ok(result.rows_affected())
|
Ok(result.rows_affected())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Records a failed attempt in the admin error log (a failure here is only logged).
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
pub(crate) async fn log_error(
|
||||||
|
&self,
|
||||||
|
source: &str,
|
||||||
|
item_id: Uuid,
|
||||||
|
kind: &str,
|
||||||
|
model: &str,
|
||||||
|
attempt: i32,
|
||||||
|
outcome: &str,
|
||||||
|
error: &str,
|
||||||
|
took: Duration,
|
||||||
|
) {
|
||||||
|
let result = sqlx::query(
|
||||||
|
"INSERT INTO generation_errors (source, item_id, kind, model, attempt, outcome, error, seconds) \
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)",
|
||||||
|
)
|
||||||
|
.bind(source)
|
||||||
|
.bind(item_id)
|
||||||
|
.bind(kind)
|
||||||
|
.bind(model)
|
||||||
|
.bind(attempt)
|
||||||
|
.bind(outcome)
|
||||||
|
.bind(error.chars().take(2000).collect::<String>())
|
||||||
|
.bind(i32::try_from(took.as_secs()).unwrap_or(i32::MAX))
|
||||||
|
.execute(&self.db)
|
||||||
|
.await;
|
||||||
|
if let Err(e) = result {
|
||||||
|
tracing::error!(error = %e, "writing the error log");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Resolves when this attempt is no longer the current one (an admin
|
/// Resolves when this attempt is no longer the current one (an admin
|
||||||
/// restarted it); checked every 10 seconds, never on a database error.
|
/// restarted it); checked every 10 seconds, never on a database error.
|
||||||
async fn superseded(&self, table: &str, id: Uuid, started_at: time::OffsetDateTime) {
|
async fn superseded(&self, table: &str, id: Uuid, started_at: time::OffsetDateTime) {
|
||||||
|
|||||||
@@ -148,7 +148,15 @@ impl Analyzer {
|
|||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
let can_fall_back = !job.on_fallback && job.fallback_model.is_some();
|
let can_fall_back = !job.on_fallback && job.fallback_model.is_some();
|
||||||
match next_step(&e, job.attempts, self.max_attempts, can_fall_back) {
|
let next = next_step(&e, job.attempts, self.max_attempts, can_fall_back);
|
||||||
|
let outcome = match next {
|
||||||
|
Next::Retry(_) => "retry",
|
||||||
|
Next::Fallback => "fallback",
|
||||||
|
Next::Fail => "failed",
|
||||||
|
};
|
||||||
|
let model = active_model(&job.model, job.fallback_model.as_deref(), job.on_fallback);
|
||||||
|
self.log_error("question", job.id, "question", model, job.attempts, outcome, e.message(), took).await;
|
||||||
|
match next {
|
||||||
Next::Retry(wait) => {
|
Next::Retry(wait) => {
|
||||||
sqlx::query(
|
sqlx::query(
|
||||||
"UPDATE questions SET status = 'pending', error = $2, \
|
"UPDATE questions SET status = 'pending', error = $2, \
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ use tokio::sync::watch;
|
|||||||
const INTERVAL: Duration = Duration::from_secs(3600);
|
const INTERVAL: Duration = Duration::from_secs(3600);
|
||||||
|
|
||||||
/// (what, statement) pairs; each returns the number of deleted rows.
|
/// (what, statement) pairs; each returns the number of deleted rows.
|
||||||
const STATEMENTS: [(&str, &str); 6] = [
|
const STATEMENTS: [(&str, &str); 7] = [
|
||||||
("sessions", "DELETE FROM sessions WHERE expires_at < now()"),
|
("sessions", "DELETE FROM sessions WHERE expires_at < now()"),
|
||||||
("auth_tokens", "DELETE FROM auth_tokens WHERE expires_at < now()"),
|
("auth_tokens", "DELETE FROM auth_tokens WHERE expires_at < now()"),
|
||||||
// Sent e-mails contain links with tokens; keep them a week for support.
|
// Sent e-mails contain links with tokens; keep them a week for support.
|
||||||
@@ -26,6 +26,7 @@ const STATEMENTS: [(&str, &str); 6] = [
|
|||||||
OR (accepted_at IS NOT NULL AND accepted_by IS NULL)",
|
OR (accepted_at IS NOT NULL AND accepted_by IS NULL)",
|
||||||
),
|
),
|
||||||
("prompt_previews", "DELETE FROM prompt_previews WHERE created_at < now() - interval '30 days'"),
|
("prompt_previews", "DELETE FROM prompt_previews WHERE created_at < now() - interval '30 days'"),
|
||||||
|
("generation_errors", "DELETE FROM generation_errors WHERE created_at < now() - interval '90 days'"),
|
||||||
];
|
];
|
||||||
|
|
||||||
pub async fn run_once(db: &PgPool) -> Result<Vec<(&'static str, u64)>, sqlx::Error> {
|
pub async fn run_once(db: &PgPool) -> Result<Vec<(&'static str, u64)>, sqlx::Error> {
|
||||||
|
|||||||
@@ -212,7 +212,8 @@ async fn cleanup_removes_only_expired_data(db: PgPool) {
|
|||||||
("emails", 1),
|
("emails", 1),
|
||||||
("invites", 0),
|
("invites", 0),
|
||||||
("invitations", 0),
|
("invitations", 0),
|
||||||
("prompt_previews", 0)
|
("prompt_previews", 0),
|
||||||
|
("generation_errors", 0)
|
||||||
]
|
]
|
||||||
);
|
);
|
||||||
let (left,): (Vec<Uuid>,) =
|
let (left,): (Vec<Uuid>,) =
|
||||||
@@ -533,6 +534,18 @@ mod analyses {
|
|||||||
let (status, attempts, ..) = row(&db, id).await;
|
let (status, attempts, ..) = row(&db, id).await;
|
||||||
assert_eq!((status.as_str(), attempts), ("failed", 3));
|
assert_eq!((status.as_str(), attempts), ("failed", 3));
|
||||||
assert_eq!(refunds(&db, s.user).await, vec![1]);
|
assert_eq!(refunds(&db, s.user).await, vec![1]);
|
||||||
|
// every failed attempt is in the admin error log
|
||||||
|
let log: Vec<(String, i32, String, String)> = sqlx::query_as(
|
||||||
|
"SELECT source, attempt, outcome, model FROM generation_errors WHERE item_id = $1 ORDER BY id",
|
||||||
|
)
|
||||||
|
.bind(id)
|
||||||
|
.fetch_all(&db)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
let expected = |attempt: i32, outcome: &str| {
|
||||||
|
("analysis".to_string(), attempt, outcome.to_string(), "test/model".to_string())
|
||||||
|
};
|
||||||
|
assert_eq!(log, vec![expected(1, "retry"), expected(2, "retry"), expected(3, "failed")]);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[sqlx::test(migrator = "db::MIGRATOR")]
|
#[sqlx::test(migrator = "db::MIGRATOR")]
|
||||||
@@ -780,6 +793,16 @@ mod analyses {
|
|||||||
let models: Vec<String> =
|
let models: Vec<String> =
|
||||||
fake.requests.lock().unwrap().iter().map(|b| b["model"].as_str().unwrap().to_string()).collect();
|
fake.requests.lock().unwrap().iter().map(|b| b["model"].as_str().unwrap().to_string()).collect();
|
||||||
assert_eq!(models, ["cheap/model", "cheap/model", "test/model"]);
|
assert_eq!(models, ["cheap/model", "cheap/model", "test/model"]);
|
||||||
|
let outcomes: Vec<(String, String)> =
|
||||||
|
sqlx::query_as("SELECT outcome, model FROM generation_errors WHERE item_id = $1 ORDER BY id")
|
||||||
|
.bind(id)
|
||||||
|
.fetch_all(&db)
|
||||||
|
.await
|
||||||
|
.unwrap();
|
||||||
|
assert_eq!(
|
||||||
|
outcomes,
|
||||||
|
[("retry".to_string(), "cheap/model".to_string()), ("fallback".to_string(), "cheap/model".to_string())]
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
#[sqlx::test(migrator = "db::MIGRATOR")]
|
#[sqlx::test(migrator = "db::MIGRATOR")]
|
||||||
|
|||||||
Reference in New Issue
Block a user