feat(hands,jobs): job-run watchdog looks a week back and alarms on jobs that stopped running
This commit is contained in:
1 parent
426b4055f4
commit
91bd50a66d
6 files changed
+620
-71
No files matched your search
@@ -277,7 +277,7 @@ that 231 times, and nothing reached a human: the outage surfaced ten days later,
|
||||
by hand, out of curiosity. The record existed, the reader did not. This job is
|
||||
the reader.
|
||||
|
||||
- `прогоны` (`:27` every hour) reads `job_runs` for the last 24 hours through
|
||||
- `прогоны` (`:27` every hour) reads `job_runs` for the last 7 days through
|
||||
the same connection the scheduler itself writes with
|
||||
(`run.conversations.db.session()` and the `JobRunRecord` model) - no second
|
||||
truth about where the journal lives.
|
||||
@@ -290,8 +290,31 @@ the reader.
|
||||
over the real outage it reads: «джоб `вайбграм` лежит 23 часа и дольше,
|
||||
144 прогона подряд с 12.09 22:00 UTC: HubError: vibegram 530 bad_json». It repeats at most once a day per job, and
|
||||
when the job recovers one short `отбой` line goes out and the counter resets.
|
||||
The window is a week, not a day, so a daily job (`отметка`, `закрытие`,
|
||||
`голосовой`) gets its three failures in a row in three days.
|
||||
- **A job that stopped running is an alarm too, with its own text**: «джоб
|
||||
`закрытие` не запускался 2 суток при периоде 24 часа». For every job with a
|
||||
cron, the period comes from the registered cron (`run.scheduler.jobs`) via
|
||||
the same `every_from_cron` the neighbour watchdog uses, and the last run is
|
||||
taken from the whole journal, not the window. The threshold is **two
|
||||
periods**: one missed fire happens honestly (a redeploy landed on the cron
|
||||
minute, and the scheduler does not catch up), two in a row do not. It is
|
||||
never finer than two of the watchdog's own periods (2 hours), because an
|
||||
hourly watcher cannot tell a shorter gap from its own blindness between
|
||||
runs. Time when the scheduler was not running non-critical jobs at all is
|
||||
not counted as a miss: before the process started, and while the
|
||||
watchdog's own hourly chain is broken (the gateway was down, or the budget
|
||||
was throttled and non-critical crons were deferred without a record). Same
|
||||
discipline as failures: once a day per job, and an `отбой` when a run shows
|
||||
up.
|
||||
- What silence does not cover, on purpose: jobs without a cron (webhooks,
|
||||
events) are not measured at all; a job that has never appeared in the
|
||||
journal waits silently for its first row; and a cron that does not reduce to
|
||||
a regular period (`куратор`, `0 2,8,12,16,20 * * *`) is skipped and named in
|
||||
the log as unguarded every run, because any number put in its place would
|
||||
be made up. `куратор` still falls under the failure count.
|
||||
- **An empty answer is a statement about the query, not about health.** Zero
|
||||
rows in 24 hours (other than this job's own) means no job ran at all, and
|
||||
rows in the last 24 hours (other than this job's own) means no job ran at all, and
|
||||
that is its own alarm with its own text - the scheduler is what to look at,
|
||||
not any one job. A failed query is no reading whatsoever: the run logs it and
|
||||
stays silent, because a database that did not answer is not evidence of
|
||||
@@ -303,22 +326,15 @@ the reader.
|
||||
carrying the key can be named anything.
|
||||
|
||||
State between runs is `мета/бобер/.состояние/прогоны.json` (who was told and
|
||||
when). A lost file means a lying job is reported again and an `отбой` for
|
||||
when, separately for failures and for silence). A lost file means a lying job is reported again and an `отбой` for
|
||||
something reported before the loss never comes - the evidence for it is gone.
|
||||
|
||||
Two blind spots, both deliberate:
|
||||
|
||||
- **The watchdog cannot report its own death.** Its own runs are excluded from
|
||||
the count (`run.job.name`), because a job that is down will not file a report
|
||||
about itself - an observer inside the object does not report the object's
|
||||
death. If `прогоны` itself starts failing, nothing here will say so; that
|
||||
needs an outside eye, like the watchdog ring above.
|
||||
- **The window is 24 hours, so a job that runs less than three times a day can
|
||||
never accumulate three consecutive failures inside it.** `отметка`,
|
||||
`закрытие`, `голосовой` (daily) and `память` (weekly) are therefore
|
||||
unwatched by this job in practice; `вайбграм`, `логи`, `куратор`, `досуг`,
|
||||
`ротация` and `сторож` are inside it. Widening the counting window past the
|
||||
query window is the fix, when it is worth it.
|
||||
One blind spot, deliberate: **the watchdog cannot report its own death.** Its
|
||||
own runs are excluded from both the failure count and the silence check
|
||||
(`run.job.name`), because a job that is down will not file a report about
|
||||
itself - an observer inside the object does not report the object's death. If
|
||||
`прогоны` itself starts failing or stops running, nothing here will say so;
|
||||
that needs an outside eye, like the watchdog ring above.
|
||||
|
||||
## Policy (`beaver_agent/policy.py`)
|
||||
|
||||
|
||||
Reference in new issue
Block a user