Watcher and Restart Job Mechanism

Job Executors may perform long-running tasks and / or use external process service/products to execute jobs.

JEF has a number of mechanisms providing resilience against potential outage scenarios across the microservice architecture.

Watcher mechanism

One of the JEF strategies includes an active internal watcher running on each job instance. The Watcher can be disabled by configurations. Watcher works with ExecutionStep and non-ExecutionStep based jobs.

The Watchers are responsible to monitor all running jobs (across all job executors) in order to detect the following scenarios:

Queued Jobs

`QUEUED` jobs may be considered as “stalled” if they have not moved to a `RUNNING` state during a timeout deadline.

`QUEUED` jobs that have stalled can be taken by another job executor with an available slot.

This logic is handled by the JEF framework; no specific work is required by plugin developers.

Running Jobs

`RUNNING` jobs may be considered as “stalled” if they haven’t updated the execution step before the timeout deadline.

If a running job stalls, it is moved to a `TIMED_OUT` state.

Timed Out Jobs

`TIMED_OUT` jobs are moved back to a `RUNNING` state if the original job execution progresses before another Job Executor takes the job.

`TIMED_OUT` jobs may be considered as definitely “stalled” if they still haven’t updated the progress after a second timeout deadline.

Jobs which remain `TIMED_OUT` after 2 deadlines can be taken by any other Job Executor instance which supports the defined action and has an available job execution slot.

If a `TIMED_OUT` job is picked up by another job executor, it is moved to a `RUNNING` state.

JEF Executors implements a basic back-pressure strategy to stop getting new jobs into `RUNNING` when no execution slot is available.

Finally, `TIMED_OUT` jobs are moved to a `FAILED` state if no other Job Executor takes responsibility for it (e.g. because they don't have job execution slot capacity left).

If the original Job Executor owning the job execution is still running and it attempts to progress the job after another Job Executor has taken the job, the original Job Executor attempt will be considered as a fatal error, and is ignored.