Health and operations

Runs

How the runs that ended in the last 7, 30 or 90 days went, where they fail, and which failures still need you.

You can

• Group the failures By action or By step and pick one to narrow the list; the chip above the list clears it.
• Open an action to see it per namespace, when several namespaces are in play.
• Turn Still failing or Fixed later off to hide those runs.
• Click a row to open the run's page, or the record's name to open the record.
• Run a still-failing run again from the step that failed with Retry.

Good to know

• Still failing: no later run of the same action on that record has succeeded. Fixed later: one has.
• The Success rate counts succeeded runs out of succeeded and failed ones. Canceled runs were stopped on purpose and do not count.
• A run that failed after its steps, while the platform waited for a backup or restore to report back, counts under no step.
• This tab covers every project; the other Health tabs cover projects on a paid plan (Plans).
• Retry takes Developer in development, Developer Plus in staging and Maintainer in production (Retry and cancel).

Instances

Every app in one list, the most urgent first, with how its last run went and how fresh its backup is.

You can

• Filter by status, and by environment (Prod, Stg, Dev) in any combination.
• Include deleted apps with Deleted; Reset clears the filters.
• Click a row to open the app.

Good to know

• The status says how the app's last run went, not whether it answers right now: for that, look at its URLs.
• Deploying: a run is under way, the app has not been deployed yet, or a failed run is about to be retried.
• Error: its last run failed. Stopped: it was canceled, disabled or deleted.
• The backup badge is Healthy under 36 hours since the last good backup, Warning from 36 to 48 hours, Critical after that.
• No Backups means the app has none, or its Auto Backup is off.

Backups

Your production apps whose last good backup is late, so you can take one now.

You can

• Take a backup now with Backup, and follow it in the app's Operations.
• Turn Critical or Warning off to hide those apps.
• List the apps backed up in the last 36 hours instead with Healthy Only.
• Click a row to open the app.
• Download or restore any backup from the sidebar's Backups.

Good to know

• It watches production apps that have Auto Backup on, and no others.
• Critical: the last good backup is more than 48 hours old, or there is none. Warning: 36 to 48 hours.
• It covers production only, so it fills for Maintainers and Admins, and Backup takes the Maintainer role.

See Automatic backups for what runs each night.

URLs & SSL

The certificates on your apps' and installers' addresses that have expired, are about to, or could not be read yet.

You can

• Hide expired certificates by turning Critical off, or expiring and unread ones by turning Warning off.
• List the valid certificates too with Valid.
• Hover a status to read why the last check found no certificate, and when it looks again.
• Read the certificate the address serves now with Check, or have it issued again, as a run you can follow, with Renew.
• Open the app or installer with the eye.

Good to know

• Certificates renew on their own weeks before they expire, and each is read about once a day: Expiring Soon (7 days or less) means renewal is failing.
• Unknown means no certificate has been read yet, for example right after the address was set up.
• Check and Renew wait while a run is going on the address or its owner; meanwhile the list refreshes every 3 seconds.
• They take the role that changes the app or installer: see URLs and certificates and an installer's URLs.
• You see the addresses of the environments your role covers, and every installer's. A Viewer sees no app's certificate here.

Server Cleanup

Whether each of your deployed servers has had its Docker cleanup in the last 7 days.

You can

• Keep only the overdue servers with Needs Cleanup.
• Click a row to open the server.
• For a server that stays on Needs Cleanup, look for its cleanup run in the server's Operations.
• As a Maintainer, run one now with Cleanup in the server's More menu (Server actions).

Good to know

• Once a day, the platform cleans every deployed server not cleaned in the last 7 days, with a run named Clean Up Docker Resources.
• It removes stopped containers, images unused for 3 days, untagged images, unused networks and the build cache, and trims the system logs to 7 days.
• It removes unused volumes, skipping any that looks like a database's, and keeps only the two newest copies of each app's earlier code.
• Nothing that is running is removed, and what is removed cannot be brought back.
• A server used by several environments shows once under each.

Core Addons

The copy of Odoo's own source each environment keeps per Odoo version, which its apps share instead of each fetching their own.

You can

• Narrow the list with the environment buttons (production, staging, development) and the project picker.
• Update the core addons of every environment in the list with Update Selected, after you confirm.
• Follow each environment's progress under its name; the environments update side by side.

Good to know

• A copy holds Odoo Community, Enterprise when the environment runs it, and Odoo's themes.
• For each environment, a queue updates every deployed copy, then restarts every deployed app that uses them.
• Each restart cuts its app off for a moment, production apps included when production is in the list.
• An environment with nothing deployed, or with no server, is skipped, and a message says why.
• Update Selected takes the Admin role; you see the environments your role covers, in approved projects on a paid plan.

States

A run's state is its steps' states put together. The cards in Operations use the names below; a run's own page says them in words (in brackets).

A step that fails is tried again after a short wait, which grows with each attempt up to an hour, until it has failed three times. A step that timed out, or whose start was refused for something a person has to set (a key, a token), is not tried again on its own.

Retry and cancel

Retry runs a failed or canceled run again from the step that stopped it; the steps that succeeded are not run again. You find it on the run's card in Workflows, on its page, on its row in Health › Runs, and in the record's header (see Other actions). It is not offered when:

• a newer run of the same record exists. The card says Newer run exists and offers Open latest run;
• the run waits in its queue behind an earlier run that has not finished.

Cancel stops a run that is waiting or playing. The steps not finished are canceled; the ones done stay done. Whatever a step was doing on the server is cut off, which can leave the change half made. A canceled run can be retried.

Both ask you to confirm. They take the role that changes the record in its environment: Developer in development, Developer Plus in staging, Maintainer in production (see Who may do what).

Workflows

The runs, newest first, each with its steps, its facts and the buttons to start, retry or cancel it.

You can

• Click a card to open or close it; the newest card, and any running or waiting one, opens by itself.
• Click a step's icon in the Pipeline to see that step, as on the Runners tab.
• Start a run that was created but not started with Start; for a run in a queue, it starts the queue.
• Run it again from the step that stopped it with Retry, or stop it with Cancel (Retry and cancel).
• Cancel a run a newer one blocks, and the rest of its queue, with Abandon queue; completed steps stay.

Good to know

• A yellow line says why a run does not move: another queue, our review of its project, or a newer run of the same record.
• Retry is not offered once a newer run of the record exists: the card says Newer run exists and offers Open latest run.
• On an app or a project, Start and Cancel need a plan with automated deployments.
• These take Developer in development, Developer Plus in staging and Maintainer in production (Who may do what).

Runners

Every step of every run, one row each, with its attempts, its output and the buttons to act on it.

You can

• Filter by namespace, action and state, or search; Stuck Jobs keeps the steps nothing will retry on its own, each with why it stopped.
• Click a row's icon for the step's card, and read its output with View Output; the history icon lists every attempt.
• Run a failed step again with Retry: when it succeeds, its run carries on with the steps after it.
• Start the automatic retries over with Reset, or stop a queued or running step with Cancel.
• Run a finished step again as a new attempt, whatever its result, with Rerun.

Good to know

• A stuck step waits for you: fix the cause first, then Retry or Reset it.
• A failed step is retried on its own until it has failed three times; the card's Status says when the next try comes (States).
• On a backup or restore that goes on after its step, Watcher Log shows its log on the server, and Check Backup or Check Restore checks again.
• Rerun takes the Maintainer role; the other buttons follow Who may do what.
• On an app, Reset, Rerun and Cancel need a plan with automated deployments.

Queues

Runs that must play in order, each one starting when the one before it is done.

You can

• Start a queue that has not started with Start.
• Start one that waits for an earlier queue, without waiting, with Force start.
• Retry the first run in it that can be retried with Retry.
• Stop it with Cancel: the run in progress, every run after it and the follow-up queues that never started. Completed runs stay.

Good to know

• A run that fails holds its queue: the runs after it wait until it is retried or the queue is canceled.
• Force start may begin before what the earlier queue sets up is in place, and cannot skip a project's review.
• Besides a run's states, a queue can be Waiting (an earlier queue, or our review of the project, holds it) or Paused.
• These take Developer in development, Developer Plus in staging and Maintainer in production (Who may do what).
• On a project, the Queues tab needs a plan with automated deployments.

Reconciliation

Each time the platform read one of your Git repositories to bring its branches and commits up to date, and how it went.

You can

• Filter by namespace, variant and state; Stalled keeps the syncs nothing has moved for an hour.
• Click a row for its details: when it started and finished, how many branches it saw and manages, and a failed one's error.
• Release a stalled sync with Cancel, the ban icon.

Good to know

• A full sync reads every branch; a targeted one reads only the branches a run needed, listed in its details.
• While a sync of a repository is queued or running, no other starts for it, so a stalled one holds back every later sync.
• A sync still running may finish anyway after you cancel it.
• Cancelling takes the Maintainer role, in every environment.
• On a project, the tab lists the syncs of the project's repository.