Plan recovery and rollback decisions
Overview
Section titled “Overview”Zbix CRM separates isolated restore testing from live recovery. A failed upgrade may leave a database at a state that cannot safely be used by an older application image. Returning a release symlink or reinstalling binaries is therefore not a complete data rollback.
Purpose
Section titled “Purpose”Give the recovery owner a controlled decision process for an upgrade failure or service loss without inventing a one-click live restore capability.
Prerequisites
Section titled “Prerequisites”-
An authorized incident/change owner and a recovery operator.
-
A known recovery point with independent verification and restore-test evidence.
-
Independent access to required external keys, approved packages and destination storage.
-
A deployment-specific, reviewed live recovery procedure.
Procedure
Section titled “Procedure”-
Identify the incident scope: application-only failure, migration failure, database loss, private-file loss or host loss. Record the current package and known schema state before changing anything else.
-
Preserve the existing state and relevant sanitized logs where safe. Stop writes only through the approved operational procedure, and avoid deleting volumes or overwriting a failed target prematurely.
-
Choose between a reviewed forward repair and recovery to a known point. Confirm the data-loss window by comparing the recovery-point time with the last accepted business activity.
-
Validate artifact provenance, integrity/restore receipts, source and target identity, available keys and target isolation. Do not select an artifact solely because its filename looks recent.
-
Carry out live recovery only under the approved deployment-specific runbook. CRM exposes test restores and recovery guidance, not a live overwrite action. Provider-managed recovery requires the actual authorized provider process.
-
After recovery, verify package/schema compatibility, database and private-file consistency, identity/licensing state, readiness and worker behavior. Test representative user operations before enabling normal traffic.
-
Document the recovered point, any data-loss interval, reconciliation work and the observed restoration duration. Use those measured results to refine the disaster-recovery plan.
Expected result
Section titled “Expected result”The incident has a documented, authorized recovery decision and a validated service restoration, with any data loss and follow-up reconciliation made explicit.
Important notes
Section titled “Important notes”-
No production RPO or RTO is guaranteed by the application or by an isolated restore receipt.
-
The Linux upgrade failure path does not downgrade the database. Services remain stopped until a reviewed recovery or repair is completed.
-
Required external secrets are outside the v2 artifact. Recovery depends on their independent custody.
-
Pooled-tenant and managed-cloud PITR recovery are not qualified as ordinary self-hosted worker operations.
Troubleshooting
Section titled “Troubleshooting”-
If the required key or artifact is unavailable, escalate to the designated custody owner and preserve all remaining evidence.
-
If the target schema or files do not match the selected recovery point, do not open the service to users; resolve the mismatch with the recovery owner.
-
If provider recovery is only a queued handoff, contact the authorized deployment provider rather than treating the queue state as restoration progress.
Related articles
Section titled “Related articles”Search the guides or talk to the ZBIX team.
Software products, application engineering and IT services from Zbix Technologies.
Visit our company website ↗