Disaster Recovery

AITS-0147 Rapid Failover and Recovery Testing Project

Scope Description

Create redundant systems and processes to allow for ‘Rapid’ failover for all applications from systems at RRB (Roosevelt Road Building) to HAB (Henry Administration Building). Complete all necessary testing for failover success to establish a timeline/timeframe in the event of failover/recovery, and document all required procedures and processes. Additionally, we must ensure that returning capabilities to RRB must also work (failback).

Deliverables and Objectives

To prepare for disaster recovery and rapid network failover by updating hardware and creating outage plans and recovery strategies for the short term, intermediate term, and long term. This will happen through careful, detailed planning and thorough testing. This includes:

Physical hardware installation and networking
Replication method requirements and design, including outline of RTO (time it will take to make the system active again based on priority level) and RPO (how much data loss we are willing to accept)
Failover testing outcomes within an isolated environment, at the application level, and a full network failover
Maintenance requirements and considerations for future work

Events and Exercises

AITS ran production services from the Disaster Recovery Environment in HAB during the weekend of 2/11 - 2/12/2023.

This included all production services running out of RRB, Production AITS Linux Servers, Production AITS Solaris Servers, Production AITS Windows, and all clients of AITS maintained production services running out of the RRB Datacenter.

Services experienced downtime on Saturday morning 2/11 when they were moved from RRB to HAB, and then again on Sunday when they were moved back to RRB. Highly available services such as SiteMinder authentication, Identity and Access Management (IAM) did remain online during the failovers. See list of all those services that were impacted.

Caffeine Break: AITS Rapid Failover and Recovery
Disaster Recovery Efforts Presentation

A presentation was held on December 2, 2022 introducing the ‘Rapid Failover and Recovery’ project to set up redundant systems and processes. AITS presented an overview for the IT community to learn more about the project and the plans for testing.

PDF of Presentation

View Video in Media Space

UIS and UIC may have trouble viewing in Media Space. Please contact us for access to the video.