= Documentation for the Rubrik Cloud Data Management (CDM) Backup and Restore Method

== Summary

The Rubrik CDM backup and restore method for ReaR allows Rubrik CDM to perform bare metal recovery of Linux systems that are supported by ReaR. It does this by including the installed Rubrik CDM RBS agent files in the ISO that is created by `rear mkrescue` via a pre-script in the fileset. The ISO is left in place under `/var/lib/rear/output/rear-<hostname>.iso` by default. During the fileset backup Rubrik will backup the main operating system files as well as the ReaR ISO file. 

Bare Metal Recovery is performed by first restoring the ReaR ISO file from Rubrik CDM to an alternate host. Next the host being restored is booted from the ISO via CD/DVD, USB, vSphere Datastore ISO, etc... Once booted running `rear recover` will prepare the host for restore and start the Rubrik CDM RBS agent. If the host has a new IP address the new RBS agent will need to be registered with the Rubrik cluster. Registration is not necessary if the recovery host is reusing the same IP address as the original. All of the files for the host are then recovered from Rubrik CDM to the recovery host's `/mnt/local` directory by the user. Once complete the user exit's ReaR and reboots the host. 

== Configuration

1. Install and configure ReaR in accordance with:
- Red Hat 
   * https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/ch-relax-and-recover_rear
- Ubuntu
   * http://manpages.ubuntu.com/manpages/disco/en/man8/rear.8.html
- SUSE
   * https://en.opensuse.org/SDB:Disaster_Recovery
   * https://documentation.suse.com/sle-ha/15-SP1/html/SLE-HA-all/cha-ha-rear.html
- Generic
   * https://github.com/rear/rear

   NOTE: Ignore any instructions to configure external storage like NFS, CIFS/SMB or ftp. Also ignore any instructions to configure a specific backup method. This will be taken care of in the next steps. 

   NOTE: Ignore any instructions to schedule ReaR to run via the host based scheduler (cron). Rubrik CDM will run ReaR via a pre-script in the fileset. If this is not preferred ReaR can be scheduled on the host, however, the ISOs created may not be in sync with the backups.

   NOTE: If installing the pre-release or development version for which there is no installer, copy the repo to the host being protected. Then run `make install` from its root directory of the repo. 

1. Install the Rubrik CDM RBS agent as directed by the Rubrik documentation.
1. Edit `/etc/rear/local.conf` and enter:

   OUTPUT=ISO
   BACKUP=CDM

1. Test `ReaR` by running `rear -v mkrescue`
1. Configure fileset backup of the host and add `/usr/sbin/rear mkrescue` as a prescript. 
1. ISOs will be saved as `/var/lib/rear/output/*.iso`

- Recovery 

1. Recover `/var/lib/rear/output/rear-<hostname>.iso` from host to be restored. 
1. Boot recovery machine using recovered ISO.
   
   NOTE: Recovered system will use the same networking as the original machine. Verify no IP conflicts will occur. 

   NOTE: If the same static IP address may be used it will need to be changed if the original machine is still running.

1. Verify Firewall is down on recovery host.
1. Run `rear recover`
1. Answer inline questions until `rear>` prompt appears.
1. Run `ps -eaf` and verify that `backup_agent_main` and `bootstrap_agent_main` are running.
1. Get the IP address of the system using `ip addr`
1. Register the new IP with the Rubrik appliance (if needed)
1. Perform a re-directed export of `/` to `/mnt/local`
1. Reboot
1. Recover other file systems as needed.

   Note: that the Rubrik RBS agent will connect as the original machine now. The host may need to be reinstalled and re-registered if the original machine is still running. 

== Known Issues

* Recovery via IPv6 is not yet supported.
* Automatic recovery from replica CDM cluster is not supported
* CDM may take some time to recognize that the IP address has moved from one system to another. When restoring using the same IP give CDM up to 10 minutes to recognize that the agent is running on another machine. This usually comes up during testing when the original machine is shutdown but not being restored to. 
* Recovery from a replica CDM cluster is only supported with CDM v4.2.1 and higher.
* Care must be taken with SUSE systems on DHCP. They tend to request the same IP as the original host. If this is not the desired behavior the system will have to be adjusted after booting from the ReaR ISO.  
* If multiple restores are performed using the same temporary IP, the temporary IP must first be deleted from Servers & Apps -> Linux and Unix Servers and re-added upon each reuse.
* ReaR's `ldd` check of other binaries or libraries may result in libraries not being found. This can generally be fixed by adding the path to those libraries to the `LD_LIBRARY_PATH` variable in `/etc/rear/local.conf`. Do this by adding the following line in `/etc/rear/local.conf`:
+
  export LD_LIBRARY_PATH-"$LD_LIBRARY_PATH:<path>"
+
To make CentoOS v7.7 work the following line was needed:
+
  export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export"
+
To make CentOS v8.0 work the following line was needed:
+
  export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64/bind9-export:/usr/lib64/eog:/usr/lib64/python3.6/site-packages:/usr/lib64/samba:/usr/lib64/firefox"

== Troubleshooting

* Verify that ReaR will recover your system without using the CDM backup and restore method. Most errors are due to configuration with ReaR itself and not Rubrik CDM. Use the default ReaR backup and restore method to test with. 
* Follow the OS specific configuration guides as mentioned at the beginning of this document. 

== Test Matrix

.Test Matrix
[%header,format=csv]
|===
Operating System,DHCP,Static IP,Virtual,Physical,LVM Root Disk,Plain Root Disk,EXT3,EXT4,XFS,BTRFS,Original Cluster,Replication Cluster
CentOS 7.3,,pass,Pass,,Pass,,,,Pass,,Pass,
CentOS 7.6,Pass,,Pass,,Pass,,,,Pass,,Pass,
CentOS 7.7,Pass,,Pass,Pass,Pass,,,,Pass,,Pass,
CentOS 8.0,Pass,,Pass,,Pass,,,,Pass,,Pass,
CentOS 5.11,,,,,,,,,,,,
CentOS 6.10,,,,,,,,,,,,
RHEL 7.6,Pass,,Pass,,Pass,,,,,,,
RHEL 7.4,,,,,,,,,,,,
RHEL 6.10,,,,,,,,,,,,
SUSE 11 SP4,,,,,,,,,,,,
SUSE 12 SP4,Pass (uses same IP as original),,Pass,,,,,,,Pass,Pass,
Ubuntu 14.04 LTS,,,,,,,,,,,,
Ubuntu 16.04 LTS,Pass,,,,Pass,,,Pass,,,Pass,
Ubuntu 17.04 LTS,,,,,,,,,,,,
|===

* Empty cells indicate that no tests were run.
