Category Archives: BRS/DR

February 19, 2015 · 1:18 pm

Testing Replicated Data on DataDomain appliances using Fastcopy

Many companies replicate data between sites. The specific methodology may be different; different vendors, different solutions, etc., but the end goal is basically the same….data redundancy to ensure as smooth a transition as possible should a DR event occur.

To look at one specific replication scenario, a number of my clients use Veeam to backup their virtual machines to backup repositories located on DataDomain appliances and then that data is replicated to a DataDomain appliance located at another site. Now I know, assuming I have green checks indicating my data is in sync in DD Enterprise Manager, that the data is replicated and if need be, usable in the secondary site. But still, it’s a good idea to test the data thus ensuring its integrity.

Looking at DataDomain articles, seems the first step in testing the replicated data is to break the replication context with the source system so that you can convert the replicated data from read-only to read-write. But I really don’t want to have to break replication connections in order to test the integrity of my data, so I called DD support who suggested I use fastcopy to make a read-write copy of the replicated data for DR/data integrity testing.

I asked, “What happens if I don’t have enough room to make a copy?” and was told that fastcopy uses “pointers”, “will not consume any additional space” and “the easiest method for r/w access for a DR test would be to fastcopy the replicated data.” Sounds too good to be true but I decided to test it as follows:

1. Created a new VM, a new backup repository on the DD, and then replicated the new DD MTree from the source to the destination DD. (Wasn’t going to try it out the first time with my production data)

2. After replication was successful, I created a new, blank MTree on the destination DD system. **If you do not create a blank MTree, the fastcopy command will fail as it is unable to create MTree’s on its own.

3. Execute the fastcopy command, an example of which is shown below:

filesys fastcopy source /data/col1/CAN-DR-Test destination /data/col1/VMTest
NOTE: The command is case sensitive, remember that when specifying your source and destination MTree

4. In DD Enterprise Manager, enable CIFS share for the replicated MTree (VMTest in this example)

5. Add VMTest as a backup repository in Veeam, importing existing backups, and then perform a restore.

In this instance, the process worked perfectly and thus ensured the client that their data was indeed replicating and accessible if needed.

Veeam: Replicating Backup Data and Failover with Data Domain appliances

Consider the following scenario. You purchase (2) Data Domain DD640 appliances, one to be deployed at your production site and the other at a DR or some other remote site. The DD 640s will also serve as the backup repository for Veeam Backup and Replication, which is being used to backup VMs hosted on VMware servers and the organization has a desire to replication the Veeam backup data from DD01 to DD02. DR and redundancy for the Veeam backup data can be accomplished by using Data Domain replication to replicate the \\DD01\Veeam CIFS data to \\DD02\Veeam-DR.

In this example, I’ll use Data Domain MTree replication to replicate data from DD01 to DD02 and show you how to access backup data should DD01 experience a failure.

1. Within DDEM, click Replication | Summary | Create Pair

2. On the Create Pair screen, select MTree as the Replication Type and then specify the Source Path, the Destination System and the Destination Path. Click OK.

3. DDEM begins to build the replication pair as shown below; it also begins the initial replication between the systems. Click Close.

4. The replication State will display a warning until replication completes. The Pre-Comp Remaining value should decline as the initial replication continues:

5. When the initial replication completes, check DD01 and DD02 and ensure the replication State reads Normal and the State Description reads Replicating

With the MTree replicated, the next step is to setup the CIFS share on DD02 using DDEM. (A CIFS share was already created on DD01)

6. In DDEM, select DD02 and then click Data Management | CIFS | Shares | Create

7. On the Create Share screen, enter a Share Name, Directory Path, and Comment then click OK.

DD02 Veeam Failover Procedure

In the event of a failure on DD01, failing over Veaam Backups to DD02 is a two-step procedure:

· Configuring \\DD02\Veeam-DR as a Veeam Backup Repository

· Re-mapping backup jobs to the new backup repository

The process to add \\DD02\Veeam-DR as a backup repository is almost identical to those of creating the first one on DD01. Open the Veeam Backup and Replication console to perform the following steps:

Configure \\DD02\Veeam-DR as a Veeam Backup Repository

1. Right-click Backup Repositories and select Add Backup Repository.

2. On the Name screen, enter a Name for the backup repository and click Next.

3. On the Type screen, select Shared folder as the backup repository type and click Next.

4. On the Share screen, enter \\DD02\Veeam-DR as the Shared folder, then specify credentials with read/write access to the share on the Data Domain. Under the Proxying server section, select Automatic selection and click Next.

5. On the Repository screen, set the Limit maximum concurrent jobs count to a number the backup repository/device can handle and click Next.

6. On the vPower NFS screen, select Enable vPower NFS server and select This server. For the Folder, accept the default and click Next.

7. On the Review screen, review the proposed settings and check Import existing backups automaticallyand Import guest file system index. Click Nextto create the new backup repository.

8. On the Apply screen, click Finish to complete the configuration of the backup repository. It could take a few minutes to create the repository since existing backup jobs are being imported:

Remapping Backup Jobs to DD02

To complete the failover to DD02, existing backup jobs must be remapped:

1. In the Veeam admin console, right-click an existing backup job and select Edit

2. Click Storage and then change the Backup Repository to DD02.

3. With DD02 selected as the backup repository, click Map backup.

4. On the Select Backup screen, map to the corresponding backup job on DD02 and click OK. When returned to the Edit Backup Job window, click Finish and repeat steps #1-4 for remaining backup jobs to complete the failover process.

RecoverPoint – Failed to configure splitter credentials

If you have RecoverPoint and use the SAN splitter for an EMC storage system, you may run into an issue if you change the password to the EMC storage account being used by RecoverPoint. After changing the password in Unisphere, you will likely see the following when attempting to update the account credentials inside the RecoverPoint GUI:

Failed to configure splitter credentials

As of 3/20/2013, this is a known issue. Here’s an email I received from EMC support:

This is actually a code bug that we are still trying to resolve. When a user changes the unisphere password or creates a new user, RP does not update it and still uses the original credentials from installation. We can see that the VNX is blocking commands from site control RPA. VNX engineering is investigating this currently.

Though its a known issue, I was able to do the following to get around the issue: Though I was using a “global” Unisphere account, I set the scope of my account to “local” and set the new password. To my amazement, the change took. I went back into the splitter properties and set the scope by to “global” and that change held and RP did not have any additional issues communicating with the SAN splitter.

Deleting old save sets from NetWorker

This is another in the series of “I’m posting this before I forget.” Had to delete some older NetWorker save sets in order to free some space on a DataDomain DD-160 and if I don’t put these down, I’m certain I’ll forget them and to avoid having to search again, here’s my steps: (using server1.ballfield.local as an example)

1. I didn’t want to remove every old backup job, but just those for a given server that were older than 3 months. To that end, I used MMINFO from the command-line of the NetWorker server to get a list of all backups that are 3 months or older and pipe the output to a text file that I’ll use to built a BAT file for deleting the save sets:

mminfo -avot -c server1.ballfield.local -q “savetime c:\temp\server1_ss_3MO.txt

This command will give you a text file similar to the following:


MMINFO Output

2. Next, I opened NetWorker to “spot check” a few of my SSIDs to verify that I was about to backup older save sets as, initially, I was somewhat confused by the savetime command switch. More about the savetime command switch can be found here.

NetWorker – Show Save Sets

Use SSIDs to verify age of the save set

3. After verification, I edited my text file as shown below and saved it as a BAT file:

Use NSRMM to detele old save sets

NSRMM is used to remove save sets. In this case, the command was:
nsrmm -dy -S SSID/CloneID
-d / delete
y / responds with “Yes”…if you do not specify the “y” switch, nsrmm will asked you for verification before each deletion
-S / specifies the SSID and CloneID (if there are any) to delete

4. In the command window, execute the BAT file. Once the BAT file is completed deleting the older save sets, execute the command nsrim -X which synchronizes the media database and completes the deletion of the save sets from NetWorker. NetWorker support advised that NSRIM should not be executed when backups are running.

5. Finally, I wanted to “clean” the space on the DataDomain manually as opposed to waiting for it to automatically do so on its schedule. I logged into the DD and on the Data Management | File System page, clicked Start Cleaning. In this case, after deleting my older save sets, I had 400GB of space I could recover.

DataDomain – Start Cleaning


DataDomain – Cleaning finished/more space available

As you can imagine, the backups function much better when there is free space to write to. In addition to removing the older save sets, you may need to adjust the NetWorker browse and retention settings (shown below) on your machines to avoid the same issue again.

NetWorker Client – Browse and Retention Policies

2 Comments

Filed under BRS/DR, Data Domain, EMC

January 19, 2013 · 12:30 am

Restoring a Domain Controller using Backup Exec 2012 SDR – Part 2

I ended part 1 with the creation of the SDR ISO. This ISO can be used with VMs (though you probably wouldn’t restore VMs with this) or burned to a CD for physical systems. Once the SDR disc is created, boot a machine with it in order to perform a system recovery.

Here’s more from the Backup Exec help topics concerning the SDR process:

You start a computer with the Simplified Disaster Recovery Disk, and then use the Recover This Computer link on the Recovery tab to recover the computer. The recovery disk uses a minimal version of the Windows 7 operating system and contains many of the drivers that are required to recover your computer. In many cases, the generic Simplified Disaster Recovery Disk can be used to completely recover the computer. However, if you change the computer’s hardware after the last SDR backup, the generic Simplified Disaster Recovery Disk might not be sufficient to recover the computer. You should run the Create Simplified Disaster Recovery Disk Wizard to determine if the generic Simplified Disaster Recovery Disk recognizes any new hardware drivers that are not present in its driver database. If the drivers are present, you can use the generic Simplified Disaster Recovery Disk to recover the computer. Otherwise, use the Create Simplified Disaster Recovery Disk Wizard to create a custom Simplified Disaster Recovery Disk for the computer, and then store the custom recovery disk in a safe location.

Again, I was using a VM to test the process of restoring a dead domain controller assuming an entire domain failure. In my case, my test VM:

Had the same OS as the original machine
Had the same service pack/patches
Had the same disk configuration
Had the same backup agent installed
Had a different name (DC02, but it could have been anything)
Was not a member of the domain

With that, I booted my new machine with my SDR Disc (after ensuring the CD-ROM was listed first in the boot order in the BIOS).

1. When prompted, accept the Symantec license agreement.

2. On the Welcome to the Simplified Disaster Recovery Disk window, click Recover This Computer

3. On the Where is this computer’s backup data located screen, select The data is located on devices attached to a remote Backup Exec server.

4. On the To which Backup Exec server do you want to connect screen, enter the Backup Exec server information and AD credentials then click Next. In my test scenario, I’m assuming the AD credentials worked because I had already logged into the Backup Exec server with this account. I’m guessing a different account would have failed…

5. On the Which backup sets to recover screen, select the Computer and Point in time of the backup to restore and click Next.

6. On the Do you want to use this volume layout for the specified backup sets screen, assuming your disk configuration is the same, you should be able to select your disk and click Next. However, I had a problem in that the wizard believe my new disk configuration was different, though it wasn’t (second screenshot). I ended up enabling the option to “Erase hard disks and recreate the volume layout shown above” to get past the disk layout problem.

Once the problem with the disk layout was “resolved”, I clicked Next…..

7. On the Recovery Summary screen, click Recover to begin the restoration.

8. The server is restored as shown below. When the recovery process completes, click Finish.

9. On the main recovery screen, click Exit and reboot the server.

10. When the server boot, test Active Directory.

This was just an easy test with a single domain controller. If multiple domain controllers existed in the environment, you could use the same process to restore them or you may decide to build new domain controllers in a virtual environment. If you will not restore other domain controllers, use NTDSUTIL to seize any FSMO roles and remove those domain controllers before you introduce new DCs into the domain.

Certainly there are many ways to restore servers….I’m not saying this is the only, or even the best way to do so, but I was impressed with how easy Backup Exec has made the recovery process. Typically I work with virtual servers so I may never run this process again but I believe its beneficial to know.

1 Comment

Filed under Active Directory, BRS/DR, Microsoft