This Wiki is brought to you by Backup Central, where you can find the Mr. Backup Blog, Forums, and a mailing list for each forum!
Backup FAQs	Service Providers	Backup Software	Backup Hardware	Backup Book Wiki	Free Stuff	Miscellaneous

Some chapters from Unix Backup & Recovery did not make it into Backup and Recovery. We have added these original chapters to the Wiki in hopes that some of you will help us maintain it.

Rational ClearCase is a software configuration management tool developed and marketed by IBM. ClearCase is most commonly used in software projects involving at least several developers but often in environments with hundreds of developers in many groups. Like many other chapters in this book, this chapter is here due to ClearCase’s popularity. Since I’ve been asked hundreds of times about how to back up and recover ClearCase, I thought it would make a great addition to this book.
The following description can be found on the ClearCase web site (http://www-306.ibm.com/software/awdtools/clearcase/index.html):

ClearCase provides comprehensive configuration management, including version con­trol, workspace management, build management, and process control—without forcing you to change your existing environment, your tools, or the way you work.

As with almost any DBMS, ClearCase must be quiescent before backups can be per­formed. Hence, the general strategy is to lock or make unavailable the items(s) being backed up, do the backup, and then reactivate the ClearCase access to those items. This may seem like a simplistic approach, but it is the most reliable. In the following sections, you will see this paradigm used extensively.

This chapter was written by Bob Fulwiler, who lives outside Seattle, Washington, and spe­cializes in Unix system administration and software configuration management.

Contents 1 ClearCase Architecture 1.1 VOBs and Views 1.2 Registry 1.3 License Server 1.4 Useful Terms 1.5 Registry Backup and Recovery Procedures 1.5.1 Registry backups 1.5.2 Registry recovery 2 VOB Backup and Recovery Procedures 2.1 VOB Backup Strategies 2.1.1 Standard strategy 2.1.2 Snapshot strategy 2.1.3 Other strategies 2.1.3.1 Disk backup. 2.1.3.2 Mirrors. 2.2 Other VOB Backup Issues 2.2.1 Remote storage pools 2.2.2 Incremental backup 2.2.3 MultiSite backup 2.3 VOB Recovery 2.3.1 General approach 2.3.2 Restore using the “standard” or 'mirror backup scenarios 2.3.3 Restore using the “'snapshot” backup scenario 2.3.4 MultiSite recovery 2.3.5 Conclusion 3 View Backup and Recovery Procedures 3.1 View Backup Strategies 3.2 View Recovery 4 Summary

ClearCase Architecture

I’ll begin with a discussion of ClearCase architecture, since many of the terms that are used to describe ClearCase may be unfamiliar.

VOBs and Views

ClearCase has a “multilevel” architecture:

• The first, or base, layer is the Unix filesystem with all the usual components and restric­tions.
  
• The second layer is Rational’s proprietary Multi-Version File System (MVFS); it is this layer which acts very much like a DBMS and allows the multiversion functionality of ClearCase.
  
• The third layer is the Versioned Object Base (VOB); the VOB is the basic data container for items under control of ClearCase. A ClearCase installation typically has several VOBs at any given time. Each VOB is owned and used by a particular group or groups in order to do their work.
  

The fourth, and last, layer is a ClearCase view. Just as a database view selectively accesses particular database items, so does a ClearCase view determine which items a worker can access at any given time. Views also have their own data storage con­tainers that hold temporary items such as program files that are currently being modi­fied. The mechanics of how these containers are accessed is not germane to this dis­cussion, but if you’re interested, you can find more information on the Rational ClearCase web site.

A view is similar to a VOB in that it has its own storage directory, but is accessed through a view tag. A VOB tag is a full pathname because it is a mount point. How­ever, a view tag is a simple name because it is accessed as part of a directory path­name. Like a VOB, a view must be explicitly activated; usually with the setview command. Views are discussed in greater detail later.

A distinction needs to be made between what ClearCase considers public and private; these concepts can apply to both VOBs and views. Public items are essentially commu­nity property to which access is controlled by the owning group, but which several peo­ple may be accessing (of course, not the same item) simultaneously. Anyone with the appropriate group permissions may access the VOBs/views in a manner very similar to the use of Unix filesystem ACLs. Private VOBs/views typically are owned and adminis­tered by one individual and are usually for her personal use in development or testing. An owner may allow access to others, but private items usually are not considered “public” property. For this reason, many sites do not back up private VOBs/views as they do pub­lic ones but instead leave that activity to the VOB/view owner. However, I show how to back up private as well as public items.

Registry

The ClearCase Registry is the “glue” that binds all of the ClearCase items and functions together. It contains pointers and records to each ClearCase item, including characteris­tics of the item, modifications, ownership, and permissions. Each time a ClearCase action is taken on a ClearCase object, the Registry is updated to reflect that action. As a result, when backups are done on VOBs and/or views, the Registry also should be backed up. It also should be restored at the same time as a VOB/view restore in order to keep all the parts of a ClearCase environment synchronized.

License Server

The ClearCase license server, like any license server, controls how many users may access ClearCase at any given time. While it usually isn’t necessary to back up the license file as frequently as VOBs/views, I would suggest that, at a minimum, you keep a copy of the license file outside the ClearCase environment or print a hardcopy in the few instances in which a manual restore or fix is required.

Useful Terms

These parts of ClearCase should be considered when implementing backups:

Versioned Object Base ('VOB)
A repository that stores versions of file elements, directory elements, derived objects, and metadata associated with these objects. With MultiSite (see “MultiSite backup,” later in this chapter), a VOB can be replicated at multiple sites.

VOB server
A machine that physically stores the VOB files and runs a process called vob_server that clients communicate with in order to access VOB data.

Registry server
A registry server is the host on which all ClearCase data storage areas (all VOBs and views) in a local area network are centrally registered. ClearCase supports a primary and backup registry server host. The backup host is used if the primary host becomes unavailable.

License server
A license server is a host whose albd_server process controls access to the licenses defined in its license database file. Any ClearCase client must be able to communi­cate with a license server in order to access any ClearCase-related data.

View
A view is a ClearCase component that provides a work area for one or more users— to edit source versions, compile them into object modules, format them into docu­ments, and so on. Users in different views can work with the same files without interfering with one another. For each element in a VOB, a view’s configuration specification selects one version from the element’s version tree. Each view also can store view-private files and view-private directories, which do not appear in other views.

MVFS
An MVFS is a directory tree that, when activated (mounted as a filesystem of type MVFS), implements a ClearCase VOB. To standard operating system commands, a VOB appears to contain a directory hierarchy. ClearCase commands also can access the VOB’s metadata. Also, “MVFS filesystem” refers to a filesystem extension to the operating system, which provides access to VOB data.

MultiSite replica
An instance of a VOB, located at a particular site; refers to the VOB’s database and all of the VOB’s data containers.

View-private file
A file that exists only in a particular view. A private file is not version controlled, except insofar as it is separate from private files in other views.

Registry Backup and Recovery Procedures

All VOBs and views are “registered” in the ClearCase storage Registry. Someone first setting up ClearCase designates a registry server host. The Registry itself is contained in the directory /usr/adm/atria/rgy (on some platforms, /var/adm/atria/rgy). This directory contains at least six files: vob_tag, vob_object, view_tag, view_object, regions, and site_ config. In order to adequately back up the Registry, all six files must be included in the backup. The albd_server process services the Registry.

Registry backups

The simplest way to ensure good registry backups is to designate one or more machines as backup registry server host(s). A backup registry host takes periodic snapshots of the primary registry host’s registry files (see the rgy_backup command in the ClearCase manuals) and client list and stores these snapshot files in the directory /usr/adm/atria/rgy/ backup (on some platforms, /var/adm/atria/rgy/backup). In the event of primary registry server failure, an administrator can run rgy_switchover to activate a backup registry server and reset all client hosts accordingly.

Here is the procedure for making a backup registry host after ClearCase is installed (for ClearCase Versions 3.0–3.2). On the primary registry host:

• Edit the rgy_hosts.conf file to identify the primary and backup registry servers. The first line of this file should contain the name of the primary registry host; the second line should have the name(s) of one or more hosts to be backup registry servers, that is:

primary_registry_host
first_backup_host,second_backup_host

• Copy the ~gy.hosts.conf file to the /usr/adm/atria/rgy directory on every client host (including the backup registry host(s)).
• Usually, the rgy_backup program runs automatically once a day on each backup registry host. However, if you feel the need to do a manual backup, log on to the backup reg­istry host as “root” and execute the command $CCHOME'/bin/rgy_backup.

If you are using one host for all ClearCase server functions, the Registry still should have periodic backups. This is done by shutting down the albd_server, backing up the Regis­try directory, and then restarting the albd_server. Since these are just flat files, any avail­able backup method may be used for this backup. The script in Example 17‑1 demon­strates the steps to perform a flat-file backup.

Example 17‑2: Registry Backup Script: regback.sh (continued)

#!/bin/sh
#

# This script should be run as "root"

#######################################################
##Site-specific section (change as appopriate)

PATH=/usr/bin:/usr/sbin:/usr/ucb:$CC
CC=$CC/cleartool # ClearCase executable
# Uncomment one of the following to set the media to be used to
# hold the registry backup. Be sure to make the same change
# on the restore script.
# dev="/dev/rmt/0" # default tape drive
# dev="/usr/adm/backup" # disk

# Step 1: Shut down the albd_server

$CC/atria_start stop

# Step 2: Back up the registry
#
# Insert your usual backup method here.
#
# If you do not have commercial or homegrown backup facilities,
# please see the Native Backup Utilities: Unix chapter for
# suggestions and sample scipts.
# Example command to back up the registry to a local
# 8-mm tape drive on a Solaris system (assuming that the 8 mm is
# the default drive):

# dump 0fuv /dev/rmt/0 /usr/adm/atria/rgy

# Step 3: Restart the aldb_server

$CC/atria_start start

#####################################################
echo "ClearCase Registry Backup complete."
#####################################################

Registry recovery

The restore process follows the same sequence of events: shut down the albd_server, restore the files, and restart the albd_server. The actual restore can be done using the same tools that were used to create the backup.

• On the secondary server (first_backup_host), execute the following command:

$ rgy_switchover -backup "second_backup_host primary_registry_host" primary_ registry_host first_backup_host

• For any client that was down or for some reason unreachable by the network (record their name in the previous step), do the following after the client is available once again:
• Log in to the client.
  
• Stop ClearCase (shut down the albd_server) by executing the command $CCHOME/ etc/atria_start stop.
  
• Change the rgy_hosts.conf so that the first line is first_backup_host.
  
• Restart ClearCase (start the albd_server process) by executing the command $CCHOME/etc/atria_start start.
  

The script in Example 17‑3 demonstrates the steps required to perform this type of resto­ration.

Example 17‑4: Registry Restore Script: regrest.sh (continued)

#!/bin/sh
#
# This script should be run as "root"

#######################################################
##Site-specific section (change as appopriate)

PATH=/usr/bin:/usr/sbin:/usr/ucb:$CC
CC=$CC/cleartool # ClearCase executable
# Uncomment one of the following to set the media to be used to
# hold the registry backup. Be sure to make the same change
# on the restore script.
# dev="/dev/rmt/0" # default tape drive
# dev="/usr/adm/backup" # disk

# Step 1: Shut down the albd_server

$CC/atria_start stop

# Step 2: Restore the registry
#
# Insert your usual restore method here.
#
# If you do not have commercial or homegrown restore facilities,
# please see the Native Backup Utilities: Unix chapter for
# suggestions and sample scipts.
# Example command to restore the registry from a local
# 8-mm tape drive on a Solaris sytem:

# ufsrestore rxfv /dev/rmt/0 /usr/adm/atria/rgy

# Step 3: Restart the aldb_server

$CC/atria_start start

##################################################
echo "ClearCase Registry Restoration complete."
##################################################

When the primary server host is once again available, follow the same procedure to make it the actual primary registry server host again. Make sure that the two servers are syn­chronized by backing up the original primary before switching back to the original con­figuration.

VOB Backup and Recovery Procedures

A VOB (Versioned Object Base) is a data repository for a directory tree. A VOB is implemented as a standard directory tree, whose top-level directory is termed the VOB storage directory. The directory contains files and subdirectories. The VOB storage area is recognizable by its vbs suffix.
The VOB must be activated on the host by mounting it as a filesystem of type MVFS (ClearCase multiversion filesystem type). See the ClearCase mount reference page for more information. As mentioned earlier, most sites leave the backup/restore of private VOBs to the individual owner, but if the site wants to include private VOBs in their nor­mal backup, the following procedures can apply to private VOBs as well.

VOB Backup Strategies

The general VOB backup strategy is to follow a three-step process:

• Make the VOB quiescent.
• Back up the storage directory.
• Reactivate the VOB.

This section discusses the various ways in which this can be done.
Common strategies for doing a VOB backup include (but are not limited to): standard and snapshot (the two most common), disk backup, and the use of mirrors. Since the main thrust is the backing up of the physical storage directory, there are many ways this can be accomplished, including the use of commercial tools.

Standard strategy

The standard backup procedure is to “lock” the VOB, thereby ensuring that it is unavail­able to other processes during the backup. It is extremely important that the VOB remain locked during the backup process; otherwise, incomplete and unpredictable results will occur. It also should be noted that backup utilities that reset atime should be avoided, because it prevents other parts of ClearCase from functioning properly.
Example 17‑5 provides an example of this type of VOB backup script.

Example 17‑6: Standard VOB Backup Script: vob_standard.s (continued)

#!/bin/sh

# Export the correct path so that all the required binaries
#can be found

case $0 in
/* ) PATH=/usr/atria/bin:/bin:/usr/bin:`/bin/dirname $0`
c=`/bin/basename $0`
 ;;
* ) PATH=/usr/atria/bin:/bin:/usr/bin:/usr/sbin
c=$0
 ;;
esac
export PATH

if [ $# -ne 1 ] ; then
echo "
Usage: $0 <lock|unlock>
(Will lock or unlock all clearcase vobs on this server...)"
exit 1
fi

HOST=`hostname|awk -F. '{print $1}'`

#Get a list of all vobs on this server
VOBS=`cleartool lsvob -host $HOST|sed 's/*//' |awk '{print $1}'`

case $1 in

lock )

for VOB in $VOBS ; do
# (Lock VOBS)
cleartool lock -c 'VOB backups in progress' -vob $VOB \
> /tmp/voblock.log 2>&1
done

 ;;

unlock )

for VOB in $VOBS ; do
# (Unlock VOBS)
cleartool unlock -vob $VOB > /tmp/vobunlock.log 2>&1
done

 ;;

* )
echo "
Usage: $0 <lock|unlock>
(Will lock or unlock all clearcase vobs on this server...)"
exit 1
 ;;

esac

Snapshot strategy

The ClearCase term for this strategy is semilive. To enable database snapshots, execute the ClearCase command vob_snapshot_setup on a VOB (see the ClearCase Reference manual for details). This approach has costs and benefits that need to be considered:
This is the least hands-on strategy, as the system automatically takes snapshots of the current VOB contents.
This strategy has the benefit of reducing the amount of time a VOB needs to be locked.
Since the snapshot is stored on disk, it increases disk space usage.
The restore procedures are more complex, and if the restored pools are older than those currently in the VOB, some data may be lost.
Use the following procedure to configure VOBs to use the snapshot utility. For more detail, see the ClearCase Reference Manual. Use the command vob_snapshot lsvob to determine which VOBs are set up for snapshotting. If the VOB you wish to snapshot is not already set up, use the vob_snapshot_setup utility to configure it for snapshotting. Consider the following command:

$ vob_snapshot_setup modparam –snap_to /snapshots/src /vobs/src

This would cause the VOB /vobs/src to be snapshot to /snapshots/src whenever the vob_ snapshot utility is run (typically from cron, but not necessarily right before backups). In order to execute the snapshot utility on a regular basis, put an entry into the root crontab:

0 22 * * * /bin/sh /usr/atria/config/cron/vob_snapshot.sh

It is important that you back up both the snapshot directory and the VOB storage direc­tory.
See Example 17‑7 for this type of VOB backup script.

Example 17‑8: Snapshot VOB Backup Script: vob_snap.sh (continued)

#!/bin/sh

case $0 in
/* ) PATH=/bin:/usr/bin:/usr/sbin:/sbin:`/bin/dirname $0`
 ;;
* ) PATH=/bin:/usr/bin:/usr/sbin:/sbin
 ;;
esac

export PATH

Lockvobs()
{
/nsr/bin/vobs.sh lock
}

Unlockvobs()
{
/nsr/bin/vobs.sh unlock
}

X=$$
VXCONFIG=/etc/vxfs_lock.list
VFSTAB=/etc/vfstab

#vxconfig file is in the format:
#vxfs_device:snapsize:snapshot_device:snapshot_mount_point

Snap()
{

#Make a startup file that will clean this up if the system
#reboots in the middle of it

echo "#!/bin/sh
X=\$\$

for LINE in \`cat $VXCONFIG\`
do

VXFS_DEVICE=\`echo \$LINE|awk -F: '{print \$1}'\`
SNAPSHOT_DEVICE=\`echo \$LINE|awk -F: '{print \$3}'\`
SNAPSHOT_RAW_DEVICE=\`echo \$SNAPSHOT_DEVICE\
|sed 's-/dsk/-/rdsk/-'\`
SNAPSHOT_MOUNT_POINT=\`echo \$LINE|awk -F: '{print \$4}'\`

grep -v \"\$SNAPSHOT_DEVICE \$SNAPSHOT_RAW_DEVICE \$SNAPSHOT_MOUNT_POINT\" $VFSTAB |sed 's/^#SNAPPED#//' > /tmp/\$X.vfstab.tmp
mv /tmp/\$X.vfstab.tmp /tmp/\$X.vfstab
done

mv /tmp/\$X.vfstab $VFSTAB

rm /etc/rc0.d/S10cleanupvxfs "\
>/etc/rc0.d/S10cleanupvxfs

cp $VFSTAB /tmp/$X.vfstab

cat $VXCONFIG|while read LINE ; do

VXFS_DEVICE=`echo $LINE|awk -F: '{print $1}'`
SNAPSIZE=`echo $LINE|awk -F: '{print $2}'`
SNAPSHOT_DEVICE=`echo $LINE|awk -F: '{print $3}'`
SNAPSHOT_RAW_DEVICE=`echo $SNAPSHOT_DEVICE|sed 's-/dsk/-/rdsk/-'`
SNAPSHOT_MOUNT_POINT=`echo $LINE|awk -F: '{print $4}'`

[ -d $SNAPSHOT_MOUNT_POINT ] || mkdir -p $SNAPSHOT_MOUNT_POINT

mount -F vxfs -o snapof=$VXFS_DEVICE,snapsize=$SNAPSIZE \
$SNAPSHOT_DEVICE $SNAPSHOT_MOUNT_POINT

sed "s-.*${VXFS_DEVICE}.*-#SNAPPED#&-" /tmp/$X.vfstab \
>/tmp/$X.vfstab.tmp

echo "$SNAPSHOT_DEVICE $SNAPSHOT_RAW_DEVICE
$SNAPSHOT_MOUNT_POINT vxfs 9 yes -" >> /tmp/$X.vfstab.tmp

mv /tmp/$X.vfstab.tmp /tmp/$X.vfstab

done

cp $VFSTAB $VFSTAB.sav.1
mv /tmp/$X.vfstab $VFSTAB

}

Unsnap()
{

cp $VFSTAB /tmp/$X.vfstab

cat $VXCONFIG|while read LINE ; do

SNAPSHOT_DEVICE=`echo $LINE|awk -F: '{print $3}'`
SNAPSHOT_RAW_DEVICE=`echo $SNAPSHOT_DEVICE|sed 's-/dsk/-/rdsk/-'`
SNAPSHOT_MOUNT_POINT=`echo $LINE|awk -F: '{print $4}'`

umount $SNAPSHOT_MOUNT_POINT
grep -v "$SNAPSHOT_DEVICE \
$SNAPSHOT_RAW_DEVICE$SNAPSHOT_MOUNT_POINT" \
$VFSTAB |sed 's/^#SNAPPED#//' > /tmp/$X.vfstab.tmp
mv /tmp/$X.vfstab.tmp /tmp/$X.vfstab

done

cp $VFSTAB $VFSTAB.sav.2
mv /tmp/$X.vfstab $VFSTAB

rm /etc/rc0.d/S10cleanupvxfs

}

case `basename $0` in
snap.sh )
Lockvobs
Snap
Unlockvobs
 ;;

unsnap.sh )

#####Take these three lines OUT if you use 5.1 & savepnpc
#####instead of cron!!
while [ `ps -ef|grep -c save` -gt 1 ] ; do
sleep 900
done
#####Take these three lines OUT if you use 5.1 & savepnpc
#####instead of cron!!

Unsna p
 ;;
esac

Other strategies

The preceding two strategies do not represent an exhaustive list of possibilities but are the two most common methods. Other methods include disk backup, mirrors, and many other possibilities; the number is limited only by the reader’s imagination. The main thing to remember, no matter how you do the backup, is the lock-backup-unlock sequence.

Disk backup.

If you have enough free disk space, you could just back up the VOB(s) to disk. The VOB would need to be locked only during the time the disk backup was being accomplished. At that point, you could unlock the VOB and back up the copy to tape or whatever media you choose.

Mirrors.

Many sites use mirroring as a fault-tolerance mechanism. This methodology also can be used to back up/restore VOBs. There are too many different mirroring tools to discuss them in depth here. However, the use of good mirroring procedures along with good backup strategies can be combined to perform the required functions. One of the advan­tages of mirroring is that the VOB needs to be locked for only as long as it takes to break the mirror and again for only as long as it takes to reestablish the mirror. The following generic procedure is fairly simple in concept but may be more problematic in implemen­tation:

• Lock the VOB.
• Break the mirror.
• Unlock the VOB.
• Back up the mirror using your normal backup procedures.
• Lock the VOB.
• Reestablish the mirror.
• Manually resynchronize the mirrors, if necessary.
• Unlock the VOB.

Other VOB Backup Issues

Once you have decided what method you will use to back up your VOBs, there are a few other issues that you will need to address.

Remote storage pools

Occasionally, you will come across VOBs that have one or more of their storage pools located remotely, i.e., not in the local VOB storage directory. These must be backed up at the same time as their owning VOB, using the same method, to ensure that all of the parts are synchronized. A good method of determining whether a VOB has any remote pools is to execute the command:

$ cleartool lspool -invob vob:vob_tag

This command lists all storage pools for the requested VOB. Any remote storage pools are annotated with a system-created comment similar to the following: remote cleartext storage pool for vob_tag VOB. This step should allow remote storage pools to be iden­tified and processed accordingly.

Incremental backup

Many people commonly use an incremental backup strategy as part of their general data and application backups. An incremental strategy also may be used to back up ClearCase VOB storage areas. However, this method has some noteworthy problems:

• When ClearCase updates its data (for example, when an element is checked in), it uses a delta format. At update time, ClearCase creates a new version of the delta file, stores it at a new location, changes the Registry to reflect the new location, and deletes the old delta file. Hence, incremental backups may capture several instances of the delta file because they have slightly different pathnames.
  
• In order to restore a VOB with incremental backups, it is necessary to restore all the dif­ferent versions as the incremental tapes are used. This may lead to disk space prob­lems if disk space is clear and will result in many different versions of the delta file being placed in the storage area even though only one is needed.
  
• The “extra” items can be eliminated by executing the checkvob command. Be aware that executing the checkvob command will result in many error/warning messages as it cleans up the unnecessary files.
  

Due to these problems, Rational recommends that only a “few” incremental backups be taken for any given VOB. However, given that restoration is easier when all the required data is in one place, it is recommended that all VOB backups be complete (level 0) in order to save on time and complexity when a restore is required.

MultiSite backup

According to Rational’s web site:
ClearCase MultiSite is a ClearCase product option that enables parallel development across geographically distributed project teams. Available on Windows NT and Unix, ClearCase MultiSite extends ClearCase’s reach to team members down the street or across the world, delivering automated, error-free replication of project databases and transparent access to all software elements.

The basic thrust is that one can have replicated VOBs/views anywhere on the network. For backups, the replica can be subject to backup separate from the primary VOB. The replica would be handled using the basic paradigm and be resynced after the backup is done. However, the “best” backup solution is to synchronize the replicas frequently. This may increase network traffic somewhat but has dividends in case restoration is needed (see “MultiSite recovery,” later in this chapter).
One of the more interesting notions is that one could have a MultiSite replica on the VOB server as well as in the usual VOB storage. This allows the replica to be offline for back­ups while the primary VOB server is still active for use. For this type of backup, all you would need to do is:

• Resync the replica (to make sure it matches the primary VOB).
• Lock the replica.
• Perform a backup using your preferred procedure.
• Unlock the replica.
• Resync to capture any changes made during the backup.

MultiSite has the advantage of being able to establish replicas nearly anywhere on the network where the basic requirements for MultiSite are met. If you have, or acquire, Mul­tiSite, this is a good solution for backups.

VOB Recovery

VOB recovery or restoration can get to be a complicated process. Rather than try to give all-inclusive directions or scripts here, we refer you to the ClearCase Administrator’s Manual (UNIX) for details. However, the following is offered as an example of VOB res­toration.

General approach

Just as VOB backup has a generic process, so does VOB restoration. The ClearCase restore_vob command is one method, but a “standard” process also can be described as follows:

1. Unmount the VOB. (This makes the VOB quiescent.)
2. Unregister the VOB and remove the vob_tag. (Removes storage from ClearCase control.)
3. Copy in the backup. (Recovers the desired data.)
4. Register the VOB and make a new VOB tag. (Puts storage under ClearCase control once again.)
5. Remount the VOB. (Makes the VOB available to users.)

Restore using the “standard” or 'mirror backup scenarios

In the first backup example above, I used a simple lock-backup-unlock methodology for performing a VOB backup. The following will restore a VOB when using that backup procedure. Moreover, since a backup of a mirrored set is basically the same type of backup as the “standard” backup, the same general paradigm may be used. The main dif­ference is the use of metadevices to perform the actual restore.

1. Unmount the VOB in order to make it both quiescent and unavailable to update:

$ cleartool umount vob_tag

1. Remove the VOB tag entry in the Registry and unregister the VOB using the physical storage area since the vob_tag has been removed:

$ cleartool rmtag -vob vob_tag
$ cleartool unregister -vob /export/vobstore/vob_name.vbs

1. Restore the physical VOB storage area. Use whichever restore command is relevant to the method of backup. If the VOB has remote storage pools, be sure to restore them as well.
2. Reregister the VOB and make a new vob_tag:

$ cleartool register -vob /export/vobstore/vob_name.vbs
$ cleartool mktag -vob -tag vob_tag [...] /export/vobstore/vob_name.vbs

1. Mount the VOB so that it is once again accessible:

$ cleartool mount [...] vob_tag

This is the basic scenario for VOB restores. Other options such as mirrors, MultiSite rep­licas, disk backups, or any other strategy will use either this paradigm or the vob_restore command illustrated in the next section.

Restore using the “'snapshot” backup scenario

If you need to recover a VOB that was “snapshot,” follow these steps:

1. Lock the VOB:

$ cleartool lock -c "recovering src vob" vob:/vobs/src

1. Restore the snapshot directory if necessary; it may still be on disk from the previous day. However, please note that $RESTORE refers to whichever method was used to do the physical backup. The primary notion is that the contents of /snapshots/src are available to the restore process.

$ $RESTORE /snapshots/src

1. Run vob_restore, and it will prompt you for each necessary step, including restoring the VOB storage from your backup media. Note that the vob_restore command will unregister the VOB.

$ vob_restore /vobs/src

1. Run checkvob to ensure that the database is not corrupt:

$ checkvob

1. Restart ClearCase on the VOB server:

$ $CCHOME/etc/atria_start stop; $CCHOME/etc/atria_start start

1. Reregister the VOB, because vob_restore unregisters it:

$ cleartool mktag -vob -tag /vobs/src /vobstore/src.vbs
$ cleartool register -vob /vobstore/src.vbs

MultiSite recovery

If you have been backing up replicated VOBs, restoration is fairly straightforward, albeit a bit complicated. If you lose the storage directory of a replicated VOB, most work may be recoverable. If the actions have been sent to replicas, they can be retrieved and used to bring the VOB up-to-date. However, please note that the restore VOB and all replicated VOBs must be consistent. If they are not consistent, irreparable damage can occur. As was mentioned in “Other VOB Backup Issues,” frequent synchronization between the local VOB and its replicas can be extremely helpful in restoration. Step 6 in the follow­ing list is where the benefits are clearest.
Here is the process of restoring a VOB using MultiSite (all activities must be performed as root):

1. Unmount the VOB on all ClearCase hosts.
2. Recover the VOB storage directory from backup.
3. Execute the command: multitool restorereplica -vob 'vob_tag.
4. Ensure that all previous MultiSite updates have been processed by the replica hosts.
5. Create new synchronization packets for all replica hosts.
6. Wait for all the replicas to send the local host synchronization/update packets.
7. Process the incoming packets with the command syncreplica -import.
8. Unlock the VOB.

Conclusion

The ability to back up and restore VOBs is as crucial to the administration of ClearCase as any software application data is to that application. As you can see, there are several ways to handle those tasks. It is beyond the scope of this book to determine which method(s) should be used at your particular site. However, ClearCase commands, in con­junction with standard Unix utilities, should make the process as painless as possible as long as you remember that, for changes outside ClearCase, the VOB needs to be quies­cent during the backup or recovery of the physical data containers within the VOB.

View Backup and Recovery Procedures

The ClearCase view is the developer’s window into the data stored in the VOBs. It is a short-term storage area for data created in the development process. A view holds both checked-out versions of files stored in the VOBs and view-private files that have no VOB counterpart. Like VOBs, views are implemented as a view storage directory. View stor­age directories can be recognized by their vws suffix. A developer activates a view by running the cleartool setview command.

The public versus private debate is intensified here because there will most likely be more private views than private VOBs. In addition, there is some debate on whether views need to be backed up at all. The crux of this argument is that, views being so vola­tile, it is not necessary to back up the contents, as they can always be derived from exist­ing resources. For private views, this is probably true. For public views that are used by multiple developers, it is usually wise to have a good backup, as some things may be hard to recreate, such as a build with a particular error that is being investigated.

View Backup Strategies

It is not a requirement that individual views be backed up in order to preserve the integ­rity of the ClearCase data. However, individual developer data may be contained in the views, and backing them up will preserve that data. Backing up views is, in most cases, simpler than backing up VOBs. Unlike a VOB, a view can have only a single storage pool, either local or remote. It is necessary to back up this storage pool with the main view storage directory.
The main problem, however, is that you cannot lock a view, so you are left with two basic approaches. First, pick a time of day (or night) when the traffic is low and warn developers not to access the view. Second, you can temporarily rename the view storage area during the backup.
The following procedure should work well for both public and private views:

1. Generate a list of views for backup.
2. Determine the location of the storage directory for a given view. This can be done using the lsview command, which will list the local and remote access paths. You can use the view server access path if the backup runs locally but will have to use the global path if the backup is being done over the network.

$ cleartool lsview -long view_tag

1. If you cannot get the developers to refrain from using the view at a given time, the fol­lowing procedure will make the view quiescent:

Stop the view server process:

$ cleartool endview -server view_tag

Rename the view storage directory; save the name for use at the end of the process.

$ mv /export/viewstore/view_tag.vws /export/viewstore/tmp-view_tag.vws

1. Determine whether the view storage directory is local or remote:

$ cd Global_path
$ ls -ld .s

1. If this is a link, then you have a remote storage pool.
2. Perform the actual backup. If the view has a remote storage pool, remember to back that up as well. You may need to use two different tapes, depending upon your network organization. The following command is for demonstration purposes only and should be replaced by your favorite backup command:

$ dump 0fuv /dev/rmt/0 view_storage_directory

1. If you renamed the view storage directory in Step 3, change the temporary directory back to its original name. You do not have to restart the view server process, as that will happen automatically when someone tries to use setview or a similar command:

$ mv /export/viewstore/tmp-view_tag.vws
/export/viewstore/view_tag.vws

Example 17‑9 presents a sample view backup script.

Example 17‑10: Simple View Backup Script: viewback.sh (continued)

#!/bin/sh

#################################################################
# This script will back up all views on its host. The script
# requires a parameter (s or m). "s" will do a simple backup,
# and "m" will do a backup where the view storage area is copied
# to a different location and the backup done on the new location.
#################################################################
# Export the correct path so that all the required binaries can
# be found

PATH=/opt/atria/bin:/bin:/usr/bin:/usr/sbin
export PATH

CC="/usr/bin/atria/cleartool"
TEMPDIR="/tmp/ccbkup"
LOGDIR="/var/adm/atria/logs"
LOGFILE=$LOGDIR/$DATE.data

date=`date '+%m/%d/%y%n'`
DATE=`echo $date | sed s:\/::g`
DEVICE="/dev/rmt/0n"

case $1 in
s) BUTYPE=999
 ;;
m) BUTYPE=0
 ;;
*) echo "Usage $0 [s|m]"
esac

HOST=`hostname|awk -F. '{print $1}'`

#Get a list of all vobs on this server
VIEWS=`cleartool lsview -host $HOST | sed 's/*//' \
| awk '{print $1}'`

# Loop thru the views
for view in $VIEWS; do
# Get the physical storage directory location
VWS_PATH=`cleartool lsview -l $VIEW_TAG | grep -i Global \
| awk -F": " '{ print $2 }'`
BUDIR=$VWS_PATH # reset the variable

if [ ! -d $TEMPDIR }; then
mkdir -p -m777 $TEMPDIR
fi
if [ $BUTYPE ]; then
# stop the view server
$CC endview $VIEW
#copy the view to the new area
cp $VWS_PATH $TEMPDIR
BUDIR=$TEMPDIR/$VIEW.vws
fi
# Back up the view
# dump 0fuv $DEVICE $BUDIR

# Write the log record
echo "$VWS_PATH" >> $LOGFILE
done
##########################################################
echo "View backups completed."
##########################################################

View Recovery

The procedure to restore a view is quite simple. The view can be restored to the same location as the original view or to a new location. Here is the process:

1. Stop the view server process:

$ cleartool endview -server view_tag

1. Rename the view storage directory so that it is moved aside:

$ mv /export/viewstore/xxx.vws \
/export/viewstore/xxx.vws.OLD

1. Restore the physical storage directory using a restore command relevant to the backup command used to create the backup. This example assumes that the dump command was used to create the backup. A gotcha at this point is remote storage. If the storage was in a remote directory, be sure to restore that directory as well.

$ restore rv /dev/rmt/0 /export/viewstore/xxx.vws

1. Restore the actual view by executing:

$ cleartool recoverview -tag view_tag

1. If you restored the view storage directory to a new location, it is necessary to reregister the view so that the new physical location is correct in the Registry. It also a good idea to remake the view tag to be sure everything is consistent:

$ cleartool register –view –replace \
/export/viewstore/xxx.vws
$ cleartool mktag –view –tag view_tag –replace /export/viewstore/xxx.vws

If you did not restore the view to a different location, the view should now be ready for use. It is not necessary to start the view server process, as the next attempt to access the view will start its server process. Example 17‑11 presents a view restoration script that is paired with the script in Example 17‑9 and uses common data.

Example 17‑12: Simple View Restoration Script: viewrest.sh (continued)

#!/bin/sh

#
#################################################################
# This script will restore a view storage area. It has two
# parameters: date (mmddyy format) and the view tag. It looks
# up in that day’s log file what was actually backed up (real or
# temp area). It then calculates the position of the backup file
# on a tape, restores the area, and if it isn't the original view
# storage directory, moves it there and cleans up the registry.
#################################################################
# Export the correct path so that all the required binaries can
# be found

PATH=/opt/atria/bin:/bin:/usr/bin:/usr/sbin
export PATH

DEVICE="/dev/rmt/0"
CC="/usr/bin/atria/cleartool"
LOGDIR="/var/adm/atria/logs"

VIEWTAG=$1
DATE=$2
LOGFILE=$LOGDIR/$DATE.data
if [ ! -f $LOGFILE ]; then
echo "Backup file entry for '$DATE' not found."
exit(1)
fi
LINE=`cat $LOGFILE | grep $VIEWTAG`
if [ $? ne 0 ]; then
echo "View tag backup reference not found."
exit (1)
fi

# OK, passed the initial edits, let’s get to it.
# Which file position on the tape?
FILECNT=`fgrep -n $VWS_PATH | awk -F: '{ print $1 }'`

# Get the full view storage path
VWS_PATH=`cleartool lsview -l $VIEW_TAG | grep -i Global \
| awk -F": " '{ print $2 }'`

# Get the file position on the tape
FILECNT=`fgrep -n $VWS_PATH $LOGFILE | awk -F: '{ print $1 }'`

# Stop the view server
$CC endview -server $VIEWTAG

#Move the old stuff aside
mv $VWS_PATH $VWS_PATH.OLD

# Position the tape
mt fsf $FILECNT

# Do the restore using your favorite procedure
restore rv $DEVICE $VWS_PATH

# Update registry info
$CC register -view -replace $VWS_PATH
$CC mktag -view -tag $VIEWTAG -replace $VWS_PATH
##########################################################
echo "View restore completed."
##########################################################

If you restored the view to a different physical location, the view and any VOBs with associated items must be resynchronized just as if you had moved the physical location of the view storage. The following steps will accomplish the “resynchronization”:

1. Generate a list of VOBs that may need to be synchronized.
2. Determine if an individual VOB has a reference to the view.
3. If so, do a checkout and then an uncheckout on some element to resync the VOB and view storages.

See Example 17‑13 and Example 17‑15 for a pair of scripts that will resync the VOBs. Example 17‑13 is a general VOB resync script and Example 17‑15 is a script that checks for VOBs that need to be resynced due to the existence of derived objects in the VOB that are linked to the view.

Example 17‑14: VOB Resynchronization Script #1 (continued)

#!/bin/sh

# This script resynchs the VOBs.
# This script should be run as "root" or "ccadm"

#######################################################
##Site-specific section (change as appopriate)

PATH=/usr/bin:/usr/sbin:/usr/ucb
CC=/usr/atria/bin/cleartool # ClearCase executable

VIEW_TAG=$1
# resynch the vobs
$CC setview -exec "./resynch.sh" $VIEW_TAG | tee /var/adm/atria/log/mvview. log

resynch.sh

Example 17‑16: VOB Resynchronization Script #2 (continued)

#!/bin/sh

# This script should be run as "root" or "ccadm"

#######################################################
##Site-specific section (change as appopriate)

PATH=/usr/bin:/usr/sbin:/usr/ucb
CC=/usr/atria/bin/cleartool # ClearCase executable

VIEW_TAG=`cleartool pwv | grep Set | awk -F": " '{ print $2 }'`
echo ""
# Inform user of what we are doing
echo "Checking for VOBs that need to be resynched due to existence
of do's in the vob linked to the view.\n\nPlease be patient as
this may take a while."

# Gen a list of vobs to be checked
COMPLETEVOBLIST=`cleartool lsvob | awk '{print $2 }'`

# Find and process effected vobs
for i in $COMPLETEVOBLIST
do
echo ".\c"
cleartool describe -l -vob $i 2>/dev/null | grep $VIEW_TAG > /dev/null
if [ "$?" = "0" ]; then
# get an element. This algorithm should return the first element
ELEMENT=`cleartool ls -l $i | awk -F/ '{ print $4 }' \
| awk -F@ '{ print $1 }' | sed 1q`

# Need full element path since we have not physically cd'ed
# to the vob directory
FULL_ELEMENT_PATH="$i/$ELEMENT"

# co + unco resynchs the vob to the new view storage

cleartool co -nc $FULL_ELEMENT_PATH
cleartool unco -rm $FULL_ELEMENT_PATH
echo "\n$i processed\n"
fi
done

Summary

Backup and recovery is an important part of ClearCase administration. While you can always devise these methods after ClearCase is operational, it is best to make these con­siderations part of the ClearCase architecture and planning process. Registry and view backup are fairly straightforward, but VOB backup and recovery can become quite com­plex. Create and dry run some backup/restore scenarios during the ClearCase implemen­tation planning phase. This might take a bit more time at the beginning but will bear many fruits after the product is implemented and developers actually are using the prod­uct.
View backups are appropriate for some environments in which files might not be checked in on a regular basis or a large amount of view-private data exists and needs some level of protection. The procedures for view backup and recovery are fairly simple and straightforward. However you choose to implement a backup/restore strategy, rest assured that it will be necessary at some time in the future. Take your time, plan well, and then rest easy knowing that loss of data will be minimal at worst, and nonexistent at best.