cloud storage tier export, recovery and testing...2021/05/07 · 1.2 cloud export. cloud export...

CLOUD STORAGE TIER EXPORT, RECOVERY

AND TESTING

Lourie Goodall TS7700 Development and Architecture Tucson, AZ

A TS7700 cloud storage tier disaster recovery preparedness and recovery user’s guide

Document Version 1.1

May 7, 2021

IBM® TS7700 Virtualization Engine

Joe Swingler TS7700 Development and Architecture Tucson, AZ

Takahiro Tsuda TS7700 Development Tokyo, Japan

Kousei Kawamura TS7700 Development Tokyo, Japan

© Copyright International Business Machines Corporation 2021 1

This page intentionally left blank


Table of Contents

1 Introduction .....................................................................................................................................5

1.1 Cloud Storage Tier .............................................................................................................................. 5

1.2 Cloud Export ......................................................................................................................................... 6

1.3 Cloud Export Recovery ..................................................................................................................... 7

1.4 Cloud Export Recovery Test Mode ............................................................................................... 7

1.5 Logical Volume Retention in the Cloud ...................................................................................... 7

1.6 Logical Volume Version Restore ................................................................................................... 8

2 Example Use Cases – Why would I use Cloud Export and Recovery? ..............................8

2.1 Primary Disaster Recovery ............................................................................................................. 9

2.2 Secondary or Tertiary Disaster Recovery ................................................................................. 9

2.3 Safe Guarded Copy, Air Gap, Logical Volume Corruption Protection .............................. 9

2.4 Regulatory Hold ............................................................................................................................... 10

2.5 Pseudo Production Test ................................................................................................................ 10

2.6 Workload Transfer ......................................................................................................................... 10

3 Configuration Requirements ................................................................................................... 11

4 How to Perform a Cloud Export ............................................................................................... 12

4.1 Configuring Cloud Pools, Accounts & Containers ................................................................. 12

4.2 Where and When to Run Cloud Export .................................................................................... 15

4.3 LIBRARY EXPORT “Export Volume“ .......................................................................................... 17 4.3.1 Export List File .................................................................................................................................................................. 18 4.3.2 Reserved File ..................................................................................................................................................................... 20 4.3.3 Export Status File ............................................................................................................................................................. 20

4.4 Sample JCL for Cloud Export Request ....................................................................................... 21

4.5 Monitoring Cloud Export Operations ....................................................................................... 22

4.6 Host Completion Message ............................................................................................................. 24

4.7 Operator Interventions ................................................................................................................. 24

4.8 Export Status File ............................................................................................................................. 25

4.9 Sample JCL Read Export Status File........................................................................................... 31

4.10 Canceling Cloud Export Operations .......................................................................................... 31

5 Planning and Considerations for a Cloud Export Recovery ........................................... 32

5.1 Retaining Database Backup Information ................................................................................ 32

5.2 Life Cycle of Database Backups in the Cloud.......................................................................... 33

5.3 Setting up a TS7700 Recovery Cluster ..................................................................................... 35

6 Performing Cloud Export Recovery ........................................................................................ 37


6.1 Monitoring Cloud Export Recovery Operation ..................................................................... 39

6.2 Restoring the Host Environment ............................................................................................... 42

7 Cloud Export Recovery Test Mode .......................................................................................... 43

7.1 Configuring for Test Mode ............................................................................................................ 44

7.2 Create an Independent Cloud Container for Testing .......................................................... 46

7.3 Disaster Recovery Testing Cleanup .......................................................................................... 48

7.4 Cloud Export Recovery Test Best Practices ........................................................................... 49

8 Volume Version Retention by Cloud Pool ............................................................................. 49

8.1 Configuring Volume Version Retention ................................................................................... 50

8.2 Volume Version Retention Consideration and Best Practices ........................................ 51

9 Volume Version Restore ............................................................................................................ 56

9.1 Overview of Volume Version Restore ...................................................................................... 57

9.2 How to Perform a Volume Version Restore ........................................................................... 57

10 Appendix A ..................................................................................................................................... 62

11 Appendix B ..................................................................................................................................... 67

12 Appendix C ..................................................................................................................................... 69


Summary of Change

Version Change Date Description 1.0 09/28/2020 Initial version of this document 1.1 05/07/2021 Added description of volume version restore


1 Introduction This paper describes IBM TS7700 Cloud Export, Cloud Export Recovery, Cloud Recovery Testing, logical volume version retention and restore concepts for the TS7700 Cloud Storage Tier feature (Feature Code 5278). TS7700 Cloud Storage Tier was first introduced in microcode release 8.42.x.x and provides the capability to store TS7700 logical volume data on a variety of cloud storage devices. TS7700 Cloud Export and Recovery suite of features was introduced in microcode release 8.51.x.x and provides the capability to export a backup into an attached cloud. This backup can then later be used to restore an empty TS7700C cluster. This suite of features also includes the ability to perform disaster recovery testing to ensure disaster preparedness and also covers the concept of retaining and restoring older versions of logical volumes in the Cloud. This paper can be used as a guide to understand and configure an environment to perform these operations.

1.1 Cloud Storage Tier A TS7700 can be configured to have logical volumes exported to a cloud object store; that is, logical volume data can be copied to an object store behind the TS7700. This is a similar concept of copying logical volumes to physical tape. A logical volume can be pre-migrated (copied) or migrated (moved) to the object store based on TS7700 advance policy management using LRU algorithms or due to other events, such as meeting a space usage requirement. Just like physical tape tier support, logical volumes targeting a cloud are managed using TS7700 cache partitions. This feature provides a tier of cold storage behind the TS7700, which improves the cost of ownership and redundancy of data within the TS7700. Like physical tape libraries, a cloud object store is an extension of the TS7700 storage capabilities. Unlike a physical tape library, cloud provides the ability to access object data from any TS7700 cluster in a Grid directly from a cloud storage vault as long as that cluster has a physical network path and has been configured with credentials to access the vault. Figure 1-1 shows two TS7700 Cloud Storage Tier configured clusters (TS7700C) in the same Grid as clusters that are tape attached (TS7700T) and disk only (TS7700D):


Figure 1-1 TS7700 Grid with Cloud and Tape

Only the TS7700C clusters are capable of accessing cloud storage vaults. Those TS7700Cs must be configured with valid cloud object store account information, authentication credentials, and container information. The TS7700 keeps track of all objects it puts in a cloud vault and shares the information with all clusters in the Grid. For more information about the Cloud Storage Tier feature and supported cloud storage systems see the IBM TS7700 R5.0 Cloud Storage Tier Guide at http://www.redbooks.ibm.com/abstracts/redp5573.html.

1.2 Cloud Export Cloud Export refers to the function that will allow a host user to invoke a LIBRARY EXPORT command to initiate an export operation for one or more TS7700 cloud pools. Prior to R5.1, the IBM Z LIBRARY EXPORT command has been used exclusively for physical tape Copy Export operations. Tape Copy Export provides a way to write secondary copies of volumes of data onto physical tape cartridges that will be ejected from a tape library and held in a secure vault. These physical tapes provide a way to archive ‘cold’ data or to restore an entire production system in the event a disaster occurs. During this process, a copy of the TS7700 database is also created and stored on each physical tape. The database is a key component to recovering a TS7700. In R5.1, the LIBRARY EXPORT command added support for export to cloud.

http://www.redbooks.ibm.com/abstracts/redp5573.html


Cloud Export is a similar concept to Copy Export. When a LIBRARY EXPORT command is invoked for a Cloud Export, the TS7700 will ensure all logical volumes within the grid that are currently queued to be pre-migrated to the cloud, complete the pre-migration process. This is on a cloud pool basis. Once all queued logical volumes, for those cloud pools specified, are exported (pre-migrated) to the cloud, the TS7700 will then perform a database backup of its meta data and export it to all cloud pools specified by the user. This document will describe in greater detail how to invoke and monitor a Cloud Export operation.

1.3 Cloud Export Recovery Cloud storage provides a cost efficient solution to storing data long-term or as a means to store a copy of data in the cloud which can be used later as a restore point. In the event a cloud backup needs to be restored, a recovery process is needed to access that data. Cloud Export Recovery provides the ability to quickly restore a TS7700 environment. This document will contain information on how to perform a Cloud Export Recovery.

1.4 Cloud Export Recovery Test Mode Disasters are rare occurrences but, when they do happen, it is paramount to ensure all those involved in the recovery solution understand how to utilize the process and that the solution works as expected. For this reason, it is important to run periodic disaster recovery tests. In addition, a read only recovery may be required to inspect or selectively restore a previous point-in-time backup without compromising production. For example, when periodic backups are used for air gap recovery purposes. The TS7700 Cloud Export Recovery feature includes the ability to run in test mode. Cloud recovery testing creates a unique situation since the cloud that contains the data is the same one being utilized by production systems. This document will describe how TS7700 Cloud Export Recovery test mode allows for a read only restore while still allowing test and restored data to be created within an independent cloud vault.

1.5 Logical Volume Retention in the Cloud Beginning with microcode release level 8.51.0.x, the TS7700 will allow the capability to retain older versions of logical volumes in a cloud storage vault. This can be done by setting a Retention Days value that is greater than 0 when creating or modifying a cloud pool in the TS7700 Management Interface (MI) Cloud Pool page. When a logical volume has been auto-expired, ejected, reused, or modified, the version in the cloud prior to the event can be retained. The TS7700 will continue to retain the one or more prior versions of the logical volume in the cloud for the number of days specified in the cloud pool retention setting. This allows any one of these prior versions to be used as a restore point without needing to worry about normal or unexpected changes to data


preventing recovery. This is a critical step in maintaining a point-in-time recovery using a cloud vault since the vault is constantly being updated with new versions. Only through prior version retention can these backup points be reliably restored. The TS7700 keeps track of all previous versions of a VOLSER that existed within the configured Cloud Pool volume version retention period. Previous versions outside of the configured retention period are automatically deleted within the storage vault. This document will further explore use cases and best practices of using logical volume version retention in a TS7700C.

1.6 Logical Volume Version Restore In microcode release level 8.51.1.x, logical volume version restore function was introduced. With this function, users can restore older volume versions retained in the cloud and become able to access the contents of the versions. Contrary to cloud export recovery, volume version restore will not require special export operation to the cloud beforehand. Multiple volume versions premigrated to the cloud are retained for the duration of the preconfigured retention period. Users can choose any retained version of a logical volume as a restore source even in the case a newer version of the volume was corrupted or was maliciously or accidentally modified or in the case that an older version of the volume was maliciously or accidently deleted. Volume version restore is performed by remapping the version to be restored (restore source version) as the latest data of another scratch volume that a user specified as restore destination (restore destination volume). Once the restore process has completed, users can access the restored version data by mounting the restore destination volume. The simplest way to restore an old version would be to restore it into the same VOLSER. However, restoring an old version to the same VOLSER would violate the rule of expire hold or Logical WORM settings. Volume version restore function uses a scratch volume as restore source to solve these potential issues. During the volume version restore procedure, no actual data movement occurs. It completes by just modifying the TS7700 database. This feature would benefit users by eliminating the following operational costs that are required for any data movement. • Time to move volume version data from the cloud storage to TS7700. • Associated expense to read the version data out of the cloud storage. • Physical resources such as memory, CPU and disk utilization.

2 Example Use Cases – Why would I use Cloud Export and Recovery?

There are a few different reasons you may want to use Cloud Export and Recovery on the TS7700. This section provides a few high-level examples of use cases where Cloud Export


and Recovery can be used. There may exist other use cases not listed in this section. It’s best to work with your IBM technical representatives to see if Cloud Export and Restore can help with your requirements.

2.1 Primary Disaster Recovery

Many IBM Z customers rely on physical tape for disaster recovery (DR). Whether it’s creating physical tapes with older native tape controllers or Copy Exports with virtual tape solutions, business continuance is dependent on operations creating one or more physical tapes which are usually shipped off site to a secure location. In the event a recovery is needed, the physical tapes are brought to a recovery site owned by the client or a 3rd party and the environment is restored. Cloud Export and Recovery can provide a direct alternative to this recovery method. Instead of creating physical tapes, virtual volumes are copied to the cloud. Periodic Cloud Export operations can then take place to mimic the “shipping offsite” of tapes. In the event a recovery is needed, a recovery site can be used to restore the database backup, generated by a Cloud Export operation, into a TS7700C where the virtual tapes can be accessed. Recovery Point Objectives can be shorter and so can Recovery Time Objectives since there is no need to physically transport tapes.

2.2 Secondary or Tertiary Disaster Recovery You may have an existing virtual tape solution where replication is used for business continuance. You may also export physical tapes for an additional layer of protection. But, perhaps the distance between replicated sites is not ideal or regulatory requirements simply require additional copies. The additional redundancy may only be needed for a subset of your workload. Cloud Export and Recovery can be used in combination with your existing business continuance strategy. Essentially, another layer of redundancy that can be used in the event all existing methods are unavailable. In such an event, your operations can still be brought back online at a recovery site using a export in the cloud.

2.3 Safe Guarded Copy, Air Gap, Logical Volume Corruption Protection Every year, more and more IBM Z accounts are looking for an additional layer of protection for their data in the event of accidental or malicious actions. Technologies are in place that allow primary disk environments to take periodic snapshots which can be used to restore to a previous point in time. Such backups can often be copied to tape where they must be retained and protected. In addition, primary data such as ML2 often reside on tape and are equally in need of protection. WORM concepts and retention concepts can provide a great deal of protection, but a point-in-time copy outside of the virtual tape system provides an extra level of protection. Using Cloud Export, periodic point in time snapshots of the TS7700 can be saved within an on-premise or off-premise cloud. With Cloud Pool version retention policies, this data can


be retained for days to weeks to months or longer. In the event of an accidental or malicious action, you can restore a point-in-time copy of your environment that represents your TS7700 prior to the event. This environment can then be used to restore and recover those volumes which were corrupted or lost.

2.4 Regulatory Hold You may have an internal or external regulatory need to freeze one or more workloads in time. This can traditionally be accomplished by freezing return-to-scratch processing or making copies of such protected data. With Cloud Export and Cloud Pool version retention policies, you can achieve the same outcome.

First, one would setup all cloud pools which contain data needing to be held so that the pools' retention periods are set to a very large number (e.g. many years). Perform a DASD flash copy and retain it or also put it on tape with a TMS backup and IPL tape. New cloud pools can be defined for new workloads from that point forward. Then, a Cloud Export can be carried out against the held pools. From that point forward, logical volumes can be reused and off loaded to the new cloud pools. The previously run Cloud Export targets volumes that are in the previous pools and are in an indefinite version retention state. As new workloads create and scratch tapes, the version retention periods of the new pools can be set to value that is significantly less or disabled entirely(set to 0 days) preventing further capacity growth.

The database backup generated by the Cloud Export can then be used for a restore into a TS7700C and used to provide access to all data in the TS7700 at the time of the hold request. When the hold is lifted, the retention settings of the held pools can be dropped and the held data will automatically be released.

2.5 Pseudo Production Test Different events within an organization may create a need to test a pseudo production environment. For example, a new application is being introduced and testing it against a true representation of production is needed. Cloud Export, in combination with a primary disk snapshot, can be used to create a point-in-time snapshot of your production environment. A Cloud Export Recovery with a test mode setting, which will render all existing cloud pool data as “Read-Only”, can then be used to prevent the pseudo production LPAR(s) from manipulating production data in the cloud.

2.6 Workload Transfer There often occurs a need to transfer a workload from one grid to another. Most often, these transfers occur between independent clients. Interconnecting such clients through replication techniques may be viewed as insecure. Therefore, a safer, two-phase approach is needed. Cloud Export can be used to put a copy of the workload into a shared cloud.


Once contained inside the cloud, the receiving party can be given access to the cloud and use Cloud Export Recovery to restore the workload into a stand-alone TS7700C. Once contained in a TS7700C, it can be merged into an existing grid or simply used as a tape copy source to migrate the new workload into the target client grid.

3 Configuration Requirements The Cloud Export suite of features is supported on TS7760, TS7770 and newer models that have gone through Cloud Storage Tier feature (FC 5278) activation and the Cloud Enablement MES process. For more information regarding the TS7700 cloud enablement process see the following link: https://www.ibm.com/support/knowledgecenter/STFS69_4.2.0/ts7760_cloud_configuration.html The Cloud Export process can be performed on any TS7700C, whether it be a stand-alone cluster or a cluster which is part of a Grid. All TS7760s (VEC) and TS7770s (VED) clusters that exist in the grid must be at microcode release level 8.51.x.x or higher before a Cloud Export can be initiated. The TS7700C receiving the LIBRARY EXPORT command must have access to all cloud vaults specified in the DBPOOLS field in the Export List file. A TS7700C that does not have the correct cloud account credentials to write to a cloud vault will fail the EXPORT command. Cloud Export Recovery and Cloud Export Recovery Test Mode processes must be done on an empty stand-alone TS7700C that is configured as a Cluster 0. The stand-alone TS7700C must be at microcode release level 8.51.x.x or higher to perform these functions. If the production TS7700 environment is using an LDAP service authentication, so too must the stand-alone TS7700C recovery system. The recovery TS7700C should be fully configured and tested to ensure LDAP is working properly before beginning a Cloud Export Recovery. This includes ensuring a valid sign-on certificate is loaded. It is highly recommended to enable the “Allow an IBM service representative to connect through physical access” or “Allow an IBM service representative to connect remotely” checkboxes when configuring for LDAP on this recovery system. In the rare case that LDAP fails, this will enable the ability for IBM support to reset the system back to local authentication while the issue with LDAP communication is resolved. Without these settings enabled, the system will be completely inaccessible. No additional IBM Z host software updates are required for these new cloud-based export and restore features. New attention messages will be sent to the host during a Cloud Export and will be described in this document.

https://www.ibm.com/support/knowledgecenter/STFS69_4.2.0/ts7760_cloud_configuration.html

https://www.ibm.com/support/knowledgecenter/STFS69_4.2.0/ts7760_cloud_configuration.html


4 How to Perform a Cloud Export Cloud Export provides a way to ensure all logical volumes within a grid associated with the cloud pools specified have been previously pre-migrated or complete pre-migration to the cloud before a snapshot of the TS7700 database is taken. The database snapshot contains all the information needed to recover a TS7700 to that point in time. This section contains information on how to use Cloud Export.

4.1 Configuring Cloud Pools, Accounts & Containers Prior to performing a Cloud Export, the TS7700 must be configured to communicate with a cloud storage system. This configuration takes place in the TS7700 MI Cloud Tier Settings menu page found under Settings. For the most part, these settings are no different than if the Cloud Tier is used simply as a storage tier. The exception is if additional cloud pools for storing database backups are needed, then the Cloud Pool version retention definitions which must be defined prior to using Cloud Export. First, a cloud pool must be created. A cloud pool provides a way to segregate data. As of microcode release 8.51.x.x, multiple cloud pools are supported providing a maximum of 256 cloud pools. For any pool that will be included in a Cloud Export backup, be sure to assign a retention period that is long enough to support the oldest backup that may be restored plus any duration needed during the restore and test process. Last, in addition to the Cloud Pools which contain logical volume data, you may want to define one or more additional pools to retain an extra copy of the Cloud Export database backup. This way, redundant copies of the database backup can be made into two or more Cloud vaults. Cloud pools used exclusively for database backups, meaning they contain no logical volumes, do not need a retention period configured. The database backups are automatically retained long enough to accommodate the longest retention period defined by any pool included in the export request. Changes made to a Cloud Pool retention period are retro-active. This means that if a pool retention period is modified, all previous logical volume versions in the modified cloud pool which remain retained will honor the new retention period. For example, if a recovery is in progress and is elongated, the retention period can be extended within the source grid to prevent data from expiring while the recovery is still accessing the retained content. Or, if a reason arises where a backup must be recoverable indefinitely, an extremely large retention period can be defined to make sure the backup remains recoverable indefinitely. The following Figure 4-1 shows an example of the TS7700 MI Cloud Pool page:


Figure 4-1 Cloud Tier Settings Cloud Pools

Logical volumes can be assigned to a cloud pool through policy management and specifically by selecting a Storage Group construct with a Cloud Premigration Rank 1 pointing to a specific cloud pool (Figure 4-2):

Figure 4-2 Storage Group Constructs

To manage when and how logical volumes will be moved to the cloud vault, a Storage Class construct must also be created that targets a logical cache partition for cloud storage. Here, cloud storage preferences are set to determine how quickly a volume will be migrated to the cloud vault (Figure 4-3):

Figure 4-3 Storage Class Constructs


Once a cloud pool is created, an account needs to be created using the Cloud Accounts page. Enter the cloud type, the access key ID and the secret access key that were generated by the cloud provider when the container was set up in the cloud vault (Figure 4-4):

Figure 4-4 Cloud Tier Settings Cloud Accounts

Next, set up the container information by entering the Container Name (the name of the container set up in the cloud vault) and selecting the name of the cloud account to be used to access the container as well as the cloud pool to associate the container with in the Containers page of the TS7700 MI. Once the container is created, enter the cloud URLs to provide an access path to the container. Then, select which clusters to associate with the URLs (Figure 4-5):

Figure 4-5 Cloud Tier Settings Container

Last, if not already configured, IBM Z workloads that should target the one or more cloud pools need to be configured to use the previously mentioned constructs. This may require workloads use unique constructs if not all workloads are required to be copied to the previous defined cloud pools. The cloud storage container is now configured and is ready to use. Logical volume data can now be exported to the cloud through the pre-migration process.


In a grid configuration, additional steps may be required when two or more peers connect using different credentials or vault names. For more in-depth information about creating cloud pools, cloud accounts, cloud containers and using policy management for those pools, review the IBM TS7700 R5.0 Cloud Storage Tier Guide at http://www.redbooks.ibm.com/abstracts/redp5573.html.

4.2 Where and When to Run Cloud Export When to run the Cloud Export routine will depend on users’ needs. Running Cloud Export can be done at scheduled intervals or after a batch job ends. Only one LIBRARY EXPORT operation can be run on a TS7700 grid at a time. This means that if both TS7700C and TS7700T clusters exist in a grid and both Cloud Export and Copy Export operations need to be performed, they must be performed serially. A Cloud Export operation can be initiated through any virtual tape drive in the TS7700 Grid; it does not need to be initiated on a virtual drive of the TS7700 that is to perform the cloud export operation. The operation will be internally routed to the TS7700 that has the valid copy of the Export List file volume and the database backup will be performed on that cluster. The target TS7700C that receives the LIBRARY EXPORT command must have access to the cloud vaults that will receive a copy of the backup. A best practice is to perform the cloud export operation on the TS7700C that has the most cloud migration activity to the cloud pool(s) specified. For instance, if each TS7700C in a grid only has access to its own cloud vault, then two separate Cloud Exports will need to be performed if a copy of the database backup is to be placed in each cloud vault. Figure 4-6 shows cluster 1 (C1) is configured to access the cloud vault specified as CLDPOOL1 and cluster 2 (C1) is configured to access the cloud vault specified as CLDPOOL2. C1 has no credentials configured to access Vault2 and C2 has no credentials configured to access Vault1, so a single Cloud Export command cannot be performed to copy the grid-wide database backup to both vaults.



Figure 4-6 TS7700 Grid Single Cloud Vault Access Per Cluster

In Figure 4-7, a single Cloud Export command can be performed that will ensure all pre-migrations are complete for both CLDPOOL1 and CLDPOOL2, and copy the database backup to both Vault1 and Vault2:

Figure 4-7 TS7700 Grid Multiple Cloud Vault Access Per Cluster


There is no concept of secondary copies in a Cloud Export like there is for tape Copy Export. The TS7700C will ensure that any logical volumes in the pre-migration queue that are assigned to the Cloud Pools specified in the Export List file of the LIBRARY EXPORT VOLSER have made it to the cloud vault. This monitoring includes the pre-migration queues of all TS7700C cluster across the grid. This is based on the time that the LIBRARY EXPORT command was received. Any logical volumes, for those Cloud Pools, that enter the queue after this timestamp will not be required to complete pre-migration to the cloud for the Cloud Export to complete. Further, as long as the TS7700C can communicate with other TS7700Cs in the grid, it will ensure all logical volumes in those pre-migration queues that target the Cloud Export pools have also completed before taking the database backup. If a remote TS7700C is not accessible during a Cloud Export operation and all other TS7700Cs have completed their pre-migration operations to the specified export pools, the TS7700 will continue the operation by performing the database backup and exporting the backup to the cloud vaults specified. A host attention message will be sent indicating a cluster was not accessible if a cluster is inaccessible during a Cloud Export process (See the section 4.5 Monitoring Cloud Export Operations more information).

The customer is expected to perform a Cloud Export on a regular basis, possibly more than once per day. The purpose of a Cloud Export is to capture a database snapshot that can be used as a recovery restore point. Keeping frequent backups will ensure a tighter recovery point objective. Consideration must be given for how long it will take to complete a Cloud Export operation if scheduling frequently. It could take minutes or hours depending on the size of the pre-migration queue. If the operation does not complete within 10 hours, the TS7700 will fail the LIBRARY EXPORT operation.

4.3 LIBRARY EXPORT “Export Volume“ The LIBRARY EXPORT command requires a VOLSER be specified when invoked. This VOLSER is a logical volume that contains the following three file sets:

• Export List – Contains instruction on what logical volumes should be included in the export operation. For Cloud Export, the logical volumes included in the Cloud Export are those which target the one or more specified Cloud Export Pools. A maximum of 80 characters per line is supported in this file. Export List version 04 is required for Cloud Export.

• Reserved – The reserved file does not contain any information and only contains a header that identifies it as the reserve file. The reserve file is required to exist.


• Export Status – The status file is used to return information about the export operation. The TS7700 will use it to return details about the database backup that was taken as well as a list of all logical volumes exported to the cloud vaults specified in the EXPPOOLS list. The database backup information can be variable length with a minimum of 200-characters per line if only one cloud pool is specified in the DBPOOLS field and a maximum of 7594-characters per line if 256 cloud pools are specified under the DBPOOLS field. Export Status version 01 is required for Cloud Export results.

This volume must be written with these files prior to initiating a LIBRARY EXPORT <volser> command. If any of these three files are missing from the volume, the LIBRARY EXPORT will fail. Records written in these files should be in a readable format. Therefore, the files should not be compressed or encrypted. The JCL should have TRTCH=NOCOMP set. The logical volume chosen for the LIBRARY EXPORT command should be of sufficient size to hold all the details about the logical volumes that are exported to the cloud. This is the information returned in the Export Status file. In the case there is not sufficient space on the volume, the TS7700 will stop writing when it hits LEOT and will add a closing warning message indicating the export was successful but the status file data is truncated due to the LEOT. This message will also be sent in an operator message (Event Code OP0919) to the host and an event will be posted on the TS7700 MI. Using 25GB logical tapes can help prevent such a case. These files are cataloged and managed like any other files written to tape. Consideration should be given on how to store and manage the Export Status information provided by the TS7700. This information may be relevant sometime in the future. The logical volume’s assigned Management Class should also be defined to have only one copy in the grid. That single copy consistency point needs to be the TS7700C cluster where the export will take place. For example, if cluster3 is the cluster which is to perform the Cloud Export, a MC with copy modes “No Copy, No Copy, Deferred, No Copy, No Copy, No Copy, No Copy, No Copy” should be used (e.g. NNDNNNNN). This is similar to how Bulk Volume Information Request (BVIR) is supported.

4.3.1 Export List File TS7700 microcode release 8.51.x.x introduces a new format version in the Export List file. Export List format version 04 or higher is required to perform a Cloud Export. Export List 04 has the following key fields: EXPORT LIST 04 OPTYPE CLOUD EXPORT PARAMETERS PHYSICAL POOL TO EXPORT: OPTIONS1,COPY,LMTDBPVL


EXPPOOLS expPool1 expPool2 DBPOOLS * DESC mynotes The main fields are described in Table 1 below:

Field Description

EXPORT LIST 04 Always the first line in the file. For Cloud Export, must be set to ‘04’. As of R5.1, version 04 is the latest supported version.

OPTYPE CLOUD | TAPE Rules:

- if OPTYPE TAPE do a Tape Copy Export operation - if OPTYPE CLOUD do a Cloud Export operation - if OPTYPE is empty then error

EXPORT PARAMETERS

PHYSICAL POOL TO

EXPORT:

Used for TAPE only. Will contain a tape pool. If OPTYPE CLOUD, this field is ignored. The field must exist regardless of Copy Export or Cloud Export for backwards compatibility reasons

OPTIONS1,COPY,LMTDBPVL

Used for TAPE only. Will contain a tape Copy Export options. If OPTYPE CLOUD, this field is ignored. The field must exist regardless of Copy Export or Cloud Export for backwards compatibility reasons

EXPPOOLS If OPTYPE CLOUD this will contain a list of Cloud Pools (by nickname) that will be monitored in the pre-migration queue. Rules:

- If ‘*’ is specified, the TS7700 will monitor logical volumes assigned to all existing cloud pools

- Cloud pool names are separated by a single space - Cloud pool names are not case sensitive - Maximum line length in the Export List file is 80

characters, therefore, multiple EXPPOOLS lines are supported to accommodate a longer list of cloud pool names. Each new line containing additional pools must start with EXPPOOLS.

- If this field is empty or missing, Export will fail - If any cloud pool name does not exist, Export will fail

If OPTYPE TAPE - …and EXPPOOLS is blank, assumes it is a tape Copy

Export operation and will ignore it - …and EXPPOOLS contain Cloud Pool nicknames,

Export will fail

DBPOOLS If OPTYPE CLOUD this field can contain a list of Cloud Pools (by nickname) in which a copy of the database backup will be exported. This pool or pools do not need to contain logical


volumes and can be the same or different from the EXPPOOLS list. This field is optional with the following rules:

- If ‘*’ is specified, the TS7700 will export a copy of the database backup to all Cloud Pools defined in the EXPPOOLS list

- Cloud pool names are separated by a single space - Cloud pool names are not case sensitive - Maximum line length in the Export List file is 80

characters, therefore, multiple DBPOOLS lines are supported to accommodate a longer list of cloud pool names. Each new line containing additional pools must start with DBPOOLS

- If DBPOOLS is empty or missing, the TS7700 will export a copy of the database backup to all Cloud Pools listed under EXPPOOLS

- If any cloud pool name listed does not exist, Export will fail

If OPTYPE TAPE - …and DBPOOLS is blank, assumes it is a tape Copy

Export operation and will ignore it - …and DBPOOLS contain Cloud Pool nicknames, Export

will fail DESC A 12-character field that can be used to include a description.

It is separated by a single space after the keyword DESC. This information will be returned in some Library Request and BVIR commands for a Cloud Export. This field is optional and is not currently used for tape Copy Export.

Table 1 LIBRARY EXPORT - EXPORT LIST 04 File Fields

4.3.2 Reserved File As mentioned previously, this file is not currently utilized and is reserved for future use. However, the file must exist and, along with the header information, must contain at least one record, although the TS7700 does not check its content. For example, create the following record:

RESERVED FILE

4.3.3 Export Status File The Export Status file must contain the following as the first line of the record:

EXPORT STATUS 01 <user field>


The 01 is the file version. The user field is optional and is a maximum of 16 characters long, separated with a single space from the version number. This user field is for user purposes only and is ignored by the TS7700. The TS7700 will use this Export Status file to return information about the database backup that was exported to the cloud pools specified in the DBPOOLS list as well as a list of logical volumes that exist in the cloud as of the time the database backup was taken for the cloud pools specified in the EXPPOOLS list. When the TS7700 returns this information, it will update the version number on the header record from EXPORT STATUS 01 to EXPORT STATUS 02.

4.4 Sample JCL for Cloud Export Request This section shows a sample JCL to perform a Cloud Export: //EXPORT JOB 1,'EXPORT',MSGLEVEL=(1,1)

//****************************************

//* DO NOT USE COMPACTION WHEN WRITING THE REQUEST FILE

//* SUBSTITUTE YOUR OWN DATA SET NAMES, JOB NAME, ETC.

//*

//****************************************

//* FILE 1: EXPORT LIST

//****************************************

//STEP1 EXEC PGM=IEBGENER

//SYSPRINT DD SYSOUT=*

//SYSIN DD DUMMY

//SYSUT2 DD DSN=HILEVELQ.EXPLIST,

// UNIT=VTS1,DISP=(NEW,KEEP),LABEL=(1,SL),

// VOL=(,RETAIN),

// DCB=(RECFM=FB,BLKSIZE=80,LRECL=80,TRTCH=NOCOMP)

//SYSUT1 DD *

EXPORT LIST 04

OPTYPE CLOUD

EXPORT PARAMETERS PHYSICAL POOL TO EXPORT:

OPTIONS1

EXPPOOLS MyPool1 MyPool2 MyPool3 MyPool4 MyPool5 MyPool6

EXPPOOLS MyPool7 MyPool8 MyPool9 MyPool10

DBPOOLS MyPool1 MyPool2

DESC July1920A

/*

//****************************************

//* FILE 2: RESERVED FILE

//****************************************

//STEP2 EXEC PGM=IEBGENER,COND=(4,LT)


//SYSIN DD DUMMY


//SYSUT2 DD DSN=HILEVELQ.RESERVED,

// UNIT=VTS1,DISP=(NEW,KEEP),LABEL=(2,SL),

// VOL=(,RETAIN,REF=*.STEP1.SYSUT2),

// DCB=*.STEP1.SYSUT2

//SYSUT1 DD *

RESERVED FILE

/*

//****************************************

//* FILE 3: EXPORT STATUS FILE

//****************************************

//STEP3 EXEC PGM=IEBGENER,COND=(4,LT)


//SYSIN DD DUMMY

//SYSUT2 DD DSN=HILEVELQ.EXPSTATS,

// UNIT=VTS1,DISP=(NEW,CATLG),LABEL=(3,SL),

// VOL=(,,REF=*.STEP1.SYSUT2),

// DCB=*.STEP1.SYSUT2

//SYSUT1 DD *

EXPORT STATUS 01

/*

4.5 Monitoring Cloud Export Operations Once the files have been written to the logical volume and the volume is demounted, the LIBRARY EXPORT <volser> command can be invoked. The TS7700 will receive the command and first read the content of the Export List file. If all the options in the Export List file pass precheck, the TS7700 will send an operator message as follows: E0000 EXPORT OPERATION STARTED FOR EXPORT LIST VOLUME JA0101

The TS7700 will capture the current time and start monitoring the pre-migration queues of all clusters within the grid. When this occurs, another operator message is sent: E1030 COPYING LOGICAL EXPORT VOLUMES FROM CACHE TO CLOUD

The current time will be used as a marker to monitor the pre-migration queues. All logical volumes that were in the queue at the time of this command timestamp will be monitored until they are pre-migrated to the Cloud. This includes the pre-migration queue of all peer TS7700C clusters where one or more logical volumes target the pools specified by EXPPOOL. Keep in mind that any logical volumes which were previously pre-migrated or migrated to the cloud are included in the export list. Only those that are in-flight or not yet copied (pre-migrated) to the cloud are monitored for completion. The TS7700 will send an operator message to the host every 5 minutes for each cloud pool specified in the EXPPOOL list to provide status. Here is an example of the status message:


E1031 COPYING TO CLOUD CLDPOOL1 (ID:3660020200731151923), CL/CNT/LASTVOL,

0/23/JA0006, 1/16/A98988, 2/0/-, 3/0/-, 4/1/X77777, 5/0/-, 6/0/-, 7/0/-

E1031 COPYING TO CLOUD CLDPOOL2 (ID:3660020200901552539), CL/CNT/LASTVOL,

0/111/JC6999, 1/0/-, 2/0/-, 3/0/-, 4/55/X00012, 5/0/-, 6/0/-, 7/0/- The messages can be read as follows (Table 2):

Field Description E1031 COPYING TO CLOUD CLDPOOL1

Message E1031 returns the status of a Cloud Export operation. In the examples above, the status is for cloud pools CLDPOOL1 and CLDPOOL2 respectfully.

ID: This is the 19-character cloud pool ID that is associated with CLDPOOL1 and CLDPOOL2

CL The cluster number. All 8 clusters (0 through 7) are always represented in this output even if they are not active clusters in the Grid.

CNT The number of logical volumes in the pre-migration queue for each cluster in the grid and for this cloud pool. A value of 0 indicates no volumes are in the queue for that cluster/cloud pool.

LASTVOL The last VOLSER in the queue for this cloud pool, which represents the youngest logical volume queued. This provides some perspective of the last VOLSER that will need to be pre-migrated before this cluster/cloud pool has completed all required pre-migrations. A ‘-‘ is used if the CNT is zero.

Table 2 E1031 Copying to Cloud Message Format

If a remote cluster is unavailable or status of its pre-migration queue cannot be obtained, the TS7700 will not abort the Cloud Export Operation. While it waits for all other logical volumes to be pre-migrated to cloud it will periodically check the unavailable cluster for status. Each time it checks and is unable to communicate with the cluster it will post the following message: "E1033 CLUSTER <cluster id> IS ASSOCIATED WITH <export pool name> (ID:<export

pool id>) PREMIGRATION STATUS UNOBTAINABLE." Once all logical volumes have been pre-migrated to the EXPPOOLS specified for all clusters that can provide a status, an operator message will be sent to the host indicating the completion of the copies: E1039 COPIED LOGICAL EXPORT VOLUMES FROM CACHE TO CLOUD

Next, the TS7700 will generate a list of active logical volumes that now reside in the cloud for all cloud pools listed in the EXPPOOLS list. This includes volumes which already existed


in the pools prior to the export request as well as those which were copied to the pools as part of the export request. The following operator message will be sent indicating the start and end of this step: E1040 CREATING CLOUD VOLUME MAP REPORT FILE

E1049 CREATED CLOUD VOLUME MAP REPORT FILE

Then, the TS7700 will generate a backup of its database and export it to the cloud pools specified by DBPOOLS. The following operator messages will be sent for each of these steps: E1100 CLOUD EXPORT PROCESSING STARTED

E1110 DB BACKUP PROCESSING STARTED

E1119 DB BACKUP PROCESSING COMPLETED

E1120 EXPORTING DB BACKUP TO CLOUD POOLS

E1129 EXPORTED DB BACKUP TO CLOUD POOLS

4.6 Host Completion Message Once the Export operation completes the following information is returned to the host: dev=8200:

Operation_Completion:

Library_Sequence_Number=BA06C

Operation_Type_Code=1(Host Initiated Library Export)

Operation_Completion_Code=0x00(Complete - No Errors or Exceptions)

Operation_Flags=0x40(Auto Mode|Count/MBytes are valid)

Export_List_Volser=JA0101

Physical_Volumes_Processed=0

Logical_Volumes_Requested=0

Logical_Volumes_Exportable=0

Logical_Volumes_Exported=206

MBytes_Exported=824000

MBytes_Moved_During_Export=0

Failure_Reason_Message=

If the operation fails, an operator message with a description of the failure will be sent. See Appendix A for a full list of potential operator messages that could be sent.

4.7 Operator Interventions During certain errors, an operator intervention could be posted with recommended actions that could be performed by an operator to correct the issue. These errors are also listed in the Export Status file (Table 3):

ID Failure-Reason Text Recommended Action


OP0919 Cloud export completed successfully. However, the cloud export status report had been truncated using logical volume <volser> due to the volume size or logical block count being exceeded

During the Cloud Export, if the export status report is too large for the Export List volume capacity or 4.1 million block count, the Cloud Export operation completes but the report is truncated. Recommended action is to retry Cloud Export using a larger logical volume (e.g. 25 GB) to allow the entire export status report to be written. If the export status report is still truncated with 25GB volume or the record count exceeds 4.1 million blocks, the user can issue a BVIR CLOUD VOLUME MAP using multiple logical volumes to get the list of logical volumes that reside in the cloud.

OP0918 Cloud pool <pool name> needs a cloud account, URL and cluster association to pre-migrate the volume <volser> to the cloud

During the a Cloud Export test, which is described later in this document, cloud pre-migration fails if the redirect pool is not configured. Recommended action is to ensure a cloud account, container, cloud URL and cluster association are created for the redirect pool (DRTEMP).

Table 3 Cloud Export and Recovery Operator Interventions

Additional operator intervention messages could be posted if a problem is detected while trying to communicate with the cloud storage device. This can occur during normal pre-migration activities, recalls or if the cloud health monitor is enabled. For more information about these subjects please refer to the IBM TS7700 R5.0 Cloud Storage Tier Guide at http://www.redbooks.ibm.com/abstracts/redp5573.html.

4.8 Export Status File On a successful completion of a Cloud Export operation, the TS7700 will generate two types of records in the EXPORT STATUS file:

• Database backup details – A list of the database backups and which cloud pools (cloud vaults) the backup was copied to

• A list of logical volumes exported to the cloud by pool for the pools specified in the EXPPOOLS list

These records are not written unless the Cloud Export was successful. The database backup detail records are variable length with a maximum length of 7594-characters per record and a maximum of 4.1 million records that can be returned in a single logical volume. A full list of all logical volume objects stored in a cloud storage device can be retrieved by using the CLOUD VOLUME MAP BVIR command. For more information on the BVIR command see the White Paper - TS7700 Bulk Volume Information Retrieval V5.1.pdf at https://www.ibm.com/support/pages/node/6355041. Both the database backup list and logical volume list begin with a single row header that provides information about the content of the data below it. The header names can be seen under the Name column in Table 4 and Table 5 below. The header row and data rows utilize the same column width values. The header and data rows are left-justified and


https://www.ibm.com/support/pages/node/6355041


padded with blanks to fit the length of the column. Some fields are variable length and the header will be adjusted to the actual length of the data accordingly. For each DBPOOL defined, a database backup record will be created. The following table defines the format for the database backup record rows in the output (Table 4):

Bytes Name Description 0-4 S/N The 5-character server serial number of the cluster the cloud export

operation was performed on. This value is required to perform a Cloud Export Recovery and should be retained.

5 Field Delimiter Comma character

6-13 BACKUP_ID A 1 to 8-digit unique backup ID generated by the TS7700. This ID is required to identify the database backup that will be used to perform a Cloud Export Recovery and should be retained. This field is left-justified and padded with blanks.


15-22 BACKUP_P A maximum of 8-character cloud pool name for the cloud vault where a copy of the database backup will be written. This cloud pool name is retrieved from the DBPOOLS list field in the EXPORT LIST file and indicates this backup ID was stored using the cloud credentials for this cloud pool. The cloud pool name should be retained, along with the cloud account and container information associated with it. This information is required to a Cloud Export Recovery. This field is left-justified and padded with blanks.


24-42 BACKUP_POOL_ID A 19-character cloud pool ID for the cloud vault where a copy of the database backup was written. This is the cloud pool ID associated with the cloud pool name specified in BACKUP_P and should also be retained to identify where the database backup specified in the Backup ID was written.


44-99 BACKUP_OBJECT_NAME

A 56-characters field that provides a full object name for the database backup that was exported to the cloud storage device. This is the name of the object in the object store. The backup object name is composed of the following format to create uniqueness in the cloud storage system:

/DBEXPORT/DBBACKUP.<5 char server serial#>.<5 char

composite_id>.<5 char distributed_id>.<14 char db2

timestamp>.001 The db2 timestamp is formatted as follows:

YYYYmmddhhmmss (example: 20200116162207) This field is left-justified and padded with blanks.



101-126 STARTING_CMD_TIMESTAMP

The time all pre-migrations were completed for a LIBRARY EXPORT command that ensures all copies made it to the cloud. This timestamp will be used to determine when the backup can be deleted once all older versions of a logical volumes that are part of this pool have also exceeded the pool version retention period and have been deleted.

A 26-character timestamp field with the following format:

YYYY-mm-dd-hh.mm.ss.sssssss

(example: 2020-08-20-19.31.35.123456)


128-135 SOURCE An 8-character field that describes the source of how the database backup was created and placed on the cloud storage device. Currently, the only supported source value is ‘ZEXPORT’, which indicates it was initiated from the IBM Z using the LIBRARY EXPORT command. This field is left-justified and padded with blanks.


137 S Status of the backup. The following are valid status values: 0 = ‘NORMAL’ normal run state 1 = ‘DRHOLD’ The backup is in a hold state and will not be expired in the cloud even if the cloud pool retention time is exceeded. A hold can be placed on a cloud pool associated with this backup as part of a Cloud Export Recovery Test operation.


139-157 FILE_SIZE The size of the database backup in bytes. This field is left-justified and padded with blanks.


159-170 DESCRIPTION Maximum 12-character user supplied description provided in the DESC field in the EXPORT LIST file. This field is left-justified and padded with blanks.


172-2474

POOLNLST Maximum 2304 characters. A space delimited list of cloud pool names that are expected to be in this backup. This list is retrieved from the EXPPOOLS field in the EXPORT LIST file. The maximum number of regular pools is 256 and the maximum length of a cloud pool name is 8 characters plus the space delimiter. So this field supports (256*8)+255 space characters. This field is left-justified and can be variable length.


2476-7594

EXPORT_POOL_ID_LIST

Maximum 5120-characters. A space delimited list of cloud pool IDs that correspond to the POOLNLST lists. These are the cloud pool IDs generated by the TS7700 when a cloud pool is created and are associated with the cloud pool name listed in the Export Pool Name List. The maximum number of regular cloud pools is 256 and the length of a cloud pool ID is 19 characters plus the space delimiter. So this field supports (256*19)+255 space characters. This field is left-justified and can be variable length.

Table 4 Database Backup Record Format


For each logical volume that is contained within an EXPPOOL at the end of the export operation, a record will be included. The following table defines the format for the logical volume export record rows in the output (Table 5):

Bytes Name Description 0-5 VOLSER A 6-character volume serial number of an active logical volume that

exists in the cloud vaults listed under EXPPOOLS at the end of this Cloud Export operation.


7-26 INSERT_VERSION The insert version of the logical volume. Used to determine the overall version of the data. Left-justified and padded with blanks.


28-47 DATA_LEVEL The data level of the logical volume. Along with the insert version, determines overall version of the data. Left-justified and padded with blanks.


49-56 SYSPLX_N Up to 8-character customer-defined host sysplex name for the host which initially created the logical volume prior to its pre-migration to the cloud. Left-justified and padded with blanks.


58-65 SYSTEM_N Up to 8-character customer-defined host system LPAR name for the host which initially created the logical volume prior to its pre-migration to the cloud. Left-justified and padded with blanks.


67-74 PRGRAM_N Up to 8-character program name associated with the application which initially created the logical volume prior to its pre-migration to the cloud. Left-justified and padded with blanks.


76-95 FILE_SIZE The size of the logical volume in bytes pre-migrated to cloud, left justified and padded with blanks.


97-115 CLOUD_ACCOUNT_ID 19-character account ID generated by TS7700 internally. The cloud account ID is generated when a new cloud account is created by a user on the TS7700 Cloud Account MI page.


117-124 CLDACC_N Up to 8-character cloud account name defined by the customer, left justified and padded with blanks.


126-130 E_FLG The cloud data encryption type. This field is not used at this time and is always 0.


132-195 ENCRYPTION_KEY_LABEL1

The encryption key label 1 when data on cloud is encrypted. This is not used at this time and is always ‘NULL’, left justified and padded with blanks.


197-261 ENCRYPTION_KEY_LABEL2

The encryption key label 2 when data on cloud is encrypted. This is not used at this time and is always ‘NULL’, left justified and padded with blanks.


263-267 CDFMT The data format on cloud. This is not used at this time and is always ‘1’, left justified and padded with blanks



269-273 DLRSN The delete reason when data on cloud is deleted. This is always ‘0’, left justified and padded with blanks.


275-300 DELETED_TIME The delete timestamp when data in the cloud would have been deleted. This field is used for volume version retention and this output contains

information about active volumes, so will always show ‘1970-01-

01-00.00.00.000000’. A 26-character timestamp field with the following format:



302-327 PREMIGRATED_TIME The pre-migration timestamp when the logical volume was pre-migrated to cloud. A 26-character timestamp field with the following format:



329-333 INITR The cluster ID which pre-migrated the logical volume to cloud (the ‘Initiator’), left justified and padded with blanks.


335-353 CLOUD_POOL 19-character ID generated by TS7700 internally. The cloud pool ID is generated when a new cloud pool is created by a user on the TS7700 Cloud Pool MI page.


355-362 CLDPOL_N Up to 8-character cloud pool name defined by the customer when a cloud pool is created, left justified and padded with blanks.


364-368 CPM_R The pre-migration rank of the cloud pool defined in the storage construct (storage group) bound for the logical volume. Left justified and padded with blanks. This field is always set to ‘1’ (rank 1) at this time.


370-374 CMPST The composite library view of data state on cloud for the logical volume, left-justified and padded with blanks. The following values may be seen: ‘5376’ (“data is pre-migrated to cloud”), ‘12544’ (“data on cloud needs to be deleted”), ‘13056’ (“data on cloud failed to be deleted and will be retried to be deleted”) or ‘13568’ (“data on cloud is being deleted”). Retained logical volume has ‘12544’ (“data on cloud needs to be deleted”) in this field AND ‘1’ in RS (“Retention State”) field.


376-380 CLxST The distributed library (local cluster) view of data state on cloud for the logical volume, left justified and padded with blanks. ‘x’ is a cluster ID where this Cloud Export was run. Since each TS7700C cluster in the grid can have a different pre-migration states, the exporting cluster may not have completed its pre-migration when the export completed. This is okay since at least one peer did complete the pre-migration process. The following value could be provided: 0 – The local cluster has not pre-migrated the logical volume to cloud yet, or it’s not configured to pre-migrate the logical volume to cloud. 4352 – The local cluster failed to pre-migrate the logical volume to cloud. A retry will be attempted automatically.


4864 – The local cluster is in the process of pre-migrating the logical volume to cloud. 5376 – The local cluster has pre-migrated the logical volume to cloud, or the local cluster can access the data on cloud when a peer cluster in the Grid has pre-migrated the data to cloud already. 12544 – The data on cloud needs to be deleted. 13056 – The local cluster failed to delete the data on cloud. A retry will be attempted automatically. 13568 – The data on cloud is being deleted by the local cluster. 14080 – The deletion of the data on cloud has completed. 20736 – The deletion of the data on cloud has completed and this cloud volume map entry is ready to be deleted completely. Retained logical volume can have ‘12544’ (“data on cloud needs to be deleted”) in this field. Retained logical volume has ‘1’ in RS (“Retention State”) field.


382-386 CCMRU The cluster bit mask which indicates which clusters have the down level entry. Left justified and padded with blanks. If it is 0, the synchronization of this cloud volume map entry is done in the Grid. If it is not 0, the corresponding cluster still has down level entry and the synchronization will be done automatically


388-515 OBJECT_NAME A maximum of 128-character. This is the fully qualified object name of this logical volume as it is stored in the cloud, left justified and padded with blanks. The format of the object name is: "Customer Provided Object Pre-Fix Name (defined in storage group bound for this logical volume)"/”4 hex digit random hash”/"Composite Library Sequence Number"/"Distributed Library Sequence Number"/"Logical Volume Serial Number"/Insert Version/Data Level/”Timestamp when this entry is created"


517-772 CONTAINER_NAME A maximum of 256-character container name defined by the customer where the logical volume is contained on cloud, left justified and padded with blanks.


774-775 RS The retention state of the logical volume. 0 – None: Latest version logical volume or Latest version logical volume which is marked deleted but not actually deleted yet. 1 – Pool Retention: Older version logical volume which is marked need to be deleted but retained due to retention days setting of Cloud Pool.


777-802 EXPIRE_TIME The timestamp when retention duration of the logical volume expires. When RETENTION_STATE value is 0, this is always ‘1970-01-01-00.00.00.000000’. When RETENTION_STATE value is 1, the timestamp is shown in ‘yyyy-mm-dd-hh.mm.ss.ssssss’ format.

Table 5 Logical Volume to Cloud Mapping

See Appendix B for an example of this output.


If the Cloud Export operation fails, the Export Status File will contain information about the failure along with or instead of the database backup and logical volume list.

4.9 Sample JCL Read Export Status File The following is a suggested JCL job to read the Export Status file since some of the data can be variable length: //CLDEXP JOB ...

//********************************************************************

//* COPIES THE EXPORT STATUS FILE - DATA IS NOT COMPACTED

//********************************************************************

//* SUBSTITUTE YOUR OWN DATA SET NAME, JOB NAME, HILEVELQ, ETC.

//********************************************************************

//* USING THE DATA SET CATALOGED IN THE REQUEST JOB

//********************************************************************

//* FILE SEQUENCE 3: EXPORT STATUS FILE

//********************************************************************

//STEP1 EXEC PGM=IEBGENER

//SYSPRINT DD SYSOUT=A

//SYSIN DD DUMMY

//SYSUT1 DD DSN=HILEVELQ.EXPSTATS,DISP=OLD,

// LABEL=(3,SL),DCB=(RECFM=U,BLKSIZE=8000)

//SYSUT2 DD DSN=HILEVELQ.EXPSTATS.RESULTS,

// UNIT=SYSDA,SPACE=(CYL,(5,5)),

// DISP=(NEW,CATLG),

// DCB=(RECFM=U,BLKSIZE=8000)

//SYSIN DD DUMMY

/*

4.10 Canceling Cloud Export Operations There are several reasons for canceling an export operation:

• After being initiated, you may realize that a cloud pool being processed for export or a targeted backup pool was incorrect.

• Other, more critical workloads are required to be run on the TS7700 and the extra overhead of running the export operation is not desired.

• A problem is encountered with the export that cannot be quickly resolved, for example problems with a cloud storage device.

• An export was initiated before an important batch job completed. A request to cancel an export operation can be initiated from any host attached to the TS7700 subsystem using one of the following methods: • Host console command: LIBRARY EXPORT,XXXXXX,CANCEL - where XXXXXX is the

volume serial number of the Export List File Volume.


• Program Interface using the Library Control System (LCS) external services CBRXLCS. How quickly the TS7700 will react to a Cloud Export cancelation will depend on where in the process the cancelation occurred. If the cancelation occurs while the TS7700 is waiting for the cloud pre-migrations to complete, the TS7700 may not react instantly to the cancelation. The TS7700 does a periodic check for pre-migration status and will discover the cancelation on the next status check interval. At that point, it will exit from the Cloud Export process and send a regular CANCELED status message. Pre-migrations will not be affected and continue as normal. If the Cloud Export was canceled while the TS7700 is exporting the database backup file to multiple cloud vaults, the TS7700 will continue the export to the current cloud vault then discontinue the export to any future cloud vault and exit from the Cloud Export process. This scenario results in an ECANCELED status message indicating an error condition because a portion of the database backups was exported to the cloud. A daily cleanup process removes the backups that made it to the cloud regardless of cloud pool retention time if the backup status is marked as failed. The daily cleanup process also removes database backups if it detects there are no logical volumes associated with the cloud pool (there is neither any latest version volume nor retained old version volume) regardless of cloud pool retention time. The completion message is sent to all hosts attached to the TS7700 indicating that the export was canceled via a host request:

"E1190 CLOUD EXPORT CANCELED BY HOST REQUEST." Examine the export status file or the host SYSLOGs to see what had been processed prior to the cancel request.

5 Planning and Considerations for a Cloud Export Recovery This section will discuss what you will need to know to be prepared for a cloud export recovery and to do a read only recovery if needed for testing or selective volume restore.

5.1 Retaining Database Backup Information It is important to retain the backup information returned in the Export Status file. This information is needed to identify the backup needed during a Cloud Export Recovery operation. The key fields in the cloud backup information listed in Table 4 above are the S/N (server serial number) and the BACKUP_ID. Users should store all cloud credential information in a safe place and use it to match the cloud pool name (BACKUP_P) and ID (BACKUP_POOL_ID) from Table 4, which identifies


where the database backup is stored. The cloud credentials will be needed to configure the stand-alone TS7700 where the recovery will be performed. A full list of all the database backups stored in the cloud can also be generated by the following two commands:

• LIBRARY REQUEST CLDBKUP SHOW <page index> • BVIR CLOUD BACKUP LIST

The Library Request command will return a list of backups stored on all cloud vaults in a TS7700 grid. Each request will return 49 backup records per page index ordered by time listing the most recent backups first. The information is limited to the cloud pool name where the backup is stored, backup ID, backup object name and the 12-character user defined description that was provided in the Export List file during the Cloud Export. The backup object name will contain the server serial number as part of its name located directly after the /DBEXPORT/DBBACKUP. A database backup object name is comprised of the following fields: DBEXPORT/DBBACKUP.<5 char machine serial#>.<5 char composite id>.<5 char

distributed id>.<14 char timestamp in db backup>.001

An example of a backup object name is as follows: /DBEXPORT/DBBACKUP.HDB50.BA012.BA12B.20200831202131.001

The long object name is to provide world-wide uniqueness and to ensure no object collisions can occur in the event other systems are connected to this cloud vault. The BVIR command also lists all backups stored on all cloud vaults in a TS7700 grid but provides much more detail about the backup. The output is similar to the information returned in the Export Status file during a Cloud Export except this is a complete list of all backups. More information about LIBRARY REQUEST commands can be found in the White Paper – TS7700 Library Request Command V5.x.pdf (or later version). More information about the BVIR command can be found in the White Paper – TS7700 Bulk Volume Information Retrieval V5.x.pdf (or later version). These documents, along with others, can be found in IBM Techdocs: https://www.ibm.com/support/pages/ibm-techdocs-technical-sales-library. 5.2 Life Cycle of Database Backups in the Cloud Another important aspect of cloud recovery is understanding how long an exported database backup will be retained. Section 1.5 introduces the concept of cloud pool level logical volume version retention. Logical volume versions can be retained for an additional

https://www.ibm.com/support/pages/ibm-techdocs-technical-sales-library

https://www.ibm.com/support/pages/ibm-techdocs-technical-sales-library


period of time by setting the number of Retention Days in the Cloud Pool page on the TS7700 MI. This Retention Days field is also used to determine how long to retain a database backup that was written to a cloud vault as part of a Cloud Export operation. A Retention Days value of 0 means not to retain any logical volume versions. However, a cloud backups will be retained for a minimum of one day even if the Retention Days of all export pools are set to 0. When a Cloud Export is performed and database backups are exported to one or more cloud vaults, the TS7700 starts tracking the expiration period of those database backups. If more than one cloud pool is specified in the EXPPOOLS list, the TS7700 uses the MAX(Retention Days) for all exported pools included in the backup (Figure 5-1).

Figure 5-1 Database Backup Cloud Retention Days

For instance, Figure 5-1shows EXPPOOLS lists three cloud pools: cldpool1, cldpool2 and cloudpool3. Each cloud pool has a different RETENTION DAYS setting. The TS7700 will use the longest retention period of the three pools, in this case 21 days + 1 day1, to determine how long to retain the database backup. The reason for this is the cloud vaults that the database backups are stored in (DBPOOLS) are not always the same cloud vaults the logical volumes were migrated to (EXPPOOLS). This is the case in the example in Figure 5-1. Since each cloud pool can have a different Retention Days setting, it is not always clear which retention period the database backup should use if the cloud pools are different.

1 Cloud pool Retention Days must be exceeded for the deletion process to occur so one day is always added to the setting


The cloud pools listed in the DBPOOLS in this example are cldpool1 and cldpool4. Pool cldpool1 has a Retention Days setting of 7 days and cldpool4 has a Retention Days setting of 0 days. If the database backups are allowed to expire after 1 day (cldpool4) or 7 days (cldpool1) then both backups will be deleted. However, logical volume versions are still being retained for cldpool2 and cldpool3 and won’t expire until after 14 and 21 days. This would be an issue if a recovery was needed after day 7 and no backup exists. For this reason, the TS7700 will use the MAX(Retention Days) of the EXPPOOLS to ensure the database backup is not deleted prior to the logical volume version retention periods being reached. Users should also consider how frequently Cloud Exports are requested. The more frequent, the better the recovery point objective since less time exists between Cloud Export backups. At the very least, Cloud Export operations should be performed more frequently than the shortest Retention Days setting for the Cloud Pools being exported. For instance, using the example in Figure 5-1, cldpool1 has a 7-day volume version retention period. A Cloud Export should be performed at least every 5-6 days for cldpool1 to ensure there is adequate recovery time for those logical volume versions that may be nearing their version retention period. If a Cloud Export backup is restored and it references volumes that have passed their retention periods, access to those volumes will fail. This is because once a retention period expires, the source cluster will delete the retained versions in the cloud making them no longer accessible. A Best Practice is to perform at least one Cloud Export operation per day with a minimum Retention Days setting of 2 or more days. Section 8, Volume Version Retention by Cloud Pool, in this document will discuss the concept of logical volume version retention periods in further detail. 5.3 Setting up a TS7700 Recovery Cluster A Cloud Export Recovery is a customer-initiated operation that restores a stand-alone TS7700C to a fully functional state where it can read and write data that was preserved in the cloud. This TS7700 must meet the following requirements:

• The recovery TS7700 cannot be part of a Grid configuration. • The recovery TS7700 must be configured as a cluster 0. • The recovery TS7700 must be at microcode level 8.51.x.x or higher to perform a

Cloud Export Recovery. • The code level on the recovery TS7700 must be at the same level or higher as the

source TS7700. • The recovery TS7700 must be fully operational and be in an online state to start

recovery.


• Logical devices do not need to be varied online to any attached host, though it is recommended to receive any host console messages which may be surfaced during the recovery process.

• The recovery TS7700 must be configured as a cloud attached system. • The TS7700 must have Ethernet access to the cloud vaults and those cloud vaults

should be in a fully functional state. At a minimum, the cloud vault that contains the database backup must be operational to start the recovery and any vaults which contain logical volume data will be configured during the recovery process.

• It is highly recommended that the recovery TS7700 contain the same feature codes as the original production cluster in order to access all previous functionality and configuration requirements. Missing feature codes could cause an issue with accessibility.

• If the production TS7700 environment was configured to use LDAP service authentication, so too must the TS7700 recovery system. The recovery TS7700C should be fully configured and tested to ensure LDAP is working properly before beginning a Cloud Export Recovery. This includes ensuring a valid sign-on certificate is loaded.

• At least one cloud pool, account, container, cloud URL and cluster association must be configured on the recovery TS7700 prior to starting a Cloud Export Recovery. These are the same steps explained in section 4.1 Configuring Cloud Pools, Accounts & Containers. These credentials must provide access to the cloud vault that contains the database backup that will be restored. Only one set of cloud credentials need to be entered. All others not associated with the container ID provided during a Cloud Export Recovery will be discarded during the restore.

• If the recovery TS7700 is not empty of data (in cache or the database), the Cloud Export Recovery will erase all data from the system.

• The recovery process will only restore logical volumes that were exported (pre-migrated) to the cloud for all cloud pools that existed at the time of the database backup. This database backup will not contain information about cloud pools or logical volumes created after the backup was created.

• After recovery, scratch logical volumes will need to be inserted. If the recovery is for a real disaster (rather than just a test), verify that all policy actions are defined appropriately for a stand-alone system. This is especially important when coming from a grid configuration where construct actions may be handled differently. Consider verifying the following settings:

• Management Class copy modes • Storage Class preference groups and partition settings • Categories and category retention periods • Cache partitions • Cloud version retention values • Security Settings • Library Port Access Groups • SSL Certificates


6 Performing Cloud Export Recovery Once the stand-alone recovery system is completely operational and new cloud credentials are created for the DBPOOLS, a Cloud Export Recovery can begin. The user must obtain the cloud container ID from the TS7700 Container page on the MI for the cloud vault where the database backup exists (Figure 6-1).

Figure 6-1 TS7700 Container ID

A user will also need the BACKUP_ID and S/N (server serial number) of the TS7700 cluster that originally created the database backup. This information was returned in the Export Status file of the logical volume that was used in the Cloud Export operation that generated the database backup (see Table 4). A Cloud Export Recovery is invoked on the TS7700 Management Interface Cloud Export Recovery page under the Service icon in the left navigation window as shown in Figure 6-2.

Figure 6-2 Cloud Export Recovery MI Page Navigation

This page is only visible in the Service menu if the TS7700 is a stand-alone system and the user was given permission to access that page. Enter the container ID, serial number and the backup ID in the Cloud Export Recovery page. If this is a true disaster recovery, uncheck the Test mode Enable box and select the Recover Cluster button. More information regarding Test Mode can be found in section 7. Below. This section will focus on a real disaster recovery.


The Cloud Export Recovery page can be seen in Figure 6-3.

Figure 6-3 Cloud Export Recovery Page

If the TS7700 detects logical volume data or meta data on the system, the following window will be displayed (Figure 6-4):


Figure 6-4 Cloud Export Recovery Data Exists Warning

This cleanup will remove any logical volume data or meta data (TS7700 database) on this recovery system. The cleanup is required before proceeding with the recovery. Selecting Yes will begin the cleanup and recovery process. Selecting No will abort the Cloud Export Recovery operation.

6.1 Monitoring Cloud Export Recovery Operation Once the recovery operation begins, a status window will show the progress of the recovery. Figure 6-5 is an example of the status window. During the recovery, the TS7700 must be taken offline. This is required to clean up and replace the database. During offline processing, the Management Interface is restarted. This will cause a temporary loss in connectivity and any users logged into the MI will be logged off. After a few minutes, MI access will be restored but with only limited accessibility. A user can access the Cloud Export Recovery status page once again to monitor progress.


Figure 6-5 Cloud Export Recovery Status Window

During recovery, the TS7700 will clean up the existing database if needed, retrieve the database backup from the cloud container using the container ID provided and restore it, then make necessary modifications to the database to conform to the new environment. During the modification steps, the TS7700 will scan for logical volumes that may have been in the process of being pre-migrated to the cloud or that have reached their retention period and may have been in the process of being deleted. Any logical volume that is in a PREMIG_INPROGRESS state will go through a cloud verification process during TS7700 online processing. Any logical volume that was in a DELETE_INPROGRESS state or that cannot be verified as being deleted from the cloud will be marked as DELETE_FAILED and will go through the delete process again after the TS7700 goes online. The new cloud credentials that were entered to provide cloud vault access to retrieve the database backup will be removed. This is done to avoid conflicts with any restored cloud credentials that may contain the same cloud pool or cloud account names. These are required to be unique so duplicates cannot exist. After a successful recovery, the status window will show 100% complete and indicate the TS7700 is being brought back online (Figure 6-6).


Figure 6-6 Cloud Export Recovery Completed Status

This Cloud Export Recovery status window will remain available until a user selects the Close Status button located at the bottom of the window. Once Close Status is selected by any MI user that has page permission, the recovery information will no longer be available. The next time the Cloud Export Recovery page is accessed, the user will be brought back to the initial view of the page as seen in Figure 6-3. If the Cloud Export Recovery failed, the status window will indicate in which step the process failed. The TS7700 will complete the following actions on a failed restore:

• Fully clean up any database work that had been started • Restore the cloud credentials associated with the Container ID that were entered by

the user at the start • Issue an operator message to the host • Bring the TS7700 back online

The user should review the host SYSLOGS for reasons why the Cloud Export Recovery failed and correct the issue.


During online process, whether it be from a successful or failed Cloud Export Recovery, the MI will once again become unavailable and any logged in users will lose connection while the web server is restarted. Once the TS7700 is fully online, web access will be available. The Cloud Export Recovery status window will be available as long as it was not previously closed using the Close Status button. Once the TS7700 is fully online after a successful recovery, all settings should be reviewed by an administrator. For instance, during Cloud Export Recovery, the storage management constructs and their actions will be restored to what was defined on the source TS7700. These settings should be evaluated after recovery to determine whether any policy management changes are needed for the new environment. In addition, any cloud pools, cloud accounts, containers and associated URLs will be restored to the values defined in the source grid. If the export restore cluster utilizes different cloud credentials, container names and/or URL values than the source grid, they must be properly configured prior to attempting host access of the restored logical volume content.

6.2 Restoring the Host Environment In order to run host applications and access the data in the recovery TS7700, the source host environment must also be restored. At a minimum, the following will need to be performed:

• Restore the tape management system control data set.

• Restore the DFSMS data catalogs, including the Tape Configuration Data Base (TCDB).

• Define the I/O gen using the library ID of the recovery TS7700.

• Update the library definitions in the SMS Control Data Set (SCDS) with the library IDs for the recovery TS7700 in the composite and distributed library definition panels.

• Activate the I/O gen and the SCDS. Also, consider updating the library ‘nicknames’ on the TS7700 to match the library names defined in DFSMS. This will allow the names on the TS7700 MI to match those used at the host for the composite and distributed libraries. This step is optional and is purely a visual aid for TS7700 MI users to ensure they are working on the correct TS7700 when reviewing or making modifications.


To set up the distributed name, select the Modify Cluster Identification item on the Actions dropdown menu in Cluster Summary. In this window, enter the composite library name or a cluster nickname into the Cluster nickname field. You can optionally provide a description (Figure 6-7).

Figure 6-7 Cluster Summary Modify Cluster Identification

To set up the composite library name for a grid system, select the Modify Grid Identification item on the Actions menu in Grid Summary. In this window, enter the composite library name used by the host into the Grid nickname field. You can optionally provide a description. This action is only available if you have two or more clusters in a grid configuration and is not available on a stand-alone system. Next, vary devices online at the host and insert logical volumes if not already done. During the restore process, any logical volume that did not have active data in the cloud vaults are deleted, so scratch volumes will need to be inserted. The same volume ranges can be re-inserted and the TS7700 will only insert volumes if it does not already exist. Once the TS7700 is in a normal state (online), it will resume normal internal operations, which includes processing volumes that are now expired and have exceeded any hold period. Therefore, it is possible some scratch volumes will become available automatically. The TS7700 is now ready to read and write data.

7 Cloud Export Recovery Test Mode Periodically, a disaster recovery test should be performed to ensure proper procedures are in place and understood. Recovery tests are often necessary to pass audit requirements that prove a disaster recovery plan does indeed work. Additionally, a user may want to restore a previous point-in-time backup to provide access to previous version of data. This may be due to extended testing or as a result of air gap recovery. The TS7700 provides a


safe way to perform cloud export recovery testing against one or more production cloud vaults. A TS7700 production environment can continue to do daily work without interruption while disaster recovery testing is being performed. This also means that data in the cloud vaults is continuously changing as new data is written, old data is deleted and existing data is modified. Operations for a cloud vault attached to a production TS7700 are not suspended in any way. One of the unique aspects of a cloud storage system is the ability to access the data from many different entry points. When the Cloud Export Recovery function is invoked from this stand-alone TS7700 recovery system, it provides another entry point into the production cloud vaults. When performing disaster recovery testing, users must be extra careful not to modify production data. Additionally, every TS7700 has internal routines that perform maintenance operations, such as deleting expired volumes. The recovery TS7700 system is no exception. To ensure data isn’t unexpectedly modified in the cloud vaults, Cloud Export Recovery should always be run with Test Mode Enabled when doing disaster recovery for any reason other than a true disaster where the production environment is lost indefinitely.

7.1 Configuring for Test Mode Follow the same procedure specified in sections under 6 Performing Cloud Export Recovery, but ensure the Test mode Enable checkbox is checked as seen in Figure 7-1.


Figure 7-1 Cloud Export Recovery Test Mode

The rest of the steps are the same as a regular Cloud Export Recovery except, during the recovery, the TS7700 will perform the following additional operations:

• All logical volumes that exist in the cloud according to this database backup will be marked with a DRHOLD flag. A DRHOLD flag prevents the volume from being modified or deleted due to TS7700 internal processing. For instance, a logical volume that reaches category expiration and is eligible to be deleted would not be deleted by this TS7700 recovery system.

• All database backups that exist in the cloud according to this recovery system will also be marked with a DRHOLD flag. These database backups were created as the result of a LIBRARY EXPORT command for a Cloud Export. The DRHOLD will prevent these backups from being deleted due to TS7700 internal processing if the backup exceeds the cloud pool version retention period on this TS7700 recovery system.

• A new cloud pool will be created during the restore process. This cloud pool will be named DRTEMP. This cloud pool will be set with READ-WRITE capability and can be used for write test operations.

• All previously existing cloud pools will be marked READ-ONLY and any new pre-migrations targeting these READ-ONLY pools will be redirected to the new DRTEMP


pool. Since all restored logical volumes target the READ-ONLY pools, all transactions for these restored volumes will use the following rules (Table 6):

Cloud Command Reaction GET GET commands will work as normal. Logical volumes will

be read from the cloud vault specified, into TS7700 disk cache as they normally are. Once restored into disk cache, they can be modified, expired and ejected, though the actual original copy in the cloud will be left alone.

PUT PUT commands are used to write new or modified logical volumes to the cloud vault. Any new pre-migrations from this recovery TS7700 targeting a READ-ONLY pool will be redirected to the DRTEMP pool. This allows the ability to test writing new test data or modifying existing data as part of disaster recovery testing without actually writing to the production cloud vault.

DELETE If a DELETE request is received for a logical volume in a READ-ONLY cloud pool, the TS7700 will mark the volume as delete required but will not perform the delete.

Table 6 READ-ONLY Cloud Pool Command Reaction

When the TS7700 comes online after a Cloud Export Recovery using Test Mode, the only cloud pool that can be written to is the DRTEMP cloud pools. All other cloud pools will be READ-ONLY at this time. The DRHOLD flag and READ-ONLY status only applies to this stand-alone TS7700 recovery cluster. This is to ensure this TS7700 does not modify anything in the original cloud vaults. This TS7700 recovery cluster is not a member of the production grid and neither this cluster nor the production cluster or grid are aware of each other.

7.2 Create an Independent Cloud Container for Testing The TS7700 recovery provides the means to do safe disaster recovery write operations by directing those operations to the DRTEMP cloud pool that was created during the restore process. To make this cloud pool usable, it must be set up with valid cloud account, container and cloud URL information. The cloud vault that DRTEMP targets should be completely independent of the production cloud vaults. This can be a temporary cloud vault located in any cloud object store supported by the TS7700 (e.g. Amazon AWS, IBM Cloud Object Store, RStor). The container does not need to be from the same cloud object store that the production data resides in.


The cloud account, container and cloud URL can be set up using the instructions found in sections 4.1 of this document. Since the cloud pool (DRTEMP) is already created, the first step should be skipped. When creating the container in the TS7700 MI Container page, select DRTEMP as the cloud pool.

Once the test cloud vault is accessible by the TS7700, disaster recovery testing can begin. Figure 7-2 shows a Cloud Export Recovery test mode environment. When the IBM Z host reads a logical volume that is stored in the production cloud vault (Vault1), the test recovery TS7700 will read from that cloud vault as normal. If the IBM Z host requests an append to an existing logical volume, whose policies target Vault1, the TS7700 will allow the append but not export this new data to Vault1 during pre-migration operations. Instead, the test recovery TS7700 will redirect the export of the modified volume to the cloud vault configured with the DRTEMP cloud pool. The TS7700 will keep track of where the new version of data resides while preserving the original version of the data by not allowing any modification activities to any volumes in a READ-ONLY pool. Likewise, new write operations to a scratch volume using policies that target Vault1 will not be exported to Vault1 during pre-migrations. These too will be redirected to the DRTEMP (TestVault) cloud vault. When reading back a modified logical volume, the test recovery TS7700 will know the volume was exported to DRTEMP (TestVault), instead of Vault1, in this Figure 7-2 example.


Figure 7-2 Cloud Export Disaster Recovery Testing

Figure 7-2 also shows how the production system continues to perform read-write activity as normal. The production system requires no configuration changes during a disaster recovery test. Any changes made on the production TS7700s are not relayed to the recovery TS7700. The exception is for Cloud Pool Retention. If a DR test is elongated and the READ-ONLY pool data retention period may expire before the test completes, the production grid’s Cloud Pool retention settings can be reconfigured to elongate the retention period and accommodate a longer test period. Retention periods are retro-active against existing data in the cloud pools. For example, if the production grid’s retention period was set to 14 days and the current restored backup was taken 10 days ago and the test is lasting longer than 4 days, the retention periods for the export pools should be elongated on the production grid. Once the test is completed, the retention periods in the production grid can be returned to their normal durations. Any change requests in the recovery TS7700, including internal operations such as cloud export backup cleanup or delete-expire, are not performed on any data with the DRHOLD flag. Any new data that is written on the test recovery TS7700 could go through delete-expire processing since scratch logical volumes inserted after the Cloud Export Recovery completed are not marked with a DRHOLD flag. This is also true for any database backups exported to a READ-WRITE cloud pool by this test recovery TS7700. Test Mode only ensures the preservation of the production cloud vault data.

7.3 Disaster Recovery Testing Cleanup


Once disaster recovery testing is complete, vary the devices offline from the IBM Z DR system. The recovery TS7700 can be shut down until its next use. Data that was redirected to the DRTEMP TestVault should be deleted. A cloud storage administrator should access the cloud container and remove the test data to prepare for the next test cycle.

7.4 Cloud Export Recovery Test Best Practices

• Use a separate cloud object store vault for the DRTEMP pool. It is not recommended to target the same production cloud vaults for disaster recovery testing due to the added risks of unintended data mixing, data modifications and deletions. Keeping it in a different vault allows a clean separation.

• Always make sure the production grid’s export pool retention periods are long enough to accommodate both the restore of the database backup and the full duration of the DR test phase. It can always be elongated during a test period if more time is needed.

8 Volume Version Retention by Cloud Pool Prior to microcode level 8.51, the TS7700 did not keep versions of logical volume. Even though a logical volume is pre-migrated to the cloud, if the logical volume is modified again or has been expired, the logical volume in the cloud is considered a stale version and is marked ready to be deleted. Beginning with microcode level 8.51, users can select how long (how many days) of stale previous versions to keep in the cloud at the Cloud Pool level. All VEC and VED clusters in a grid must be at microcode release level 8.51.x.x or higher to use versioning. This section describes how a user configures versioning and how volume versioning works. Since a cloud vault is a real time storage target for TS7700 logical volumes, a point-in-time database backup used for recovery is only valid if the data it references still exists within the cloud. Though the database backup represents a frozen point in time, production continues to modify, reuse and delete content in the same cloud vault. Through volume version retention, the point-in-time database recovery backup will reference data which continues to exist given the production grid retains such previous versions in the cloud. Therefore, Volume Version Retention is a critical pre-requisite to performing successful cloud exports and restores. The configured durations must be long enough to account for the oldest backup wanting to be restored plus any days needed to complete a recovery test. If a test is initiated and the duration needs to be elongated, the production grid’s retention period can be dynamically extended and existing retained data will retroactively be retained longer to accommodate the extended test. Once the test is completed, the retention period can be lowered and excess retained data will be removed from the cloud.


8.1 Configuring Volume Version Retention Users can configure how long previous stale versions of a logical volume are kept using a new Cloud Pool setting. The retention period can be configured as Retention Days by Cloud Pool on the TS7700 MI Cloud Pools page (Figure 8-1).

Figure 8-1 Cloud Pool Retention Days

Users can set retention days when creating a new cloud pool or by selecting Modify Cloud Pool from the Action menu. To keep previous stale versions of logical volumes, a value greater than 0 must be specified. A retention days value of 0 means not to retain any previous stale logical volume versions (Figure 8-2). After modifying the Cloud Pool settings, a task tracking entry is made. Results of the change can be seen in the Task page under Monitor in the TS7700 MI. Retention is only applicable to stale versions. The most recent version of a logical volume is retained indefinitely. Only when it is replaced with a new or modified version or deleted will the retention period become applicable. The logical volume must be pre-migrated to a cloud storage vault for its stale version to qualify for version retention. Volumes which have not pre-migrated to the cloud prior to an update won’t have that version retained.


Figure 8-2 Modify Cloud Pool

Ensure the storage group setting, for the logical volumes intended to use the cloud pool with Retention Days, is correct by reviewing the TS7700 MI Storage Group page. The Storage Group should be targeting the expected Cloud Premigration Rank 1 cloud pool (Figure 8-3).

Figure 8-3 Storage Group

8.2 Volume Version Retention Consideration and Best Practices The retention days setting for a Cloud Pool allows the ability to retain previous stale versions of logical volumes in a cloud object store. This retention period occurs due to any event which deletes, ejects or replaces a version of a logical volume in the cloud. This includes reuse, modification, ejects and delete-expires. The previous version of the volume can now be held until the cloud pool retention days


have been exceeded even though the most recent logical volume is now deleted, ejected, modified or reused. These expired versions of the volume will be retained for “Retention Days” + 1 days. Figure 8-4 represents how volume version retention works.

Figure 8-4 Cloud Pool Retention Days Example

Figure 8-4, takes us through the life of logical volume VOL00, which is assigned to a cloud pool with a 5-day retention period: Day 0: VOL00 (version 100) was written and pre-migrated to Cloud Pool A. Day 6: VOL00 was updated to version 101 so VOL00 (version 100) became the old version volume. Day 12: “5 (Retention Days) + 1” days passed from day 6 so VOL00 (version100) exceeded the cloud pool retention period and is deleted from Cloud Pool A. Day 14: VOL00 was updated to version 102 so VOL00 (version 101) became the old version volume. Day 20: “5 (Retention Days) + 1” days passed from day 14 so VOL00 (version101) exceeded the cloud pool retention period and is deleted from Cloud Pool A.


Day 22: VOL01 (version 100) was written and pre-migrated to Cloud Pool A. It is a different logical volume than VOL00 so does not affect VOL00 (version 102) was still the active version. As mentioned earlier in this document, the Retentions Days setting in Cloud Pools also affects the database backups that were exported to Cloud Pool by Cloud Export operation. These database backups are kept until they reach the maximum retention days of the export pools listed in the EXPPOOLS field of the Export List file (+ 1 days). This is described in section 5.2 Life Cycle of Database Backups in the Cloud on page 33. The retention days for Cloud Pools used exclusively for database backup targets are ignored. Meaning, if the vault is only to contain a database backup and no logical volumes it’s retention period is not applicable to the retention duration of the database backup. Figure 8-5 further describes how logical volumes and database backups are associated. In this example, the EXPPOOLS specified during the Cloud Export are Cloud Pool “A” and “B” and database backup file are exported to both pools.


Figure 8-5 Cloud Pool Retention Volumes & Database Backups


In Figure 8-5, the following operations occur: Day 0: When the Cloud Export started, VOL00 (version 100) and VOL30 (version 100) are in the pre-migration queue. VOL00 (version 100) was pre-migrated to Cloud Pool A, VOL30 (version 100) was pre-migrated to Cloud Pool B and the Cloud Export exported the database backup to both Cloud Pool A and B. Day 6: VOL00 and VOL30 were updated to version 101 so VOL00 (version 100) and VOL30 (version 100) became older versions of those volumes. Day 12: “5 (Retention Days) + 1” days passed from day 6 so VOL00 (version100) exceeded the cloud pool retention period and is deleted from Cloud Pool A. “11 (maximum retention days of Export Pools) + 1” days passed from day 0 so the database backup files also exceeded the cloud pool retention period and were deleted from both Cloud Pool A and B. Day 14: VOL01 (version 100) was written and pre-migrated to Cloud Pool A. This is a different logical volume than VOL00 so VOL00 (version 101) is still the latest version. Day 16: VOL30 was updated to version 102 so VOL30 (version 101) became an older version of the volume. Day 18: “11 (Retention Days) + 1” days passed from day 6 so VOL30 (version100) exceeded the cloud pool retention period and is deleted from Cloud Pool B. Day 28: “11 (Retention Days) + 1” days passed from day 16 so VOL30 (version101) exceeded the cloud pool retention period and is deleted from Cloud Pool B. When the user schedules a Cloud Export operation, the user should take into account when the database backup will be deleted and when these older versions of logical volumes in the Export Pools specified in the EXPPOOLS list will be deleted. To ensure an adequate recovery point, best practice is to ensure the interval of Cloud Export operations are shorter than “maximum retention days of export pool”. To get a logical volume level summary of the number of versions and a sum of how much space those versions are occupying in the cloud, per cloud pool, use the following LIBRARY REQUEST command:

LVOL, <volser>, CLDINFO The cloud pool is selected based on the Storage Group Cloud Premigration Rank fields. As of microcode level 8.51.x.x, only Cloud Premigration Rank 1 is supported. The Storage Group used for any version of a logical volume can be different for every mount. To view more details about each stale version of a logical volume, users can run the LIBRARY REQUEST CLDVERS command: LVOL, <volser>, CLDVERS, <page index> This request will return information about each version of this logical volume including the version of the data, cloud pool and account, premigration time, the full object name and


other important information. Each LIBRARY REQUEST CLDVERS call will return data for up to 7 version records. If more than 7 logical volume versions exist, use the “page index” to return the next set of 7. Since retaining stale versions of logical volumes means consuming more space in a cloud storage vault, a LIBRARY REQUEST command is also provided as a way to set alerts on capacity consumption: CLDSET, USELOW, <cloud pool name> <TB>

CLDSET, USEHIGH, <cloud pool name> <TB> Each cloud pool can have individual alerts that send an operator message when the TS7700 has detected that the amount of space consumed in the cloud vault for this cloud pool has exceeded the threshold value (in TB) for more than five minutes. When the USELOW threshold is exceeded for more than five minutes for a cloud pool, the TS7700 will send an operator message once. When the condition is cleared (the capacity falls below the USELOW threshold), another operator message is sent indicating the threshold status is now clear. When the USEHIGH threshold is exceeded for more than five minutes for a cloud pool, the TS7700 will send an operator message every 15 minutes until the condition is cleared. These alerts allow an early warning system and then a more urgent warning system so steps can be taken correct the problem. More information about these and other LIBRARY REQUEST commands can be found in the White Paper – TS7700 Library Request Command V5.x.pdf document mentioned previously.

9 Volume Version Restore Microcode 8.51.0.x provides a capability to retain volume versions in a cloud. Beginning with microcode level 8.51.1.x, users can restore volume versions retained in the cloud. The volume version restore function allows the ability to restore an older version of a logical volume into another scratch logical volume. Users can choose any volume version retained in the cloud as a restore source in case the latest version of a volume was corrupted, was maliciously or accidentally modified, or a previous version of the VOLSER was maliciously or accidently deleted. The volume version restore function becomes usable only when all clusters in a grid are with the microcode release level 8.51.1.x or higher. When the volume version restore operation is executed, all clusters in a grid must be online, otherwise it fails. It is not required for all clusters in a grid to have the access to the restore source version in the cloud, but the volume version restore operation needs to be executed from a cluster which has the access to the restore source version.


During the restore processing, the TS7700 will change the VOLSER of the restore source version to that of a restore destination volume. Once the restore process has completed, the restore destination volume is now considered as active data and is the access point to the restored version.

9.1 Overview of Volume Version Restore The volume version restore itself completes by updating the TS7700 database to remap the restore source version to the VOLSER of restore destination volume. The actual data retained in the cloud will not be touched, so the data attributes of the restore source version, such as Logical WORM and compression stats, will be taken over by the restore destination volume. Other volume attributes are inherited from the restore destination volume. Depending on the partition or copy mode assigned to the destination scratch volume, recall or grid copy will be performed to follow the new volume attribute, if necessary, after the restore completed. IBM Z created logical volumes in the TS7700 contain header and trailer information put there by the TS7700 and IBM Z. Some of these headers and trailers reference the volume serial or VOLSER. During a volume version restore procedure, the restore destination volume will simply point to the instance of restore source version in the cloud which still contains the previous volume's VOLSER information within its header and trailers. If the volume is recalled into a TS7700C disk cache, the TS7700C will automatically update these internal headers and trailers during the recall and/or during host access. By deferring the header and trailer updates, the volumes being restored only need to be accessed if a recall is requested and the source version remains immutable in the object store. It also allows many volumes to be restored quickly given no cloud access is required until the volume is actually accessed for host and/or copy activity. If the new volume is modified, the new modified version will be a candidate to premigrate to the cloud and the previous instance is then retained again if the cloud pool has a retention policy configured. Figure 9-1 illustrates the overview of volume version restore procedure.

Figure 9-1 Overview of Volume Version Restore Procedure

9.2 How to Perform a Volume Version Restore

VOL001DL=103

VOL001DL=102

VOL001DL=101

CloudTS7700

TS7700 DBVOL001 (Data Level=101) is restored to ABC001. Volume header is left unchanged.

VOL001

…

ABC001

VOL001DL=103

VOL001DL=102

VOL001DL=101

CloudTS7700

TS7700 DB

VOL001

…

ABC001

ABC001Volume header is overwritten by the new VOLSER

during recall.

Eventually the restored version is recalled to the cache…

a


Volume version restore can be performed by issuing library request commands on a cluster which will drive the restore processing. This section describes how to perform a volume version restore using the library request commands. First, a user needs to reserve one or more logical volumes from an existing scratch pool, which will be used for one or more recovery volumes. This is accomplished using the following LIBRARY REQUEST command: CLDVR, RESERVE, <number>, <category> This command moves the specified number of volumes from the scratch category to FF19 category, where the FF19 category is a special category used for reserving restore destination volumes. The command output shows which volumes were reserved to FF19 category as shown below: CLDVR RESERVE V1 .0

40 SCRATCH VOLUMES SUCCESSFULLY RESERVED

ABC000

ABC001

ABC002

...

ABC039

If only a limited number of scratch volumes were reserved, the command output is as shown below: CLDVR RESERVE V1 .0

ONLY 7 SCRATCH VOLUMES RESERVED

ABC000

ABC001

ABC002

...

ABC007

Please make sure the scratch category referenced matches the category of the system plex that will access the restored volume. After reserving one or more volumes, you must define a destination private category for future volumes you want to restore. This can be done by using the following LIBRARY REQUEST command: CLDVR, DESTCAT, <category> Once a destination category is configured, it will remain configured as long as the TS7700 remains online and does not need to be set for each restore request. If more than one private category is to be used, you will need to update this value prior to any restores that should target a particular private category. If the changed category represents a different system plex, you will also want to reserve scratch volumes from that plexes scratch category using the previous step. If the incorrect scratch category is used by accident or intentionally, then you will need to insert the restored volume VOLSER into the plexes TMS as a foreign tape prior to access.


The "CLDVR, DESTCAT, <category>" command output is as shown below: CLDVR DESTCAT V1 .0

DESTINATION CATEGORY WAS SUCCESSFULLY REGISTERED

Next, we will begin the process of restoring volumes. This process requires two steps per volume. The first step is to choose one of the previously reserved volumes as the destination volume of the restore. This information must be provided for every single restore since the field is cleared once a restore of a volume completes. You assign which volume from the FF19 category you would like to use next using the following command: CLDVR, DESTVOL, <volser> If no restore destination volume is specified, TS7700 automatically choose a restore destination volume from FF19 category in ascending order of the time reserved. The second step of a volume version restore requires you to provide the old volume's VOLSER and its TS7700 based insert and data levels. This informs the TS7700 exactly which version of a retained volume you are wanting to restore from an attached cloud storage. You can use BVIR "CLOUD VOLUME MAP" to list all possible candidates, including the insert version and data level. The BVIR output also includes a timestamp of when the version was created in the cloud making it easier to choose the appropriate version. Keep in mind that the timestamps provided in BVIR are all UTC. You can also use the LIBRARY REQUEST LVOL, <volser>, CLDVERS, <page index> command described in the section 8.2 to list possible versions of a specific retained volume. Refer to the command output to see what insert version and data level the desired version has. The "CLDVR, DESTVOL, <volser>" command output is as shown below: CLDVR DESTVOL V1 .0

DESTINATION VOLUME WAS SUCCESSFULLY REGISTERED

CLDVR, <volser>, <insert version>, <data level> The above command will do four things. First, it will move the DESTVOL volume from the FF19 category to the DESTCAT private category. Next, it will update the TS7700 cloud mapping in the grid so that all instances of DESTVOL now reference the retained volume in the cloud. Next, it will mark the referenced version in the cloud as active data so that retention no longer applies. Last, it will update a subset of the DESTVOL volume's attributes to match those of the original retained version. Attributes inherited from the restore source version are listed here: • last modified time • world ID • write mount count • file size • last data creation time • compression method


• logical volume format ID • wrap counter handling • channel length bytes If the retained version was created by a cluster prior to microcode level 8.51.1.x, the following attributes are recovered using volume header when the restored version is recalled to the cache for the first time. Until the restored version is recalled, BVIR VOLUME STATUS output or Management Interface Virtual Volume Details page may show incorrect values for those attributes. • compression method • logical volume format ID • wrap counter handling • channel length bytes If the retained version was created by a cluster prior to microcode level 8.51.1.x, the following attributes cannot be recovered. • last modified time • write mount count • last data creation time Other attributes are inherited from the restore destination volume. The example of those attributes is listed here: • volume policies • properties level • insert version • data level (set to data level of the restore destination volume + 1) Once the volume version restore has completed, volume ownership of the restore destination volume is moved to the cluster performing the restore. The response lines of the "CLDVR, <volser>, <insert version>, <data level>" command are as shown below: CLOUD LOGICAL VOLUME VERSION RESTORE V1 .0

VOLUME VERSION SUCCESSFULLY RESTORED TO VVVVVV

where VVVVVV is the VOLSER of restore destination volume. If the operation fails, a message with a description of the failure will be displayed. See Appendix C for a full list of potential operator messages that could be sent. After each restore, you must then again assign a new DESTVOL before the next requested recovery. If the next volume's destination category is different, you must also assign a new DESTCAT value. If a specific set of policies are required of the restored volumes, such as copy policies or partition information, you should issue an LMPOLICY command from z/OS to the DESTVOL


prior to issuing the LI REQ restore operation. This way, the restored volume will be assigned the appropriate policies at the time of restore. The above restore process never accesses the retained version in the cloud, allowing you to restore many volumes in a short period of time. The entire volume version restore process only applies updates to the TS7700 internal database. Only when a volume is requested for host activity or grid replication activity will the immutable instance in the cloud be recalled into a TS7700C disk cache. At that time, the volume's internal labels will be updated to match the new volume's VOLSER. If modified, the new modified version will be premigrated to the cloud and the prior restored version will then again be a candidate for retention.


10 Appendix A Copy and Cloud Export messages that may occur using Export List V4:

Message Description Action Needed "E1000 OPENING EXPORT LIST VOLUME XXXXXX

FAILED. NO OPTYPE REC"

This message is generated when opening Export List File Volume fails during copy/cloud export operation. Opening Export List File Volume fails because it does not have OPTYPE record.

Check that Export List Volume has OPTYPE record.

"E1001 OPENING EXPORT LIST VOLUME XXXXXX

FAILED. INVALID OPTYPE" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because invalid optype is specified in OPTYPE record.

Check that "TAPE" or "CLOUD" is specified in OPTYPE record in Export List Volume.


FAILED. NO EXPPOOL REC" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because it does not have EXPPOOLS record.

Check that the Export List Volume has EXPPOOLS record.


FAILED. NO EXPPOOL SPECIFIED" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because no Cloud Pool nickname is specified in EXPPOOLS record.

Check that any Cloud Pool nickname is specified in EXPPOOLS record in Export List Volume.


FAILED. INVALID EXPPOOL" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because any one of Cloud Pool nicknames specified in EXPPOOLS record does not exist in the database (CLOUD_POOL table).

Check that all of the Cloud Pool nicknames specified in EXPPOOLS record in Export List Volume are configured correctly.


FAILED. INVALID DBPOOL" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because any one of Cloud Pool nicknames specified in DBPOOLS record does not exist in the database (CLOUD_POOL table).

Check that all of the Cloud Pool nicknames specified in DBPOOLS record in Export List Volume are configured correctly.


FAILED. NO CLOUD POOL DEFINED IN DB" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File

Check that at least one Cloud Pool is configured correctly.


Volume fails because "*" is specified in EXPPOOLS record to specify all Cloud Pools, but no Cloud Pool record exists in the database (CLOUD_POOL table).


FAILED. TAPE EXPORT CONTAINS EXPPOOL" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because it contains Cloud Pool in Export Pool record even though OPTYPE is TAPE.

Check that the EXPPOOLS record of Export List Volume does not contain any Cloud Pool (contains record header "EXPPOOLS" only) in the case of OPTYPE TAPE.


FAILED. TAPE EXPORT CONTAINS DBPOOL" This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because it contains Cloud Pool in the database pool record even though OPTYPE is TAPE.

Check that the DBPOOLS record of Export List Volume does not contain any Cloud Pool (contains record header "DBPOOLS" only) in the case of OPTYPE TAPE.

"E1009 OPENING EXPORT LIST VOLUME %s FAILED.

EXPORT LIST VOLUME SHOULD NOT BE IN EXPPOOL"

This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because it is configured to be pre-migrated to a Cloud Pool specified by EXPPOOLS record.

Check that the Export List Volume is NOT configured to be pre-migrated to a Cloud Pool specified by EXPPOOLS record.

"E1010 OPENING EXPORT LIST VOLUME %s FAILED.

INTERNAL ERROR OCCURED"

This message is generated when opening Export List File Volume fails during cloud export operation. Opening Export List File Volume fails because of the other reasons.

Check that the Export List Volume format is correct.

"E1015 EXPORT TERMINATED, VEC/VED LOWER THAN

R5.1 EXISTS IN THE GRID"

This message is generated when the Grid does not meet configuration requirement.

Check that all TS7760s (VEC) and TS7770s (VED) int the Grid are at microcode release level 8.51.x.x or higher.

"E1020 EXPORT TERMINATED, CLOUD DEGRADED

DETECTED."

This message is generated when Cloud degradation is detected.

Check that the Cloud environment is healthy.

"E1030 COPYING LOGICAL EXPORT VOLUMES FROM

CACHE TO CLOUD"

None

"E1031 COPYING TO CLOUD <export pool name>

(ID:<export pool id>), CL/CNT/LASTVOL,

0/999999999/ZZZZZZ, 1/999999999/ZZZZZZ,

2/999999999/ZZZZZZ, 3/999999999/ZZZZZZ,

4/999999999/ZZZZZZ, 5/999999999/ZZZZZZ,

6/999999999/ZZZZZZ, 7/999999999/ZZZZZZ"

This message shows progress of the Cloud pre-migrations. CL is cluster id.

None


CNT is pending pre-mig count against the export pool. If pending pre-mig count is invalid, CNT is set as NA and sends either a E1032 or E1033 error code. LASTVOL is the last volume in the cloud pre-mig queue against the export pool.

"E1032 CLUSTER <cluster id> IS ASSOCIATED WITH

<export pool name> (ID:<export pool id>)

CLUSTER UNAVAILABLE."

This message is generated when the cluster is offline or in service.

Check that the cluster is offline or in service.

"E1033 CLUSTER <cluster id> IS ASSOCIATED WITH

<export pool name> (ID:<export pool id>)

PREMIGRATION STATUS UNOBTAINABLE."

This message is generated when Cloud Export failed to get the Premig pending status from the cluster.

Check that the Cloud environment is healthy and Grid network has no problem.

"E1035 CLOUD EXPORT TERMINATED, EXCESSIVE TIME

FOR COPY TO CLOUD"

This message is generated when all of the lvols which were in Cloud Premig queue when Cloud Export started took more than 10 hours to be pre-migrated.

Call for IBM support.

"E1037 CLOUD EXPORT TERMINATED, FAILED TO CHECK

PREMIG STATUS"

This message is generated when Cloud Export failed to check the Premig status.

Check that the Cloud environment is healthy and Grid network has no problem.

"E1039 COPIED LOGICAL EXPORT VOLUMES FROM CACHE

TO CLOUD"

This message is generated when all of the lvols which were in Cloud Premig queue when Cloud Export started have been kuppre-migrated.

None

"E1040 CREATING CLOUD VOLUME MAP REPORT FILE"

This message is generated when Cloud Export starts to generate Cloud Volume Map report.

None

"E1045 CLOUD EXPORT TERMINATED, FAILED TO

CREATE CLOUD VOLUME MAP"

This message is generated when Cloud Export failed to generate Cloud Volume Map report.


"E1047 NO EXPORT VOLUME EXISTS IN EXPPOOLS"

This message is generated when there was no lvol in Cloud Pools specified as EXPPOOLS.

None

"E1049 CREATED CLOUD VOLUME MAP REPORT FILE"

This message is generated when Cloud Export succeeded to generate Cloud Volume Map report.

None

"E1060 WRITING CLOUD VOLUME MAP TO STATUS FILE" None


This message is generated when Cloud Export starts to write Cloud Volume Map report to Export Status file.

"E1065 CLOUD EXPORT TERMINATED, FAILED TO WRITE

CLOUD VOLUME MAP TO STATUS FILE"

This message is generated when Cloud Export failed to write Cloud Volume Map report to Export Status file.

Call for IBM Support.

"E1066 EXPORT SUCCESSFUL BUT CLOUD VOL MAP

TRUNCATED IN STATUS FILE DUE TO LEOT"

This message is generated when Export List volume reached LEOT while Volume Map report is being written. Cloud Volume Map report is truncated in Export Status file.

Verify the logical volume size is large enough to hold the volume map report. If a 25GB logical volume is already used then the number of logical volumes in the cloud exceeds the capacity of the tape. Use the BVIR CLOUD VOLUME MAP command with multiple logical volumes to retrieve the full list.

"E1069 WROTE CLOUD VOLUME MAP TO STATUS FILE"

This message is generated when Cloud Export succeeded to write Cloud Volume Map report to Export Status file.

None

"E1070 WRITING DB BACKUP INFO TO STATUS FILE"

This message is generated when Cloud Export starts to write the database backup list to Export Status file.

None

"E1075 CLOUD EXPORT TERMINATED, FAILED TO WRITE

DB BACKUP INFO TO STATUS FILE"

This message is generated when Cloud Export failed to write the database backup list to Export Status file.


"E1076 EXPORT SUCCESSFUL BUT DB BACKUP INFO

TRUNCATED IN STATUS FILE DUE TO LEOT"

This message is generated when Export List volume reached LEOT while the database backup list is being written. Cloud Volume Map report is truncated in Export Status file.


"E1079 WROTE DB BACKUP INFO TO STATUS FILE"

This message is generated when Cloud Export succeeded to write the database backup list to Export Status file.

None

"E1100 CLOUD EXPORT PROCESSING STARTED" This message is generated when taking a database backup and exporting it to cloud during cloud export operation.

None

"E1110 DB BACKUP PROCESSING STARTED" This message is generated when taking a database backup during cloud export operation.

None

"E1119 DB BACKUP PROCESSING COMPLETED" This message is generated when a database backup was taken during cloud export operation.

None


"E1120 EXPORTING DB BACKUP TO CLOUD POOLS" This message is generated when exporting a database backup to customer specified cloud pools during cloud export operation.

None

"E1121 <cloud pool nickname> (ID:<cloud pool

ID>) CLOUD VAULT IS INACCESSIBLE."

This message is generated when the Cloud Pool is inaccessible.


"E1128 EXPORT TERMINATED, EXPORT ERROR

OCCURRED."

This message is generated when exporting the database backup file to Cloud Pools failed for some reason.


"E1129 EXPORTED DB BACKUP TO CLOUD POOLS" This message is generated when completed exporting database backups to customer specified cloud pools during cloud export operation.

None

"E1190 CLOUD EXPORT CANCELED BY HOST REQUEST."

This message is generated when Cloud Export is canceled by host request.

None

"E1191 CLOUD EXPORT CANCELED BY OFFLINE

PENDING."

This message is generated when Cloud Export is canceled because the cluster enters offline pending state.

Retry the Cloud Export once the TS7700 is back online.

"E1198 EXPORT TERMINATED, INTERNAL ERROR

OCCURRED."

This message is generated when Cloud Export failed because of an unknown reason.


"E1199 CLOUD EXPORT PROCESSING COMPLETED"

This message is generated when completed taking database backup and exporting it to cloud during cloud export operation.

None

Table 7 Cloud Export Status Messages


11 Appendix B

Sample Cloud Export logical volume content:

"VOL1JA0101 "

"HDR10000000000000000000000000000000000000000000000000000000000000000000000000000"

"HDR20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

"EXPORT LIST 04 "

"OPTYPE CLOUD "

"EXPORT PARAMETERS PHYSICAL POOL TO EXPORT:12 "

"OPTIONS1,COPY,LMTDBPVL "

"EXPPOOLS EXPOOL "

"DBPOOLS EXPOOL "

"DESC Backup090920 "

(TM)

"EOF10000000000000000000000000000000000000000000000000000000700000000000000000000"

"EOF20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

"HDR10000000000000000000000000000000000000000000000000000000000000000000000000000"

"HDR20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

"RESERVED FILE "

(TM)

"EOF10000000000000000000000000000000000000000000000000000000100000000000000000000"

"EOF20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

"HDR10000000000000000000000000000000000000000000000000000000000000000000000000000"

"HDR20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

"EXPORT STATUS 02 "

"S/N ,BACKUP_ID,BACKUP_P,BACKUP_POOL_ID ,BACKUP_OBJECT_NAME

,STARTING_CMD_TIMESTAMP ,SOURCE ,S,FILE_SIZE ,DESCR

IPTION ,POOLNLST,EXPORT_POOL_ID_LIST"

"36600,2 ,EXPOOL ,3660020200731151923,DBEXPORT/DBBACKUP.36600.BA006.BA06C.20

200909210826.001 ,2020-09-09-

21.07.39.000000,ZEXPORT ,0,26968064 ,Backup090920,EXPOOL ,3660020200731151923"

"VOLSER,INSERT_VERSION ,DATA_LEVEL ,SYSPLX_N,SYSTEM_N,PRGRAM_N,FILE_SIZE

,CLOUD_ACCOUNT_ID ,CLDACC_N,E_FLG,ENCRYPTION_KEY_LABEL1

,ENCRYPTION_KEY_LABEL2 ,CDFMT,DLRSN,DEL

ETED_TIME ,PREMIGRATED_TIME ,INITR,CLOUD_POOL ,CLDPOL_N,

CPM_R,CMPST,CL2ST,CCMRU,OBJECT_NAME

,CONTAINER_NAME

,RS,EXPIRE_TIME "

"JA0000,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.07.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,41c6/BA006/

BA01C/36600/JA0000/56/101/20200731161850

,contjup


,0 ,1970-01-01-00.00.00.000000"

"JA0001,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.16.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,6659/BA006/

BA01C/36600/JA0001/56/101/20200731161857

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0002,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.18.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,3927/BA006/

BA01C/36600/JA0002/56/101/20200731161903

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0003,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.18.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,10ea/BA006/

BA01C/36600/JA0003/56/101/20200731161909

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0004,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.24.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,346b/BA006/

BA01C/36600/JA0004/56/101/20200731161916

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0005,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.28.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,525b/BA006/

BA01C/36600/JA0005/56/101/20200731161922

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0006,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,1 ,2020-


09-09-20.13.08.000000,2020-07-31-

16.19.40.000000,2 ,3660020200731151923,EXPOOL ,1 ,12544,12544,0 ,49ec/BA006/

BA01C/36600/JA0006/56/101/20200731161929

,contjup

,1 ,2020-09-20-00.00.00.000000"

"JA0007,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.47.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,5459/BA006/

BA01C/36600/JA0007/56/101/20200731161935

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0008,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.48.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,0d0b/BA006/

BA01C/36600/JA0008/56/101/20200731161942

,contjup

,0 ,1970-01-01-00.00.00.000000"

"JA0009,56 ,101 , , , ,1000952

,3660020200731152452,DRIZZT,0 ,NULL

,NULL ,1 ,0 ,1970-

01-01-00.00.00.000000,2020-07-31-

16.19.54.000000,2 ,3660020200731151923,EXPOOL ,1 ,5376 ,5376 ,0 ,3b7c/BA006/

BA01C/36600/JA0009/56/101/20200731161949

,contjup

,0 ,1970-01-01-00.00.00.000000"

(TM)

"EOF10000000000000000000000000000000000000000000000000000001400000000000000000000"

"EOF20000800008000000000000000000000000000000000000000000000000000000000000000000"

(TM)

(TM)

12 Appendix C Messages that may occur during Volume Version Restore:

Message Description "NUMBER OUT OF RANGE (1-40)"

This message is generated if the 3rd keyword of LIBRARY REQUEST "CLDVR RESERVE <number> <category>" command is smaller than 1 or larger than 40.

"NO VOLUME EXISTS IN THE TARGET CATEGORY (CCCC)"


This message is generated if there was no scratch volume in the category (CCCC) specified by the 4th keyword of LIBRARY REQUEST "CLDVR RESERVE <number> <category>" command.

"NO SCRATCH VOLUME RESERVED"

This message is generated if LIBRARY REQUEST "CLDVR RESERVE <number> <category>" command failed to reserve volumes due to an internal error.

"CATEGORY IS NOT SCRATCH"

This message is generated if the 4th keyword of LIBRARY REQUEST "CLDVR RESERVE <number> <category>" command specifies a non-scratch category.

"CATEGORY IS NOT PRIVATE"

This message is generated if the 3rd keyword of LIBRARY REQUEST "CLDVR DESTCAT <category>" command specifies a non-private category.

"VOLUME IS NOT RESERVED"

This message is generated if the 3rd keyword of LIBRARY REQUEST "CLDVR DESTVOL <volser>" command specifies a volume which is not in FF19 category.

"NO AVAILABLE VOLUME FOR RESTORE DESTINATION"

This message is generated if there is no available volume for restore destination in FF19 category when tried to restore a volume version.

"VOLUME NOT FOUND"

This message is generated if the specified restore destination volume did not exist in FF19 category when tried to restore a volume version.

"CANNOT RESTORE THE CURRENT VERSION DATA"

This message is generated if the specified restore source version was the current version when tried to restore a volume version.

"VERSION NOT FOUND"

This message is generated if the specified restore source version did not exist when tried to restore a volume version.

"NOT ALL THE CLUSTERS IN THE GRID ARE ONLINE"

This message is generated if some of the clusters in the grid are not online when tried to restore a volume version.

"RESTORE DESTINATION CATEGORY IS NOT REGISTERD"

This message is generated if no destination category is specified when tried to restore a volume version.

"THE CLUSTER HAS NO ACCESS TO THE CLOUD COPY"

This message is generated if tried to restore a volume version from a cluster which have no access to the version.

"FAILED TO RESTORE A VERSION (DESTVOL:VVVVVV)"

This message is generated if volume version restore to the destination volume VVVVVV failed due to an internal error.

"FAILED TO MAKE VOLUME CONSISTENT (VVVVVV)"


This message is generated if the volume version restore to the restore destination volume VVVVVV was successful but failed to make the volume consistent.

cloud storage tier export, recovery and testing...2021/05/07 · 1.2 cloud export. cloud export...

Documents