Solution:
Vault Servers Disaster Recovery solution consists of creating a backup from the Autodesk Data Management Server (ADMS) Console and restoring it.
Back Up or Restore a Vault
It is possible to backup and restore a Vault environment without using the ADMS Console. You can find more information in this "White Paper".
Back up and Restore Vault with Third-Party Tools
If choosing the 3rd party process, it is the responsibility of the Administrator to have considered every possible disaster scenario and documented the steps needed to recover from each one. If no such plans have been documented, the likelihood of data loss increases. Autodesk Support will help where they can, but success cannot be guaranteed.
The Vault Administrator must consider how often they want a backup and set it up. (Back Up or Restore a Vault)
The ADMS Console backup consists of a SQL backup of the databases used, and files located in the Filestore.
If a serious problem with either the databases or files in the filestore is suspected or detected, Autodesk will recommend the Last Known Good (LKG) or "restorable" ADMS Consoles backup, is restored. Existing databases and files will be overwritten resulting in limited data loss. The amount of data loss will depend on, how long ago the LKG backup was created and how much data has changed since then.
Ransomware and other disasters will oftentimes render unusable all data at a site. For this reason it's always a good idea to keep at least one backup off-site.
ADMS Console backups do not happen automatically after installation. Backups need to be configured. There are some decisions that must be made by an intelligent human. For example:-
- Time of Backup. When is the best time?
- Backup frequency. Every weekday?
- Backup location. What is the backup destination?
- Full Backup or Incremental? (Incremental Backups are only supported with Vault Professional).
There are other aspects to recovering from a disaster that are not covered by the Vault Server software.
The Vault Server software does not cover:-
- The detailed steps needed to undertake, in each environment, if the server machine is destroyed.
- The detailed steps needed to undertake, in each environment, if the Vault Server administrators is unavailable.
- The time taken to copy and restore large amounts of data in each environment,.
- The cost, to a business, per hour, while the vaulted data is unavailable.
- The cost of replacing hardware and software licenses.
- Non-default configurations of third party components critical to Vaults functionality. (Windows, SQL, IIS for example).
A forward looking and conscientious Vault Administrator would
- Identify a worst case scenario,
- Plan for the worst case.
- Document the plan.
The documented plan should be detailed enough so that a colleague who is not familiar with the technology can follow it in your absence.
Server environments change and so it is important that the DRP is always kept up to date.
What if the LKG ADMS Console backup was taken a long time ago?
The frequency of the backups and their retention is a decision made by the administrator and the owner of the data. If the last backup was taken a long time ago. An option might be to restore it, then ask the users to scan their local workspaces for files and check them into the newly restored Vault.
What if you have been regularly taking Vault backups, but only identified database corruption that you later find has been present for some time? What options can you try?
- Run the SQL Maintenance Plan. This can sometimes fix issues. (SQL Maintenance Plan)
- Report the issue to Autodesk Support. They can run some healing scripts that might help. (Contact Support)
- Restore the LKG to the server and recreate the missing data.
- Alternatively, follow the steps in "What if there is no ADMS Console backup at all?"
What if there is no ADMS Console backup at all?
If it is still possible to access the Vault data in the old server, use "Vault Mirror" or the Vault clients "Get" feature to download the latest version of each file to the local workspace and check it into the new Vault.
What if there is no ADMS Console Backup, and it's not possible to access the old server?
Make no mistake, the options available to you are running out. Install a new Vault Server and ask the users of the Vault to scan their local workspaces for any files that they can check into the Vault. The Vault
Autoloader and
Inventor 'Check in Project' commands can be used for bulk loading data.
What if there is no ADMS Console Backup, it's not possible to access the old server, and the users have all deleted the data in their local workspaces?
Start a new Vault Server and recreate the lost data.
SQL Replication - What if the Publisher was destroyed and an ADMS Console backup exists.
- Install a new Vault Server,
- Restore the ADMS Console Backup.
- Replicate that database out to the Subscribers. This will overwrite any data they retain.
(In some organizations, it might be possible to create Subscribers and retain the old Subscribers, for data recovery purposes.)
SQL Replication - Can a Subscriber be promoted to be a Publisher if the Publisher is destroyed?
There is no supported way that can promote a Subscriber to a Publisher.
SQL Replication - What if the Publisher was destroyed with no backup, leaving only subscribers?
In that situation, the high level steps would be to:-
- Take a SQL Backup of the Vault Databases on the subscriber with the most data.
- Copy the Filestores from all Vault Servers connected to the SQL Subscriber and merge them.
- Restore the databases on a replacement SQL\Vault Server environment.
- When restoring the databases, verify that the WITH KEEP_REPLICATION option is not ticked.
- Once the databases are all attached, detach them from the SQL instance.
- Start the Autodesk Data Management Server Console.
- Attach the Master (KnowledgeVaultMaster) Vault.
- Before attaching the remaining databases, delete any records of them in the ADMS Console user interface.
- Attach the Vault and Library databases.
- Enable the databases and filestores.
- Replicate the re-attached databases to the subscribers.
Note: It's possible and probable that some degree of data loss will occur. Autodesk Support are limited in the amount of support that can be offered in this scenario.
SQL Replication - What if a Subscribers database became corrupt, but users could still log in and add data?
Users should be told to stop adding new data immediately. There is a high chance that they could lose their data unless they have a local backup of the files. This can be achieved by stopping the Vault Servers web service.
In this situation, the only solution supported by Vault would be to either reinitialize the subscription to the subscriber, or unreplicate affected databases and re-replicate them. This could lead to loss of data stored n the subscriber. To minimize the chances of data loss, it would be necessary to take an emergency backup of the subscribers (Server A) databases and filestores and restore them to a new, independent Vault Server (Server B). Once the database from Server A, has been restored to Server B, Server A can then have its replication disabled and re-enabled. This would be safe in the knowledge that, data unreplicated to the Publisher, still exists on Server B. Server B can then be used to retrieve files from that can then be uploaded to Server A, once the subscription is working again.
What if files are missing from the restored filestore?
Each file will have multiple versions. Some file versions may be missing, post-recovery. In that situation Vault's Filestore Validation background process will scan the filestore and record the missing files in the AVFSConsole log file for that server. The Vault Administrator can then decide the following options:-
- Accept some file versions are missing from the Vault and the message in the AVFS Console log will persist in future filestore validations.
- Get the latest version of the file from the Vault. Delete the file from the Vault and re-add it from Inventor of Autoloader.
For legal reasons, Autodesk cannot make changes to the Vault database to "trick" the software into ignoring or skipping missing file versions. Manually updating the database would inevitably lead to data mismatches that would result in production mistakes that Autodesk cannot accept liability for.
Is it possible to manually edit the SQL database to fix corruption?
