Using cloud storage
In Totara you can now use cloud storage with your site. The main use cases for cloud storage are:
- Regular, scheduled backups to keep copies of your files online
- Limiting the size of shared dataroot in web clusters
- Sharing content between different environments (e.g. staging and production sites)
The production server in the local hosting facility uses the regular filedir in dataroot. Whenever new content is uploaded to your Totara site a copy of each file is sent to the S3 cloud store. Content is never deleted from the S3 cloud store.
Sharing of filedir in web cluster via cloud
Site is configured to use a cluster of web servers in order to improve scalability and performance. Each node is using local filedir as a fast cache and the shared cloud store is updated after all changes in any cluster node. When adding a new node local filedir starts empty and is gradually filled during file access.
Sharing of file contents between different environments
Any cloud file content store can be configured to be read-only and used for restoring of missing content files only. This eliminates the need to make a full copy of local filedir when testing upgrades or staging production server changes.
Offloading file contents
Some operating system file systems allow tracking of the last access time for each file. If there is a full backup in the cloud, then the administrator can create a script that deletes large files that have not been accessed recently. If the missing files are accessed then the contents are automatically downloaded from the cloud store.
Setting up cloud storage
Follow these steps to set up cloud storage for your site:
- Install PHP libraries for cloud providers.
- Add settings to config.php.
- Upgrade or install site if necessary.
- Push existing content files to cloud storage using ```totara/cloudfiledir/cli/store.php --push```
Installing PHP libraries
Before using this plugin you must install optional libraries via composer in /libraries/directory.
Cloud store options
All configuration is stored directly in config.php files in the $CFG->totara_cloudfiledir_stores array. The administration interface can be used to review a list of configured stores.
The internal identifier of the store.
Either s3 or azure depending on supported cloud API.
Name of the bucket (or container) in the cloud.
Provider-specific connection options.
Setting this to True means new file contents will be added to store.
Setting this to True means file contents will be deleted from store when not used locally any more.
Setting this to True means use the store contents to recover contents that are missing in local filedir.
Enable/disable switch for all store operations.
The maximum size of files that will be uploaded immediately to the cloud. Files larger than this value will be uploaded later via cron task or CLI script.
Here is an example of these settings once they have been configured:
Cloud storage options
The Amazon S3 API is supported by the vast majority of cloud storage solutions. Totara supports arbitrary connection options.
Supported connection options are described in Amazon's SDK documentation:
Here are some alternative storage solutions that should be compatible with this plugin:
- MinIO - https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html
- OpenStack Swift - https://docs.openstack.org/mitaka/config-reference/object-storage/configure-s3.html
- Google Storage - https://cloud.google.com/storage/docs/migrating#migration-simple
- OpenIO - https://docs.openio.io/latest/source/integrations/cookbook_nextcloud.html
Azure Blob Storage
Microsoft Azure Blob Storage is a notable exception because it does not support the S3 API de facto standard. Connection configuration options are described at https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string.
Custom store providers
Support for different API can be easily implemented in a small custom PHP class extending \totara_cloudfiledir\local\provider\base, see \totara_cloudfiledir\local\provider\s3 and \totara_cloudfiledir\local\provider\azure classes.
Cron sync task
If any store has a maxinstantuploadsize value set, then the system administrator should enable push task in scheduled tasks settings.
CLI script for maintenance of cloud filedir
The command line script /totara/cloudfiledir/cli/store.php can be used to do the following:
- List all available stores
- Fetch list of content files when connecting a new cloud store with existing content
- push local file contents to newly connected empty cloud store or upload missing contents to existing cloud store
- Reset flags used for skipping of invalid local content
- Print list of problems related to cloud store
CLI script for maintenance of local filedir
There is a new CLI script for maintenance of local filedir directory, see /admin/cli/check_filedir.php.
These are the expected use cases for this script:
- Checking consistency of all content files in local filedir
- Restoring missing content files from external stores
- Deleting orphaned files from local filedir