What is File Store?

Introduction

File Store manages the storage of original and archival files for Archive Item records in the archive. It integrates with an external Amazon Web Services (AWS) S3 storage bucket to securely facilitate the file transfer and file management activities within the administrative workflow of the archive.

Each Archive Item record is allocated a folder where an unlimited number of files and subfolders can be uploaded, stored and retrieved. Ingested media is automatically transferred to File Store when added to the system. Other file management tasks, such as upload, renaming, deleting, and downloading files, are carried out using the archive’s File Store browser or the File Store manager inside the Administration control panel.

Within the archive, access to stored files is managed through the user account permission system. This allows complete control over who can view files, edit the contents of a container, and download files. The viewing, editing, and exporting of File Store files are also protected by the content restrictions applied to a record.

The design and implementation of File Store is supportive of an ‘on-site off-site’ backup strategy, where files are backed up in two or more geographically separate locations. With File Store, the archival backup would be stored off-site in the Cloud on AWS’s S3 service and on-site at your office as you would have already been doing. Furthermore, managed storage devices, such as Network Attached Storage (NAS) systems, provide tools for automatically syncing data between the device and S3. In some cases, this makes it possible to load archival files onto the NAS device which are in turn copied to S3 where they become available in File Store.

The files managed by File Store won’t be stored on Keeping Culture’s AWS account. Instead, your organisation would need to provide, and pay for, an AWS account where the archive backups can be stored. See Storage and bandwidth costs below. However, using File Store is optional, so there is no obligation to bare this extra expense.

Storage and bandwidth costs

There is no cost to setup and use File Store with the archive, but the data stored on the AWS account use with File Store will incur a monthly cost.

AWS use a ‘pay for what you use’ model, priced in US dollars. Therefore, the monthly on-going costs will vary depending on the archive’s usage and fluctuations in the exchange rate.

At the time of writing (July 2021), the key expenses with using the S3 service at AWS’s Sydney datacentre are:

• Storage (Standard class): US$0.025 per GB per month.
• Download bandwidth (egress): US$0.114

Upload bandwidth (ingress) is free. AWS also charge each time a ‘request’ is made for a file on S3. Over the course of a month, these request changes might add up to one or two US dollars.

For example, an archive with 2,200 GB (or 2.2TB) of archival quality files will expect to be charged US$55.00 per month for storage. If the users of the archive, then downloaded 50 GB of files in the month, either directly from the archive or via a backup synchronisation process, data download would cost US$5.70.

The total cost of this example would be approximately US$62 per month once adding in a small amount for request fees.

The S3 service has cheaper storage classes for files that are accessed infrequently. The Standard-IA class costs US$0.019 per GB per month. This would make the example’s 2,200 GB storage cost US$41.80. However, requests are charged at a significantly higher rate. Consequently, using the Standard-IA storage class may only make sense in certain situations.

Initially the S3 bucket is configured with the Standard storage class. It is then possible to change to the Standard-IA class once confirming that there are economic advantages in making this change.

How you choose to work with S3 service can save you money, particularly if the situation where data is synced between S3 and the local on-site storage device. Data uploaded into S3 does not incur a charge, but data downloaded will incur a US$0.114 per GB expense. Therefore, syncing data from the local backup to the S3 backup is almost free, whereas syncing from the S3 backup to the local backup will incur a download fee.

A strategy of populating the local backup with the archive originals is not only cheaper but more practical as well.

File Store permissions

The following User Group permissions have been added to the archive under the ‘File Store’ tab on the Group Permissions page:

• Access File Store Browser: Allows a User to access the File Store browser.
• Show File Store in Metadata: Allows a User to view the File Store items in the metadata column of the Item View screen.
• Edit File Store Items: Allows a User to upload, rename and delete files and folders in the File Store browser.
• Allow Archive File Download: Allows a User to download ‘archive-prepared’ files in the File Store browser.
• Allow Submission File Download: Allows a User to download submission files in the File Store browser.

To display the File Store browser and File Store manager the user account must have Administration access and Administer File Store permissions enabled.

To display the File Store metadata control in the ‘Item’ tab of the View page, the User account must have Administer File Store and Access File Store in metadata permissions enabled.

How to setup File Store

The File Store setup instructions are unique for each archive. To access these instructions:

1. Login to the archive using the Super Administrator account.
2. Open the main menu, select the Administrator
3. Locate and click File Store browser.
4. The instructions are displayed on the File Store browser page prior to being setup.