This page will provide you with information on how to use a Command line script function to upload and download data from the UNSW Data Archive. It can also be used to migrate data online before it can be downloaded. The Command line script is for users with a technical background and with large volumes of data. The Command line script allows automation, easier repeatability, and control.

This page contains the following elements:

 

Preparing to use the Data Archive

Before you start uploading your research data to the Data Archive, make sure it is organised and in alignment with UNSW recommendations and, that you know the level of access you have to the Collections inside the Data Archive. Read the Preparing data for upload and the RDMP and the Data Archive pages for more information. To access the data archive you will need:

 

Should I use the Command line script?

The UNSW Data Archive Web application provides access to an interface enabling you to upload, download, locate, and review your research data. The Data Archive Web application is suitable for researchers who wish to search the Data Archive and to either upload or download less than 50 gigabytes (GB) of data and/or less then 1000 files (during a single session).

Researchers who frequently upload or download data, and/or generate datasets, may wish to schedule a regular upload or download of data using the Command Line Script or use SFTP rather than manually uploading and downloading data via the Web application.

We have also provided basic scripts for obtaining Checksums for data on the archive, and for migrating data online so it can be downloaded.

Note: The script can be used for automation of large, manual, uploads of data to the store.

  • Researchers who upload or download more than 50 GB of data and/or more than 1000 files, at a time, please see the SFTP Client guide.
  • Researchers who wish to schedule a regular upload or download of data, please follow the steps set out below.

 

System/setup requirements

To upload and download research data using the Command line script you will need:

  • The latest version of Java.
  • The source and destination paths of the data you wish to upload and/or download.

The command-line script is available on the Github here, you will need to link an account to the UNSW's Github organisation to access this.

Alternatively you can contact the UNSW IT Service Centre (9385 1333) to have Java installed, to obtain the script, or if you need further assistance. 

If you are running the script regularly you can contact the servicedesk to request a UNSW Data Archive Token be generated, which replaces your zPass in the script.

 

Help and further information: