Research Platform Services Wiki

If you're a researcher, we'll help you do stuff.

User Tools

Site Tools


Sidebar

data_management:mediaflux:howto:downloaddata_with_spartan

Spartan HPC Users Mediaflux Tools

Tools on Spartan itself

There are some Mediaflux tools available via the module system on spartan.

Access Module

To access, utilise the module system (see https://dashboard.hpc.unimelb.edu.au/software/ for details on Spartan's the module system). On the Spartan login node, issue this module command (interactively or in your .bashrc file)

% module load unimelb-mf-clients

The Configuration File

The tools require that you set up a Configuration File specifying the details of the Mediaflux server and your account.

Tools

  • unimelb-mf-upload - upload data to Mediaflux
  • unimelb-mf-download - download data from Mediaflux
  • unimelb-mf-check compare/check data (between local file system and Mediaflux) -
  • mexplorer - Interactive drag and drop tool to upload/download/share/query data
    • This is just a shell wrapper for the standard Arcitecta-supplied Explorer jar file
    • It does not use a configuration file, all server details are entered into the GUI

Please note that the upload and download Java clients that we provide have the ability to run multiple parallel threads (one per file). This can speed up data movement, provided there is sufficient bandwidth. Remember that you are working in a shared environment, and the resources are not infinite. Therefore we ask that you don't use more than 4 threads. You may need to do some testing and see if you get any performance benefit from additional threads.

data_management/mediaflux/howto/downloaddata_with_spartan.txt · Last modified: 2020/02/06 13:56 by Neil Killeen