Research Platform Services Wiki

If you're a researcher, we'll help you do stuff.

User Tools

Site Tools


Sidebar

data_management:mediaflux:howto:downloaddata_with_spartan

Spartan HPC Users Mediaflux Tools

Tools on Spartan itself

There are some Mediaflux tools available via the module system on spartan.

Access Module

To access, utilise the module system (see https://dashboard.hpc.unimelb.edu.au/software/ for details on Spartan's the module system). On the Spartan login node, issue this module command (interactively or in your .bashrc file)

% module load unimelb-mf-clients

The Configuration File

The tools require that you set up a Configuration File specifying the details of the Mediaflux server and your account.

Tools

  • unimelb-mf-upload - upload data to Mediaflux
  • unimelb-mf-download - download data from Mediaflux
  • unimelb-mf-check compare/check data (between local file system and Mediaflux) -
  • mexplorer - Interactive drag and drop tool to upload/download/share/query data
    • This is just a shell wrapper for the standard Arcitecta-supplied Explorer jar file
    • It does not use a configuration file, all server details are entered into the GUI

Please note that the upload and download Java clients that we provide have the ability to run multiple parallel threads (one per file). This can speed up data movement, provided there is sufficient bandwidth. Remember that you are working in a shared environment, and the resources are not infinite. Therefore we ask that you don't use more than 4 threads. You may need to do some testing and see if you get any performance benefit from additional threads.

Other Tools Relevant to Spartan

  • Sinks
    • These are Mediaflux constructs that define an end point that you can send data to from Mediaflux. We typically have sftp and scp sinks defined. With sinks, data can be transformed on the fly (e.g. in Neuroscience we often transform DICOM to some other format as the data are sent to a sink)
    • For big data, the restartable and parallel-transferral upload/download clients (above) are a better choice than the sink.
    • The Mediaflux Explorer has the ability to send data to sinks (see the video in the Mediaflux Explorer how to videos
    • For Spartan, the Mediaflux server mediaflux.researchsoftware.unimelb.edu.au has a pre-defined sink for Spartan called scp2spartan that you can use to stream your data to Spartan
    • You need to supply your Spartan credential (not logged) to the GUI of the Explorer and path to send the data to
data_management/mediaflux/howto/downloaddata_with_spartan.txt · Last modified: 2019/10/17 16:03 by Robert Hutton