Background

    It is often necessary to transfer large amounts of data between systems, whether your to or from your local machine or to or from another cluster or other system. This page gives some suggestions on techniques that can be used, depending on the situation.

    Setup ssh agent

    If you have not done so already, it is recommended to set up SSH Agent with Agent Forwarding to enable quicker authentication, batch transfers (particularly with rclone).

    Transferring to/from your Local machine to a cluster

    Cyberduck
    In most cases setting up Cyberduck is the preferred interface, especially if you are not comfortable with the command line. It supports not only SFTP, but also S3 buckets. See their page on how to configure it to use keys, and this ▶️video for setting up Cyberduck (requires URI SSO to view).
    rclone
    Rclone is a command line utility that supports many types of connections, including SFTP/SSH, Dropbox, and Google Drive. See this page for a complete list. It also supports several operations, the most useful of which is probably rclone sync, which allows copying entire directories in a way that can be resumed if the connection breaks during transfer. For SFTP/SSH, you want to configure it to use the ssh-agent so your password is not stored in the config file. Note that transferring to a remote cluster, such as Brown’s CCV, which requires an extra prompt for two-step verification will not work with Rclone.
    rsync
    rsync is a standard command line utility available on macOS and Linux. It also supports resumable transfers. It relies on SSH to setup the underlying connection.
    sftp/scp
    scp and sftp are standard command line utilities available on macOS and Linux. It is possible to install the ssh command line utilities on Windows as well.

    Remote to remote

    Any of rclone, rsync and sftp/scp should work when transferring between clusters. For rclone, you may need to first run module load rclone, depending on how it was installed on the system. Use which rclone to determine if rclone is an available command.

    Andromeda

    Because Andromeda requires a VPN connection, it is not possible to send data to it or pull data from it from other clusters. You must connect to Andromeda and initiate the transfer from there.

    Unity

    When connecting to Unity from another cluster, if you haven’t setup your ssh to with ForwardAgent=yes as discussed above, you can instead add a key (generating one if necessary) from the remote cluster. You can use cat ~/.ssh/id_rsa.pub to get the contents of the key, and then paste it into the Unity portal. You should then be able to use scp/sftp/rsync to complete the transfer.

    Long transfers

    If your transfer is going to take more than one day to complete, or if you need to be able to disconnect from the remote machine while the transfer continues, it is recommended that you run the transfer inside screen or tmux, both of which allow you to have more than one shell session within the same connection, as well as detach and reattach. Here’s an example of pulling data from Brown’s CCV/Oscar to Andromeda:
    1. Connect to ssh4 (normal interactive sessions are run through ssh3, which would be slowed by the transfer) 
      laptop$ ssh -o forwardagent=yes andromeda_username@ssh4.hac.uri.edu
    2. Start a screen session 
      ssh4$ screen
    3. Star the transfer 
      ssh4$ rsync -avp --log-file=transfer.log ccv_username@ssh.ccv.brown.edu:/path/on/Oscar/ /data/group/brownproject/
    4. Disconnect from the screen session, allowing to to continue in the background Ctrl+a,Ctrl+d
    5. You can reattach to the screen session later with 
      ssh4$ screen -DR

    Examples

    Setting up rclone for sftp

    $ rclone config
e/n/d/r/c/s/q> n
name > unity
storage > sftp
host > unity.uri.edu
user > use your username from Unity
port > Enter
SSH password y/g/n> n
key_file> Enter
The passphrase to decrypt the PEM-encoded private key file. y/g/n> n
key_use_agent> true
use_insecure_cipher> Enter
disable_hashcheck> Enter
Edit advanced config? (y/n) y/n> n
y/e/d> y
e/n/d/r/c/s/q> q

    See Also

    For transferring to Unity, check this document on Uploading Files