Volcano Job Plugin -- SSH User Guide

Background

SSH Plugin is designed for the login without password for pods within a volcano job , which is necessary for workloads such as MPI. It often works with SVC plugin.

Key Points

• If ssh-key-file-path is configured, please ensure the private and public keys exist under the target directory. Suggest keeping default value in most scenarios.
• If ssh-private-key or ssh-public-key is configured, please ensure the value is correct. Suggest keeping the default keys in most scenarios.
• Once SSH plugin is configured, a secret whose name joins the job name and -ssh will be created, which contains authorized_keys/id_rsa/config and id_rsa.pub. It will be mounted to the given path as a volume for all containers (including initContainers) within the job.
• You can get all the hostnames within the job in /root/.ssh/config by default. This file contains the pairs of hostname and subdomain.
• If SSH plugin is configured, you can sign in any other pods in the same job by ssh hostname without password.

Arguments

Note:

• DEFAULT_PRIVATE_KEY and DEFAULT_PUBLIC_KEY are not fully listed for they are too long. Please refer to the examples behind for cases.
• Volcano is not responsible for the validation of ssh-key-file-path. So please guarantee it correct yourself.
• Suggest keeping blank and making use of the default values in most scenarios just like the example behind. If that, Volcano will help generate a pair of keys and finish all the configuration by default.

Examples

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: mpi-job
spec:
  minAvailable: 3
  schedulerName: volcano
  plugins:
    ssh: []   ## SSH plugin register
    svc: []
  tasks:
    - replicas: 1
      name: mpimaster
      template:
        spec:
          containers:
            - command:
                - /bin/bash
                - -c
                - |
                  mkdir -p /var/run/sshd; /usr/sbin/sshd;
                  MPI_HOST=`cat /etc/volcano/mpiworker.host | tr "\n" ","`;
                  sleep 10;
                  mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 --prefix /usr/local/openmpi-3.1.5 python /tmp/gpu-test.py;
                  sleep 3600;
              image: lyd911/mindspore-gpu-example:0.2.0
              name: mpimaster
              ports:
                - containerPort: 22
                  name: mpijob-port
              workingDir: /home
          restartPolicy: OnFailure
    - replicas: 2
      name: mpiworker
      template:
        spec:
          containers:
            - command:
                - /bin/bash
                - -c
                - |
                  mkdir -p /var/run/sshd; /usr/sbin/sshd -D; 
              image: lyd911/mindspore-gpu-example:0.2.0
              name: mpiworker
              resources:
                limits:
                  nvidia.com/gpu: "1"
              ports:
                - containerPort: 22
                  name: mpijob-port
              workingDir: /home
          restartPolicy: OnFailure

Note:

• This example will create a mpi job with 1 master and 2 workers.

• Because the SVC plugin is enabled, you can get all the hosts in any pod by environment variables. Also, you can get the hosts in /root/.ssh/config if you use the default ssh configuration.

  [root@mpi-job-master-0 /]# cat /root/.ssh/config
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  Host mpi-job-mpimaster-0
  HostName mpi-job-mpimaster-0.mpi-job
  Host mpi-job-mpiworker-0
  HostName mpi-job-mpiworker-0.mpi-job
  Host mpi-job-mpiworker-1
  HostName mpi-job-mpiworker-1.mpi-job
  

• You can sign in other hosts in master pod as follows.

  [root@mpi-job-master-0 /]# ssh mpi-job-mpiworker-0
  Warning: Permanently added 'mpi-job-mpiworker-0.mpi-job,X.X.X.X' (ECDSA) to the list of known hosts.
  Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 3.10.0-1160.36.2.el7.x86_64 x86_64)
  
  * Documentation:  https://help.ubuntu.com
  * Management:     https://landscape.canonical.com
  * Support:        https://ubuntu.com/advantage
  
  This system has been minimized by removing packages and content that are
  not required on a system that users do not log into.
  
  To restore this content, you can run the 'unminimize' command.
  Last login: Thu Apr 14 07:19:05 2022 from 10.244.0.67
  root@mpi-job-mpiworker-0:~# 
  

Note

• Please ensure sshd service is available in all containers.