Troubleshooting

Symptom:

Error: Failed to connect to SSH host: dial tcp: connection refused

Diagnostic Steps:

1. Verify SSH host is reachable:

   $ ping slurm-head.example.com

2. Check SSH port:

   $ nc -zv slurm-head.example.com 22

3. Test SSH connection manually:

   $ ssh fuzzball-service@slurm-head.example.com

4. Check Orchestrate logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator | grep -i "ssh"

5. Check connection timeout setting:

   $ kubectl get fuzzballorchestrate fuzzball -n fuzzball-system -o yaml | grep connectionTimeout

Solutions:

• Firewall blocking: Ensure firewall allows SSH connections from Orchestrate on Slurm head node

  # firewall-cmd --permanent --add-service=ssh
  
  # firewall-cmd --reload

• Wrong SSH port: Verify sshPort in configuration matches actual SSH port. Check SSH port on head node:

  # netstat -tlnp | grep sshd

• DNS resolution: Use IP address instead of hostname if DNS is unavailable

  sshHost: "192.168.1.100"  # Use IP directly
  

• Connection timeout too short: Increase timeout for high-latency environments

  spec:
    orchestrator:
      provisioner:
        slurm:
          connectionTimeout: 90  # Increase to 90 seconds
  

Symptom:

Error: ssh: handshake failed: ssh: unable to authenticate

Diagnostic Steps:

1. Verify credentials for password auth (enter password manually when prompted):

   $ ssh fuzzball-service@slurm-head.example.com

   Or for key auth:

   $ ssh -i /path/to/key fuzzball-service@slurm-head.example.com

2. Check SSH logs on head node (RHEL/CentOS):

   # tail -f /var/log/secure

   Or on Ubuntu/Debian:

   # tail -f /var/log/auth.log

3. Verify public key is installed:

   $ ssh fuzzball-service@slurm-head.example.com 'cat ~/.ssh/authorized_keys'

Solutions:

• Password expired: Update password on Slurm head node

  # passwd fuzzball-service

• Public key not installed: Copy public key to authorized_keys

  $ ssh-copy-id -i fuzzball-key.pub fuzzball-service@slurm-head.example.com

• Wrong key format: Ensure private key is in PEM format. Convert to PEM if needed:

  $ ssh-keygen -p -m PEM -f fuzzball-key

• Key permissions: Fix file permissions

  $ chmod 600 ~/.ssh/id_rsa
  
  $ chmod 644 ~/.ssh/id_rsa.pub
  
  $ chmod 700 ~/.ssh

Symptom:

Error: host key mismatch for slurm-head.example.com

Diagnostic Steps:

1. Get current host key:

   $ ssh-keyscan -t rsa slurm-head.example.com

2. Compare with configured key:

   $ kubectl get fuzzballorchestrate fuzzball -n fuzzball-system -o yaml | grep sshHostPublicKey

Solutions:

• Update host key in configuration:

  spec:
    orchestrator:
      provisioner:
        slurm:
          sshHostPublicKey: "slurm-head.example.com ssh-rsa AAAAB3NzaC1yc2E..."
  

• Temporary workaround (not recommended for production):

  spec:
    orchestrator:
      provisioner:
        slurm:
          skipHostKeyVerification: true
  

Symptom:

Error: sbatch: command not found

Diagnostic Steps:

1. Check if Slurm commands exist:

   $ ssh fuzzball-service@slurm-head.example.com 'which sbatch'
   
   $ ssh fuzzball-service@slurm-head.example.com 'which squeue'
   
   $ ssh fuzzball-service@slurm-head.example.com 'which scancel'

2. Check user’s PATH:

   $ ssh fuzzball-service@slurm-head.example.com 'echo $PATH'

3. Find Slurm binaries:

   $ ssh fuzzball-service@slurm-head.example.com 'sudo find / -name sbatch 2>/dev/null'

Solutions:

• Set binaryPath in configuration:

  spec:
    orchestrator:
      provisioner:
        slurm:
          binaryPath: "/opt/slurm/bin"
  

• Add to user PATH on Slurm head node, add to ~/.bashrc or ~/.profile:

  export PATH=$PATH:/opt/slurm/bin
  

• Create symlinks (if admin):

  # sudo ln -s /opt/slurm/bin/sbatch /usr/local/bin/sbatch
  
  # sudo ln -s /opt/slurm/bin/squeue /usr/local/bin/squeue
  
  # sudo ln -s /opt/slurm/bin/scancel /usr/local/bin/scancel

Symptom:

Error: sbatch: error: Access denied

Diagnostic Steps:

1. Check user permissions:

   $ ssh fuzzball-service@slurm-head.example.com 'sacctmgr show user fuzzball-service'

2. Verify account access:

   $ ssh fuzzball-service@slurm-head.example.com 'sacctmgr show assoc where user=fuzzball-service'

3. Check partition access:

   $ ssh fuzzball-service@slurm-head.example.com 'sinfo -p compute'

Solutions:

• Add user to Slurm database:

  # sudo sacctmgr add user fuzzball-service account=default

• Grant access to partition:

  # sudo sacctmgr add account default partition=compute

• Check AllowAccounts in slurm.conf (should show AllowAccounts=ALL or include your account):

  # sudo grep "PartitionName=compute" /etc/slurm/slurm.conf

Symptom: Workflow status stuck in PENDING, no error messages

Diagnostic Steps:

1. Check Slurm controller status:

   $ ssh slurm-head 'scontrol ping'

2. Check pending jobs:

   $ ssh slurm-head 'squeue -t PD'

3. Check controller logs:

   $ ssh slurm-head 'sudo tail -f /var/log/slurm/slurmctld.log'

4. Check Orchestrate logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator | grep -A 10 "ProvisionSubstrate"

Solutions:

• Restart Slurm controller (if unresponsive):

  # sudo systemctl restart slurmctld

• Increase timeout in configuration:

  policy:
    timeout:
      execute: "5m"  # Increase if jobs are slow to start
  

• Check resource availability:

  $ ssh slurm-head 'sinfo -Nel'

Symptom:

Error: sbatch: error: Batch job submission failed: Invalid partition name specified

Diagnostic Steps:

1. List available partitions:

   $ ssh slurm-head 'sinfo -s'

2. Check partition in provisioner definition:

   $ fuzzball node provisioner get slurm-compute-small

Solutions:

• Use correct partition name:

  definitions:
    - id: "slurm-compute"
      provisioner: "slurm"
      spec:
        cpu: 4
        mem: "8GiB"
        partition: "compute"  # Must match actual partition name
  

• Remove partition specification to use default:

  definitions:
    - id: "slurm-compute"
      provisioner: "slurm"
      spec:
        cpu: 4
        mem: "8GiB"
        # partition not specified = use default
  

Symptom:

Error: sbatch: error: Batch job submission failed: Invalid qos specification

Diagnostic Steps:

1. List available QoS:

   $ ssh slurm-head 'sacctmgr show qos format=name,priority'

2. Check user’s allowed QoS:

   $ ssh slurm-head 'sacctmgr show assoc where user=fuzzball-service format=qos'

Solutions:

• Use valid QoS:

  spec:
    orchestrator:
      provisioner:
        slurm:
          options:
            qos: "normal"  # Must be a valid QoS
  

• Remove QoS specification to use default:

  spec:
    orchestrator:
      provisioner:
        slurm:
          options:
            # qos not specified = use default
  

Symptom:

Error: fuzzball-substrate: command not found

Diagnostic Steps:

1. Check if Substrate is installed:

   $ ssh slurm-head 'ssh <compute-node> which fuzzball-substrate'

2. Check compute node PATH:

   $ ssh slurm-head 'ssh <compute-node> echo $PATH'

3. Find Substrate binary:

   $ ssh slurm-head 'ssh <compute-node> find / -name fuzzball-substrate 2>/dev/null'

Solutions:

• Install Substrate on all compute nodes. On each compute node:

  # sudo cp fuzzball-substrate /usr/local/bin/
  
  # sudo chmod +x /usr/local/bin/fuzzball-substrate

• Use absolute path in script: Modify Substrate invocation if needed

• Add to PATH for all users. On compute nodes, add to /etc/environment:

  PATH=/usr/local/bin:/usr/bin:/bin
  

Symptom:

Error: sudo: sorry, user fuzzball-service is not allowed to execute '/usr/local/bin/fuzzball-substrate serve' as root

Diagnostic Steps:

1. Check sudo permissions:

   $ ssh slurm-head 'ssh <compute-node> sudo -l'

2. Check sudoers configuration:

   $ ssh slurm-head 'ssh <compute-node> sudo cat /etc/sudoers.d/fuzzball-service'

Solutions:

• Add sudo permission for Substrate. On all compute nodes, create /etc/sudoers.d/fuzzball-service:

  # echo "fuzzball-service ALL=(ALL) NOPASSWD: /usr/local/bin/fuzzball-substrate" | sudo tee /etc/sudoers.d/fuzzball-service
  
  # sudo chmod 440 /etc/sudoers.d/fuzzball-service

• Use wildcards for flexibility. Allow Substrate with any arguments:

  fuzzball-service ALL=(ALL) NOPASSWD: /usr/local/bin/fuzzball-substrate *
  

• Validate sudoers file:

  # sudo visudo -c

Symptom: Slurm job starts but Substrate never connects to Orchestrate

Diagnostic Steps:

1. Check if Substrate process is running:

   $ ssh slurm-head 'ssh <compute-node> ps aux | grep fuzzball-substrate'

2. Check Substrate logs:

   $ ssh slurm-head 'cat slurm-<job-id>.out'

3. Check network connectivity:

   $ ssh slurm-head 'ssh <compute-node> ping <orchestrator-host>'
   
   $ ssh slurm-head 'ssh <compute-node> nc -zv <orchestrator-host> <orchestrator-port>'

4. Check firewall rules on compute node:

   # sudo iptables -L -n -v | grep <orchestrator-port>

Solutions:

• Allow outbound connections from compute nodes. On compute nodes:

  # sudo firewall-cmd --permanent --add-port=<orchestrator-port>/tcp
  
  # sudo firewall-cmd --reload

• Check Orchestrate endpoint in Substrate configuration:

  $ ssh slurm-head 'ssh <compute-node> cat /etc/fuzzball/substrate.yaml'

• Verify DNS resolution:

  $ ssh slurm-head 'ssh <compute-node> nslookup <orchestrator-host>'

• Use IP address if DNS is problematic. In Substrate configuration:

  orchestrator:
    endpoint: "192.168.1.10:8080"  # Use IP instead of hostname
  

Symptom:

Error: No matching provisioner definition found for resource requirements

Diagnostic Steps:

1. Check workflow resource requirements:

   $ cat workflow.yaml | grep -A 5 "resource:"

2. List provisioner definitions:

   $ fuzzball node provisioner list

3. Compare requirements to available resources:

   $ fuzzball node provisioner get slurm-compute-small

Solutions:

• Reduce workflow resource requests:

  resource:
    cpu:
      cores: 4  # Reduced from 64
    memory:
      size: "8GiB"  # Reduced from 512GiB
  

• Create larger provisioner definition:

  definitions:
    - id: "slurm-compute-xlarge"
      provisioner: "slurm"
      spec:
        cpu: 64
        mem: "512GiB"
        partition: "himem"
  

• Use Slurm node features to target appropriate nodes:

  definitions:
    - id: "slurm-himem"
      provisioner: "slurm"
      spec:
        cpu: 32
        mem: "256GiB"
        constraint: "himem"
  

Symptom: Job killed by Slurm with “Out of memory” error

Diagnostic Steps:

1. Check actual memory usage:

   $ ssh slurm-head 'sacct -j <job-id> --format=JobID,MaxRSS,MaxVMSize'

2. Check memory limit:

   $ ssh slurm-head 'scontrol show job <job-id> | grep Mem'

Solutions:

• Increase memory in workflow:

  resource:
    memory:
      size: "32GiB"  # Increased from 16GiB
  

• Create provisioner definition with more memory:

  definitions:
    - id: "slurm-highmem"
      provisioner: "slurm"
      spec:
        cpu: 16
        mem: "128GiB"
        partition: "himem"
  

• Optimize application memory usage: Review application logs and optimize code

Symptom: Jobs running much slower than expected

Diagnostic Steps:

1. Check node load:

   $ ssh slurm-head 'ssh <compute-node> uptime'

2. Check other jobs on node:

   $ ssh slurm-head 'squeue -w <compute-node> -o "%.18i %.9P %.20j %.8u %.2t %.10M"'

3. Check CPU allocation:

   $ ssh slurm-head 'sacct -j <job-id> --format=JobID,AllocCPUS,CPUTime,Elapsed'

Solutions:

• Use CPU affinity:

  resource:
    cpu:
      cores: 16
      affinity: "SOCKET"  # Bind to CPU sockets
  

• Use exclusive node allocation (if needed):

  spec:
    orchestrator:
      provisioner:
        slurm:
          options:
            exclusive: "true"
  

• Request specific node features:

  definitions:
    - id: "slurm-fast"
      provisioner: "slurm"
      spec:
        cpu: 32
        mem: "64GiB"
        constraint: "haswell|broadwell"  # Newer CPUs
  

Symptom: Workflow status remains PENDING, then the workflow fails when the provisioner definition’s registrationDeadline expires

Diagnostic Steps:

1. Check workflow details:

   $ fuzzball workflow get <workflow-id>

2. Check provisioner status:

   $ fuzzball node provisioner list

3. Check Slurm queue:

   $ ssh slurm-head 'squeue -u fuzzball-service'

4. Check orchestrator logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=100

Solutions:

• Check for provisioner definition mismatch: Ensure workflow resources match available definitions

• Wait for Slurm resources: Jobs may be queued waiting for compute nodes, but only until the definition’s registrationDeadline expires (default 15m, measured from sbatch submission). After that, Fuzzball cleans up by signalling the batch job, which only works on a job that is already running: a job still pending in Slurm rejects the signal and stays queued, so cleanup fails, retries a few times, and the workflow is then failed with the job still in the queue. Remove it yourself with scancel. If your Slurm queue routinely makes jobs wait longer than the deadline, raise registrationDeadline – see Node Registration Deadline.

• Cancel and resubmit:

  $ fuzzball workflow cancel <workflow-id>
  
  $ fuzzball workflow submit --file workflow.yaml

Symptom: Dependent jobs start before their dependencies complete

Diagnostic Steps:

1. Check job dependencies in workflow:

   $ cat workflow.yaml | grep -A 5 "depends_on"

2. Check job execution order:

   $ fuzzball workflow jobs <workflow-id>

Solutions:

• Verify dependency specification:

  jobs:
    job-a:
      name: "first-job"
      # ... job definition
  
    job-b:
      name: "second-job"
      depends_on:
        - job-a  # Must match job ID, not name
      # ... job definition
  

• Use consistent job identifiers: Ensure dependency references use correct job IDs

Symptom: Jobs are killed before completion with timeout errors

Diagnostic Steps:

1. Check provisioner TTL setting:

   $ fuzzball node provisioner get slurm-compute-small | grep ttl

2. Check job timeout policy:

   $ cat workflow.yaml | grep -A 3 "timeout:"

3. Check actual job runtime:

   $ ssh slurm-head 'sacct -j <job-id> --format=JobID,Elapsed,Timelimit'

Solutions:

• Increase provisioner TTL:

  definitions:
    - id: "slurm-compute-long"
      provisioner: "slurm"
      ttl: 28800  # Increase to 8 hours
      spec:
        cpu: 8
        mem: "16GiB"
  

• Reduce job timeout to fit within TTL:

  jobs:
    my-job:
      policy:
        timeout:
          execute: "1h"  # Ensure it's less than provisioner TTL
  

Symptom:

Error: Failed to pull container image: authentication required

Diagnostic Steps:

1. Check image exists:

   $ docker pull <image-uri>

2. Check registry credentials:

   $ fuzzball secret list | grep registry

3. Check Substrate image pull logs:

   $ ssh slurm-head 'cat slurm-<job-id>.out | grep -i "pull"'

Solutions:

• Use public images for testing:

  image:
    uri: "docker://alpine:3.16"  # Public image
  

• Configure registry credentials:

  $ fuzzball secret create registry-creds \
    --username <username> \
    --password <password> \
    --server registry.example.com

• Use image pull secrets:

  image:
    uri: "docker://registry.example.com/myapp:v1"
    pullSecret: "registry-creds"
  

Symptom: Long delay between job submission and execution

Diagnostic Steps:

1. Check Slurm queue time:

   $ ssh slurm-head 'sacct -j <job-id> --format=JobID,Submit,Start,Elapsed'

2. Check scheduler logs:

   $ ssh slurm-head 'sudo tail -f /var/log/slurm/slurmctld.log'

3. Check compute node availability:

   $ ssh slurm-head 'sinfo -Nel'

Solutions:

• Use priority QoS:

  spec:
    orchestrator:
      provisioner:
        slurm:
          options:
            qos: "high"
  

• Reserve nodes for Fuzzball. Create reservation:

  $ ssh slurm-head 'sudo scontrol create reservation \
    starttime=now duration=infinite \
    user=fuzzball-service nodes=node[01-10] \
    reservationname=fuzzball'

  Use reservation:

  spec:
    orchestrator:
      provisioner:
        slurm:
          options:
            reservation: "fuzzball"
  

Symptom: Substrate spends long time pulling images

Solutions:

• Use local registry mirror:

  image:
    uri: "docker://local-mirror.example.com/alpine:3.16"
  

• Pre-pull common images on all compute nodes:

  # sudo apptainer pull docker://alpine:3.16
  
  # sudo apptainer pull docker://ubuntu:22.04

• Use smaller base images. Use alpine instead of ubuntu:

  image:
    uri: "docker://alpine:3.16"  # ~5MB instead of ubuntu:22.04 (~77MB)
  

Orchestrator:

# In FuzzballOrchestrate CRD
spec:
  orchestrator:
    logLevel: "debug"

Substrate on compute nodes, edit Substrate config:

# cat <<EOF | sudo tee /etc/fuzzball/substrate.yaml
logging:
  level: debug
EOF

Create a script to gather diagnostic data:

#!/bin/bash
# diagnostic-collect.sh

echo "=== Fuzzball Configuration ==="
kubectl get fuzzballorchestrate fuzzball -n fuzzball-system -o yaml

echo "=== Orchestrate Logs ==="
kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=500

echo "=== Provisioner Definitions ==="
fuzzball node provisioner list

echo "=== Slurm Status ==="
ssh slurm-head 'scontrol show config | head -20'
ssh slurm-head 'sinfo'
ssh slurm-head 'squeue'

echo "=== Recent Slurm Jobs ==="
ssh slurm-head 'sacct -u fuzzball-service --starttime $(date -d "1 hour ago" +%Y-%m-%d) --format=JobID,JobName,State,ExitCode,Elapsed'

echo "=== Compute Node Status ==="
ssh slurm-head 'scontrol show nodes'

echo "=== Recent Workflows ==="
fuzzball workflow list --limit 10

Test Slurm integration interactively:

$ cat >test.sh << EOF
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1

echo "Hostname: $(hostname)"
echo "Date: $(date)"
echo "CPU info:"
lscpu | head -20
echo "Memory info:"
free -h
echo "Substrate check:"
which fuzzball-substrate
fuzzball-substrate --version
EOF

$ ssh slurm-head 'sbatch test.sh'

If you’re unable to resolve an issue:

1. Collect diagnostic information using the script above

2. Review Slurm documentation at https://slurm.schedmd.com

3. Contact CIQ support with:

   • Problem description
   • Steps to reproduce
   • Diagnostic information
   • Configuration files (with sensitive data redacted)
   • Relevant log excerpts

Symptom:

Error: Failed to connect to SSH host: dial tcp: connection refused

Diagnostic Steps:

1. Verify SSH host is reachable:

   $ ping pbs-head.example.com

2. Check SSH port:

   $ nc -zv pbs-head.example.com 22

3. Test SSH connection manually:

   $ ssh fuzzball-service@pbs-head.example.com

4. Check orchestrator logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator | grep -i "ssh"

5. Check connection timeout setting:

   $ kubectl get fuzzballorchestrate fuzzball -n fuzzball-system -o yaml | grep connectionTimeout

Solutions:

• Firewall blocking: Ensure firewall allows SSH connections from Orchestrate on PBS head node

  # firewall-cmd --permanent --add-service=ssh
  
  # firewall-cmd --reload

• Wrong SSH port: Verify sshPort in configuration matches actual SSH port. Check SSH port on head node:

  # netstat -tlnp | grep sshd

• DNS resolution: Use IP address instead of hostname if DNS is unavailable

  sshHost: "192.168.1.100"  # Use IP directly
  

• Connection timeout too short: Increase timeout for high-latency environments

  spec:
    orchestrator:
      provisioner:
        pbs:
          connectionTimeout: 90  # Increase to 90 seconds
  

Symptom:

Error: ssh: handshake failed: ssh: unable to authenticate

Diagnostic Steps:

1. Verify credentials for password auth (enter password manually when prompted):

   $ ssh fuzzball-service@pbs-head.example.com

   Or for key auth:

   $ ssh -i /path/to/key fuzzball-service@pbs-head.example.com

2. Check SSH logs on head node (RHEL/CentOS):

   # tail -f /var/log/secure

   Or on Ubuntu/Debian:

   # tail -f /var/log/auth.log

3. Verify public key is installed:

   $ ssh fuzzball-service@pbs-head.example.com 'cat ~/.ssh/authorized_keys'

Solutions:

• Password expired: Update password on PBS head node

  # passwd fuzzball-service

• Public key not installed: Copy public key to authorized_keys

  $ ssh-copy-id -i fuzzball-key.pub fuzzball-service@pbs-head.example.com

• Wrong key format: Ensure private key is in PEM format. Convert to PEM if needed:

  $ ssh-keygen -p -m PEM -f fuzzball-key

• Key permissions: Fix file permissions

  $ chmod 600 ~/.ssh/id_rsa
  
  $ chmod 644 ~/.ssh/id_rsa.pub
  
  $ chmod 700 ~/.ssh

Symptom:

Error: ssh: handshake failed: ssh: host key mismatch

Diagnostic Steps:

1. Get current host key:

   $ ssh-keyscan pbs-head.example.com

2. Compare with configuration: Check if sshHostPublicKey in FuzzballOrchestrate CRD matches

Solutions:

• Update host key in configuration with output from ssh-keyscan

• Disable host key verification (not recommended for production):

  spec:
    orchestrator:
      provisioner:
        pbs:
          skipHostKeyVerification: true
  

Symptom:

Error: qsub: command not found

Diagnostic Steps:

1. Check if PBS commands exist:

   $ ssh fuzzball-service@pbs-head.example.com 'which qsub'
   
   $ ssh fuzzball-service@pbs-head.example.com 'which qstat'
   
   $ ssh fuzzball-service@pbs-head.example.com 'which qdel'

2. Check user’s PATH:

   $ ssh fuzzball-service@pbs-head.example.com 'echo $PATH'

3. Find PBS binaries:

   $ ssh fuzzball-service@pbs-head.example.com 'sudo find / -name qsub 2>/dev/null'

Solutions:

• Set binaryPath in configuration:

  spec:
    orchestrator:
      provisioner:
        pbs:
          binaryPath: "/opt/pbs/bin"
  

• Add to user PATH on PBS head node, add to ~/.bashrc or ~/.profile:

  export PATH=$PATH:/opt/pbs/bin
  

• Create symlinks (if admin):

  # sudo ln -s /opt/pbs/bin/qsub /usr/local/bin/qsub
  
  # sudo ln -s /opt/pbs/bin/qstat /usr/local/bin/qstat
  
  # sudo ln -s /opt/pbs/bin/qdel /usr/local/bin/qdel

Symptom:

Error: qsub: Unauthorized Request

Diagnostic Steps:

1. Check user permissions:

   $ ssh fuzzball-service@pbs-head.example.com 'qstat -B'

2. Verify queue access:

   $ ssh fuzzball-service@pbs-head.example.com 'qstat -Q'

3. Check PBS server configuration:

   $ ssh fuzzball-service@pbs-head.example.com 'qmgr -c "print server"'

Solutions:

• Add user to PBS allowed users:

  # sudo qmgr -c "set server authorized_users += fuzzball-service@*"

• Grant queue access:

  # sudo qmgr -c "set queue workq enabled = True"
  
  # sudo qmgr -c "set queue workq started = True"

• Check queue ACLs (should show allowed users or be disabled):

  # sudo qmgr -c "print queue workq"

Symptom: Workflow status stuck in PENDING, no error messages

Diagnostic Steps:

1. Check PBS server status:

   $ ssh pbs-head 'qstat -B'

2. Check pending jobs:

   $ ssh pbs-head 'qstat -i'

3. Check PBS server logs:

   $ ssh pbs-head 'sudo tail -f /var/spool/pbs/server_logs/$(date +%Y%m%d)'

4. Check Orchestrate logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator | grep -A 10 "ProvisionSubstrate"

Solutions:

• Restart PBS server (if unresponsive):

  # sudo systemctl restart pbs

• Increase timeout in configuration:

  policy:
    timeout:
      execute: "5m"  # Increase if jobs are slow to start
  

• Check resource availability:

  $ ssh pbs-head 'pbsnodes -a'

Symptom:

Error: qsub: Unknown queue

Diagnostic Steps:

1. List available queues:

   $ ssh pbs-head 'qstat -Q'

2. Check queue in provisioner definition:

   $ fuzzball node provisioner get pbs-workq-small

Solutions:

• Use correct queue name:

  definitions:
    - id: "pbs-compute"
      provisioner: "pbs"
      spec:
        cpu: 4
        mem: "8GiB"
        queue: "workq"  # Must match actual queue name
  

• Remove queue specification to use default:

  definitions:
    - id: "pbs-compute"
      provisioner: "pbs"
      spec:
        cpu: 4
        mem: "8GiB"
        # queue not specified = use default
  

Symptom:

Error: fuzzball-substrate: command not found

Diagnostic Steps:

1. Check if Substrate is installed:

   $ ssh pbs-head 'ssh <compute-node> which fuzzball-substrate'

2. Check compute node PATH:

   $ ssh pbs-head 'ssh <compute-node> echo $PATH'

3. Find Substrate binary:

   $ ssh pbs-head 'ssh <compute-node> find / -name fuzzball-substrate 2>/dev/null'

Solutions:

• Install Substrate on all compute nodes. On each compute node:

  # sudo cp fuzzball-substrate /usr/local/bin/
  
  # sudo chmod +x /usr/local/bin/fuzzball-substrate

• Use absolute path in script: Modify Substrate invocation if needed

• Add to PATH for all users. On compute nodes, add to /etc/environment:

  PATH=/usr/local/bin:/usr/bin:/bin
  

Symptom:

Error: sudo: sorry, user fuzzball-service is not allowed to execute '/usr/local/bin/fuzzball-substrate serve' as root

Diagnostic Steps:

1. Check sudo permissions:

   $ ssh pbs-head 'ssh <compute-node> sudo -l'

2. Check sudoers configuration:

   $ ssh pbs-head 'ssh <compute-node> sudo cat /etc/sudoers.d/fuzzball-service'

Solutions:

• Add sudo permission for Substrate. On all compute nodes, create /etc/sudoers.d/fuzzball-service:

  # echo "fuzzball-service ALL=(ALL) NOPASSWD: /usr/local/bin/fuzzball-substrate" | sudo tee /etc/sudoers.d/fuzzball-service
  
  # sudo chmod 440 /etc/sudoers.d/fuzzball-service

• Use wildcards for flexibility. Allow Substrate with any arguments:

  fuzzball-service ALL=(ALL) NOPASSWD: /usr/local/bin/fuzzball-substrate *
  

• Validate sudoers file:

  # sudo visudo -c

Symptom: PBS job starts but Substrate never connects to Orchestrate

Diagnostic Steps:

1. Check if Substrate process is running:

   $ ssh pbs-head 'ssh <compute-node> ps aux | grep fuzzball-substrate'

2. Check Substrate logs:

   $ ssh pbs-head 'cat pbs-<job-id>.out'

3. Check network connectivity:

   $ ssh pbs-head 'ssh <compute-node> ping <orchestrator-host>'
   
   $ ssh pbs-head 'ssh <compute-node> nc -zv <orchestrator-host> <orchestrator-port>'

4. Check firewall rules on compute node:

   # sudo iptables -L -n -v | grep <orchestrator-port>

Solutions:

• Allow outbound connections from compute nodes. On compute nodes:

  # sudo firewall-cmd --permanent --add-port=<orchestrator-port>/tcp
  
  # sudo firewall-cmd --reload

• Check Orchestrate endpoint in Substrate configuration:

  $ ssh pbs-head 'ssh <compute-node> cat /etc/fuzzball/substrate.yaml'

• Verify DNS resolution:

  $ ssh pbs-head 'ssh <compute-node> nslookup <orchestrator-host>'

• Use IP address if DNS is problematic. In Substrate configuration:

  orchestrator:
    endpoint: "192.168.1.10:8080"  # Use IP instead of hostname
  

Symptom:

Error: No matching provisioner definition found for resource requirements

Diagnostic Steps:

1. Check workflow resource requirements:

   $ cat workflow.yaml | grep -A 5 "resource:"

2. List provisioner definitions:

   $ fuzzball node provisioner list

3. Compare requirements to available resources:

   $ fuzzball node provisioner get pbs-workq-small

Solutions:

• Reduce workflow resource requests:

  resource:
    cpu:
      cores: 4  # Reduced from 64
    memory:
      size: "8GiB"  # Reduced from 512GiB
  

• Create larger provisioner definition:

  definitions:
    - id: "pbs-workq-xlarge"
      provisioner: "pbs"
      spec:
        cpu: 64
        mem: "512GiB"
        queue: "bigmem"
  

• Use PBS node features to target appropriate nodes:

  definitions:
    - id: "pbs-himem"
      provisioner: "pbs"
      spec:
        cpu: 32
        mem: "256GiB"
        select: "mem>256GB"
  

Symptom: Job killed by PBS with “Out of memory” error

Diagnostic Steps:

1. Check actual memory usage:

   $ ssh pbs-head 'qstat -f <job-id> | grep resources_used.mem'

2. Check memory limit:

   $ ssh pbs-head 'qstat -f <job-id> | grep Resource_List.mem'

Solutions:

• Increase memory in workflow:

  resource:
    memory:
      size: "32GiB"  # Increased from 16GiB
  

• Create provisioner definition with more memory:

  definitions:
    - id: "pbs-highmem"
      provisioner: "pbs"
      spec:
        cpu: 16
        mem: "128GiB"
        queue: "bigmem"
  

• Optimize application memory usage: Review application logs and optimize code

Symptom: Jobs running much slower than expected

Diagnostic Steps:

1. Check node load:

   $ ssh pbs-head 'ssh <compute-node> uptime'

2. Check other jobs on node:

   $ ssh pbs-head 'qstat -n | grep <compute-node>'

3. Check CPU allocation:

   $ ssh pbs-head 'qstat -f <job-id> | grep resources_used.cpupercent'

Solutions:

• Use CPU affinity:

  resource:
    cpu:
      cores: 16
      affinity: "SOCKET"  # Bind to CPU sockets
  

• Use exclusive node allocation (if needed):

  definitions:
    - id: "pbs-exclusive"
      provisioner: "pbs"
      exclusive: "job"
      spec:
        cpu: 32
        mem: "64GiB"
        queue: "workq"
  

• Request specific node features:

  definitions:
    - id: "pbs-fast"
      provisioner: "pbs"
      spec:
        cpu: 32
        mem: "64GiB"
        select: "host=node[10-20]"
  

Symptom: Workflow status remains PENDING, then the workflow fails when the provisioner definition’s registrationDeadline expires

Diagnostic Steps:

1. Check workflow details:

   $ fuzzball workflow get <workflow-id>

2. Check provisioner status:

   $ fuzzball node provisioner list

3. Check PBS queue:

   $ ssh pbs-head 'qstat -u fuzzball-service'

4. Check Orchestrate logs:

   $ kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=100

Solutions:

• Check for provisioner definition mismatch: Ensure workflow resources match available definitions

• Wait for PBS resources: Jobs may be queued waiting for compute nodes, but only until the definition’s registrationDeadline expires (default 15m, measured from qsub submission). After that, Fuzzball cleans up by signalling the batch job, which only works on a job that is already running: a job still queued in PBS rejects the signal and stays queued, so cleanup fails, retries a few times, and the workflow is then failed with the job still in the queue. Remove it yourself with qdel. If your PBS queue routinely makes jobs wait longer than the deadline, raise registrationDeadline – see Node Registration Deadline.

• Cancel and resubmit:

  $ fuzzball workflow cancel <workflow-id>
  
  $ fuzzball workflow submit --file workflow.yaml

Symptom: Dependent jobs start before their dependencies complete

Diagnostic Steps:

1. Check job dependencies in workflow:

   $ cat workflow.yaml | grep -A 5 "depends_on"

2. Check job execution order:

   $ fuzzball workflow jobs <workflow-id>

Solutions:

• Verify dependency specification:

  jobs:
    job-a:
      name: "first-job"
      # ... job definition
  
    job-b:
      name: "second-job"
      depends_on:
        - job-a  # Must match job ID, not name
      # ... job definition
  

• Use consistent job identifiers: Ensure dependency references use correct job IDs

Symptom: Jobs are killed before completion with timeout errors

Diagnostic Steps:

1. Check provisioner TTL setting:

   $ fuzzball node provisioner get pbs-workq-small | grep ttl

2. Check job timeout policy:

   $ cat workflow.yaml | grep -A 3 "timeout:"

3. Check actual job runtime:

   $ ssh pbs-head 'qstat -f <job-id> | grep resources_used.walltime'

Solutions:

• Increase provisioner TTL:

  definitions:
    - id: "pbs-workq-long"
      provisioner: "pbs"
      ttl: 28800  # Increase to 8 hours
      spec:
        cpu: 8
        mem: "16GiB"
  

• Reduce job timeout to fit within TTL:

  jobs:
    my-job:
      policy:
        timeout:
          execute: "1h"  # Ensure it's less than provisioner TTL
  

Symptom:

Error: Failed to pull container image: authentication required

Diagnostic Steps:

1. Check image exists:

   $ docker pull <image-uri>

2. Check registry credentials:

   $ fuzzball secret list | grep registry

3. Check Substrate image pull logs:

   $ ssh pbs-head 'cat pbs-<job-id>.out | grep -i "pull"'

Solutions:

• Use public images for testing:

  image:
    uri: "docker://alpine:3.16"  # Public image
  

• Configure registry credentials:

  $ fuzzball secret create registry-creds \
    --username <username> \
    --password <password> \
    --server registry.example.com

• Use image pull secrets:

  image:
    uri: "docker://registry.example.com/myapp:v1"
    pullSecret: "registry-creds"
  

Symptom: Long delay between job submission and execution

Diagnostic Steps:

1. Check PBS queue time:

   $ ssh pbs-head 'qstat -f <job-id> | grep -E "qtime|stime"'

2. Check PBS server logs:

   $ ssh pbs-head 'sudo tail -f /var/spool/pbs/server_logs/$(date +%Y%m%d)'

3. Check compute node availability:

   $ ssh pbs-head 'pbsnodes -a'

Solutions:

• Use priority queues:

  spec:
    orchestrator:
      provisioner:
        pbs:
          defaultQueue: "priority"  # Use higher priority queue
  

• Reserve nodes for Fuzzball. Create PBS reservation:

  $ ssh pbs-head 'sudo pbs_rsub -U fuzzball-service -N fuzzball -l select=10 -R $(date +%s) -E $(date -d "+1 year" +%s)'

  Use reservation:

  spec:
    orchestrator:
      provisioner:
        pbs:
          options:
            reservation: "fuzzball"
  

Symptom: Substrate spends long time pulling images

Solutions:

• Use local registry mirror:

  image:
    uri: "docker://local-mirror.example.com/alpine:3.16"
  

• Pre-pull common images on all compute nodes:

  # sudo apptainer pull docker://alpine:3.16
  
  # sudo apptainer pull docker://ubuntu:22.04

• Use smaller base images. Use alpine instead of ubuntu:

  image:
    uri: "docker://alpine:3.16"  # ~5MB instead of ubuntu:22.04 (~77MB)
  

Orchestrator:

# In FuzzballOrchestrate CRD
spec:
  orchestrator:
    logLevel: "debug"

Substrate on compute nodes, edit Substrate config:

cat <<EOF | sudo tee /etc/fuzzball/substrate.yaml
logging:
  level: debug
EOF

Create a script to gather diagnostic data:

#!/bin/bash
# diagnostic-collect.sh

echo "=== Fuzzball Configuration ==="
kubectl get fuzzballorchestrate fuzzball -n fuzzball-system -o yaml

echo "=== Orchestrate Logs ==="
kubectl logs -n fuzzball-system deployment/fuzzball-orchestrator --tail=500

echo "=== Provisioner Definitions ==="
fuzzball node provisioner list

echo "=== PBS Status ==="
ssh pbs-head 'qstat -B'
ssh pbs-head 'qstat -Q'
ssh pbs-head 'pbsnodes -a'

echo "=== Recent PBS Jobs ==="
ssh pbs-head 'qstat -x -u fuzzball-service'

echo "=== Compute Node Status ==="
ssh pbs-head 'pbsnodes -aSv'

echo "=== Recent Workflows ==="
fuzzball workflow list --limit 10

Test PBS integration interactively:

$ cat <<EOF > test.sh
#!/bin/bash
#PBS -l select=1:ncpus=1

echo "Hostname: $(hostname)"
echo "Date: $(date)"
echo "CPU info:"
lscpu | head -20
echo "Memory info:"
free -h
echo "Substrate check:"
which fuzzball-substrate
fuzzball-substrate --version
EOF

$ ssh pbs-head 'qsub test.sh'

If you’re unable to resolve an issue:

1. Collect diagnostic information using the script above

2. Review PBS documentation at https://support.altair.com/csm?id=altair_product_documentation

3. Contact CIQ support with:

   • Problem description
   • Steps to reproduce
   • Diagnostic information
   • Configuration files (with sensitive data redacted)
   • Relevant log excerpts