- Add trainer.kubeflow.org/trainjob-ancestor-step label to replicatedJob
template so Trainer controller injects PET_* env vars
- Change TrainJob command from python to torchrun so PET_* env vars
are used for distributed training setup
- Remove command from Runtime (keep it in TrainJob only)
Verified: 2-node 16-GPU distributed training running successfully
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Kubeflow Trainer torch runtime auto-injects torchrun only when
command is not set in the container spec. Training script is now
specified at TrainJob level via trainer.command instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add IB CNI annotation (hostdevice-net), /dev/shm, /dev/infiniband mounts
- Add nvidia.com/hostdev: 2 resource for IB NICs
- Add privileged securityContext with IPC_LOCK for NCCL
- Add NCCL env vars (NCCL_DEBUG, NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME)
- Set image to nvcr.io/nvidia/pytorch:24.10-py3
- Set numNodes: 2 for multi-node distributed training
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- README: add Queue/ClusterTrainingRuntime/TrainJob explanations with key options
- README: note single-node testing before multi-node
- Integration: set nvidia.com/gpu to 8 per node, Queue capacity to 24 (3 nodes)
- Integration: TrainJob example set to numNodes: 1 for install testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Volcano: set default_ns.nodegroup: nd for all components
- Kubeflow Trainer: set manager.nodeSelector.nodegroup: nd
- JobSet: set controller.nodeSelector.nodegroup: nd
- Fix PodGroup priorityClassName: normal-priority -> training-priority
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
selee