selee
/
kubeflow_trainer_with_volcano
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6 Commits 1 Branch 0 Tags 531 KiB
Commit Graph

6 Commits

Author SHA1 Message Date
selee f62425843b Update integration config for InfiniBand GPU environment 
- Add IB CNI annotation (hostdevice-net), /dev/shm, /dev/infiniband mounts
- Add nvidia.com/hostdev: 2 resource for IB NICs
- Add privileged securityContext with IPC_LOCK for NCCL
- Add NCCL env vars (NCCL_DEBUG, NCCL_IB_DISABLE=0, NCCL_SOCKET_IFNAME)
- Set image to nvcr.io/nvidia/pytorch:24.10-py3
- Set numNodes: 2 for multi-node distributed training

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 15:57:03 +09:00
selee 6b72590351 Add resource descriptions to README, set GPU to 8 per node 
- README: add Queue/ClusterTrainingRuntime/TrainJob explanations with key options
- README: note single-node testing before multi-node
- Integration: set nvidia.com/gpu to 8 per node, Queue capacity to 24 (3 nodes)
- Integration: TrainJob example set to numNodes: 1 for install testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 15:44:49 +09:00
selee 6b413fe7d7 Update README: InfiniBand GPU 3-node config, simplify YAML examples 
- Change node config to InfiniBand GPU 3 nodes
- Remove inline YAML blocks (templates already in configs/)
- Task 4 now references configs/volcano-trainjob-integration.yaml directly
- Add ibstat to prerequisite checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 15:27:18 +09:00
selee dafa90e6d0 Update README and integration config to match official docs 
- README: reflect podGroupPolicy.volcano usage (not manual schedulerName)
- README: remove PriorityClass section (not needed for now, FIFO is default)
- README: add component placement summary table (all on nodegroup: nd)
- README: add nodegroup: nd prerequisite and verification commands
- README: update troubleshooting for podGroupPolicy-based integration
- Integration config: remove PriorityClass, renumber sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-09 16:15:24 +09:00
selee a9cd113b8f Add nodeSelector(nodegroup: nd) to all components, fix PriorityClass mismatch 
- Volcano: set default_ns.nodegroup: nd for all components
- Kubeflow Trainer: set manager.nodeSelector.nodegroup: nd
- JobSet: set controller.nodeSelector.nodegroup: nd
- Fix PodGroup priorityClassName: normal-priority -> training-priority

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-09 15:34:19 +09:00
selee 66e93c2035 Initial setup: Volcano + Kubeflow Trainer installation environment 
- Add Volcano Helm chart v1.14.0 (tgz)
- Add Kubeflow Trainer Helm chart sha-48e7a93 (tgz)
- Add ClusterTrainingRuntime manifests (runtimes.yaml)
- Add custom values for Volcano (gang scheduling, binpack for GPU)
- Add custom values for Kubeflow Trainer
- Add Volcano-Trainer integration config sample (Queue, PodGroup, PriorityClass)
- Add comprehensive installation manual (README.md)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-09 15:21:10 +09:00