selee
/
kubeflow_trainer_with_volcano
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12 Commits 1 Branch 0 Tags 531 KiB
Commit Graph

9 Commits

Author SHA1 Message Date
selee a5171437a2 Add CIFAR-10 dataset pre-placement requirement to README 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 16:40:48 +09:00
selee 965b87ced3 Update README: torchrun usage guide, verified training results 
- Rewrite known issues as usage notes with torchrun/PET_* explanation
- Add trainer.command to TrainJob options table
- Update checklist with 2-node 16-GPU training verification

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 16:38:32 +09:00
selee 2b1ad10c2c Remove command from Runtime, move to TrainJob trainer.command 
Kubeflow Trainer torch runtime auto-injects torchrun only when
command is not set in the container spec. Training script is now
specified at TrainJob level via trainer.command instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 16:26:30 +09:00
selee 00529c5ca9 Comment out JAX runtime, update README with install results 
- JAX runtime commented out (CRD does not support spec.mlPolicy.jax)
- Mark verification checklist items as completed
- Add known issues section (torchrun injection, JAX support)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 16:23:06 +09:00
selee 146baa41e8 Add ConfigMap training script, update IB/NCCL config in README 
- Add ConfigMap with train_nccl.py (VGG11 + CIFAR10 + NCCL)
- Runtime: mount training-scripts ConfigMap to /workspace/scripts
- Runtime: set command to python /workspace/scripts/train_nccl.py
- Runtime: remove /dev/infiniband hostPath (hostdevice-net CNI handles it)
- Runtime: add NCCL_DEBUG_SUBSYS=INIT,NET,IB
- Runtime: add cifar-10 dataset hostPath mount
- README: add NCCL_DEBUG_SUBSYS, cifar-data, MASTER_ADDR auto-config note

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 16:10:10 +09:00
selee 6b72590351 Add resource descriptions to README, set GPU to 8 per node 
- README: add Queue/ClusterTrainingRuntime/TrainJob explanations with key options
- README: note single-node testing before multi-node
- Integration: set nvidia.com/gpu to 8 per node, Queue capacity to 24 (3 nodes)
- Integration: TrainJob example set to numNodes: 1 for install testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 15:44:49 +09:00
selee 6b413fe7d7 Update README: InfiniBand GPU 3-node config, simplify YAML examples 
- Change node config to InfiniBand GPU 3 nodes
- Remove inline YAML blocks (templates already in configs/)
- Task 4 now references configs/volcano-trainjob-integration.yaml directly
- Add ibstat to prerequisite checks

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-10 15:27:18 +09:00
selee dafa90e6d0 Update README and integration config to match official docs 
- README: reflect podGroupPolicy.volcano usage (not manual schedulerName)
- README: remove PriorityClass section (not needed for now, FIFO is default)
- README: add component placement summary table (all on nodegroup: nd)
- README: add nodegroup: nd prerequisite and verification commands
- README: update troubleshooting for podGroupPolicy-based integration
- Integration config: remove PriorityClass, renumber sections

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-09 16:15:24 +09:00
selee 66e93c2035 Initial setup: Volcano + Kubeflow Trainer installation environment 
- Add Volcano Helm chart v1.14.0 (tgz)
- Add Kubeflow Trainer Helm chart sha-48e7a93 (tgz)
- Add ClusterTrainingRuntime manifests (runtimes.yaml)
- Add custom values for Volcano (gang scheduling, binpack for GPU)
- Add custom values for Kubeflow Trainer
- Add Volcano-Trainer integration config sample (Queue, PodGroup, PriorityClass)
- Add comprehensive installation manual (README.md)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-09 15:21:10 +09:00