Initial setup: Volcano + Kubeflow Trainer installation environment
- Add Volcano Helm chart v1.14.0 (tgz) - Add Kubeflow Trainer Helm chart sha-48e7a93 (tgz) - Add ClusterTrainingRuntime manifests (runtimes.yaml) - Add custom values for Volcano (gang scheduling, binpack for GPU) - Add custom values for Kubeflow Trainer - Add Volcano-Trainer integration config sample (Queue, PodGroup, PriorityClass) - Add comprehensive installation manual (README.md) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
selee
commit
66e93c2035
|
|
@ -0,0 +1 @@
|
|||
.env
|
||||
|
|
@ -0,0 +1,369 @@
|
|||
# Volcano + Kubeflow Trainer 설치 매뉴얼
|
||||
|
||||
멀티노드 GPU 분산 학습을 위한 Volcano Scheduler와 Kubeflow Trainer v2 설치 가이드.
|
||||
|
||||
## 환경 정보
|
||||
|
||||
| 항목 | 값 |
|
||||
|------|-----|
|
||||
| Kubernetes | v1.34.3 |
|
||||
| Helm | v4.0.4 |
|
||||
| 노드 구성 | Master 3 + Worker 13 (총 16) |
|
||||
| Volcano | v1.14.0 |
|
||||
| Kubeflow Trainer | sha-48e7a93 |
|
||||
| JobSet (의존성) | v0.10.1 |
|
||||
|
||||
## 디렉토리 구조
|
||||
|
||||
```
|
||||
kubeflow_trainer_with_volcano/
|
||||
├── .env # 환경변수 (git 제외)
|
||||
├── .gitignore
|
||||
├── README.md # 이 문서
|
||||
├── charts/
|
||||
│ ├── volcano/
|
||||
│ │ └── volcano-1.14.0.tgz # Volcano Helm chart
|
||||
│ └── kubeflow-trainer-0.0.0-sha-48e7a93.tgz # Kubeflow Trainer Helm chart
|
||||
├── manifests/
|
||||
│ └── kubeflow-trainer/
|
||||
│ └── runtimes/
|
||||
│ └── runtimes.yaml # ClusterTrainingRuntime manifests
|
||||
└── configs/
|
||||
├── volcano-values.yaml # Volcano 커스텀 values
|
||||
├── kubeflow-trainer-values.yaml # Kubeflow Trainer 커스텀 values
|
||||
└── volcano-trainjob-integration.yaml # Volcano-Trainer 연동 설정 샘플
|
||||
```
|
||||
|
||||
## 사전 요구사항
|
||||
|
||||
- Kubernetes 클러스터 (v1.26+)
|
||||
- Helm v3 이상
|
||||
- `kubectl` 클러스터 접근 설정 완료
|
||||
- GPU 노드에 NVIDIA device plugin 설치 완료
|
||||
- 클러스터에 기존 Volcano/Kubeflow CRD가 없는 상태
|
||||
|
||||
```bash
|
||||
# 사전 확인
|
||||
kubectl version
|
||||
helm version
|
||||
kubectl get nodes
|
||||
nvidia-smi # GPU 노드에서 확인
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Volcano 설치
|
||||
|
||||
Volcano는 Kubernetes를 위한 배치 스케줄링 시스템으로, gang scheduling을 통해 분산 학습 Pod들이 동시에 스케줄링되도록 보장합니다.
|
||||
|
||||
### 2.1 네임스페이스 생성
|
||||
|
||||
```bash
|
||||
kubectl create namespace volcano-system
|
||||
```
|
||||
|
||||
### 2.2 Helm으로 Volcano 설치 (로컬 chart 사용)
|
||||
|
||||
```bash
|
||||
helm install volcano charts/volcano/volcano-1.14.0.tgz \
|
||||
-n volcano-system \
|
||||
-f configs/volcano-values.yaml \
|
||||
--wait --timeout 5m
|
||||
```
|
||||
|
||||
### 2.3 설치 확인
|
||||
|
||||
```bash
|
||||
# Pod 상태 확인
|
||||
kubectl get pods -n volcano-system
|
||||
|
||||
# 기대 출력:
|
||||
# volcano-admission-xxxxx 1/1 Running
|
||||
# volcano-controllers-xxxxx 1/1 Running
|
||||
# volcano-scheduler-xxxxx 1/1 Running
|
||||
|
||||
# CRD 확인
|
||||
kubectl get crd | grep volcano
|
||||
|
||||
# 기대 출력:
|
||||
# commands.bus.volcano.sh
|
||||
# jobs.batch.volcano.sh
|
||||
# podgroups.scheduling.volcano.sh
|
||||
# queues.scheduling.volcano.sh
|
||||
|
||||
# 기본 Queue 확인
|
||||
kubectl get queue -n volcano-system
|
||||
```
|
||||
|
||||
### 2.4 Volcano 삭제 (필요시)
|
||||
|
||||
```bash
|
||||
helm uninstall volcano -n volcano-system
|
||||
kubectl delete namespace volcano-system
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Kubeflow Trainer 설치
|
||||
|
||||
Kubeflow Trainer v2는 TrainJob CRD를 통해 분산 학습 워크로드를 관리합니다.
|
||||
|
||||
### 3.1 네임스페이스 생성
|
||||
|
||||
```bash
|
||||
kubectl create namespace kubeflow-trainer
|
||||
```
|
||||
|
||||
### 3.2 Helm으로 Kubeflow Trainer 설치 (로컬 chart 사용)
|
||||
|
||||
```bash
|
||||
helm install kubeflow-trainer charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz \
|
||||
-n kubeflow-trainer \
|
||||
-f configs/kubeflow-trainer-values.yaml \
|
||||
--wait --timeout 5m
|
||||
```
|
||||
|
||||
### 3.3 ClusterTrainingRuntime 설치
|
||||
|
||||
```bash
|
||||
kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
|
||||
```
|
||||
|
||||
### 3.4 설치 확인
|
||||
|
||||
```bash
|
||||
# Trainer controller 확인
|
||||
kubectl get pods -n kubeflow-trainer
|
||||
|
||||
# 기대 출력:
|
||||
# kubeflow-trainer-controller-manager-xxxxx 1/1 Running
|
||||
|
||||
# CRD 확인
|
||||
kubectl get crd | grep trainer
|
||||
|
||||
# 기대 출력:
|
||||
# clustertrainingruntimes.trainer.kubeflow.org
|
||||
# trainjobs.trainer.kubeflow.org
|
||||
# trainingruntimes.trainer.kubeflow.org
|
||||
|
||||
# ClusterTrainingRuntime 확인
|
||||
kubectl get clustertrainingruntimes
|
||||
|
||||
# 기대 출력:
|
||||
# deepspeed-distributed
|
||||
# torch-distributed
|
||||
# ...
|
||||
```
|
||||
|
||||
### 3.5 Kubeflow Trainer 삭제 (필요시)
|
||||
|
||||
```bash
|
||||
kubectl delete -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
|
||||
helm uninstall kubeflow-trainer -n kubeflow-trainer
|
||||
kubectl delete namespace kubeflow-trainer
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Volcano ↔ Kubeflow Trainer 연동 설정
|
||||
|
||||
Volcano 스케줄러를 Kubeflow TrainJob과 연동하여 gang scheduling, queue 기반 리소스 관리를 활성화합니다.
|
||||
|
||||
### 4.1 Training Queue 생성
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: scheduling.volcano.sh/v1beta1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: training-queue
|
||||
spec:
|
||||
weight: 1
|
||||
reclaimable: true
|
||||
capability:
|
||||
cpu: "100"
|
||||
memory: "512Gi"
|
||||
nvidia.com/gpu: "16"
|
||||
EOF
|
||||
```
|
||||
|
||||
Queue 상태 확인:
|
||||
```bash
|
||||
kubectl get queue training-queue -o yaml
|
||||
```
|
||||
|
||||
### 4.2 PriorityClass 생성
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: training-priority
|
||||
value: 100
|
||||
globalDefault: false
|
||||
description: "Priority class for distributed training workloads"
|
||||
EOF
|
||||
```
|
||||
|
||||
### 4.3 Volcano 연동 ClusterTrainingRuntime 생성
|
||||
|
||||
Volcano 스케줄러를 사용하는 커스텀 런타임을 생성합니다.
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
name: torch-distributed-volcano
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torch
|
||||
spec:
|
||||
mlPolicy:
|
||||
torch:
|
||||
numProcPerNode: 1
|
||||
numNodes: 1
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
scheduling.volcano.sh/queue-name: training-queue
|
||||
spec:
|
||||
schedulerName: volcano
|
||||
containers:
|
||||
- name: node
|
||||
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
EOF
|
||||
```
|
||||
|
||||
또는 전체 연동 설정 샘플 적용:
|
||||
```bash
|
||||
kubectl apply -f configs/volcano-trainjob-integration.yaml
|
||||
```
|
||||
|
||||
### 4.4 연동 테스트
|
||||
|
||||
Volcano 스케줄러를 사용하는 TrainJob 제출:
|
||||
|
||||
```bash
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: test-volcano-integration
|
||||
namespace: default
|
||||
spec:
|
||||
runtimeRef:
|
||||
name: torch-distributed-volcano
|
||||
trainer:
|
||||
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
|
||||
numNodes: 2
|
||||
numProcPerNode: "1"
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
EOF
|
||||
```
|
||||
|
||||
### 4.5 연동 확인
|
||||
|
||||
```bash
|
||||
# TrainJob 상태 확인
|
||||
kubectl get trainjob test-volcano-integration
|
||||
|
||||
# PodGroup 자동 생성 확인 (Volcano가 관리)
|
||||
kubectl get podgroup
|
||||
|
||||
# Pod 스케줄러 확인 (schedulerName이 volcano인지)
|
||||
kubectl get pods -l trainer.kubeflow.org/trainjob-name=test-volcano-integration \
|
||||
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.schedulerName}{"\n"}{end}'
|
||||
|
||||
# Volcano Queue 사용량 확인
|
||||
kubectl get queue training-queue
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 검증 체크리스트
|
||||
|
||||
- [ ] Volcano Pod 3개 Running (admission, controller, scheduler)
|
||||
- [ ] Volcano CRD 생성됨 (jobs, podgroups, queues 등)
|
||||
- [ ] Kubeflow Trainer controller Running
|
||||
- [ ] Kubeflow Trainer CRD 생성됨 (trainjobs, trainingruntimes 등)
|
||||
- [ ] ClusterTrainingRuntime 목록 확인 (deepspeed-distributed, torch-distributed 등)
|
||||
- [ ] Training Queue 생성됨
|
||||
- [ ] 테스트 TrainJob이 Volcano 스케줄러로 스케줄링됨
|
||||
|
||||
---
|
||||
|
||||
## 트러블슈팅
|
||||
|
||||
### Volcano 관련
|
||||
|
||||
**Volcano admission webhook 타임아웃**
|
||||
```bash
|
||||
# webhook 인증서 확인
|
||||
kubectl get secret volcano-admission-secret -n volcano-system
|
||||
# admission pod 로그 확인
|
||||
kubectl logs -n volcano-system -l app=volcano-admission
|
||||
```
|
||||
|
||||
**Pod가 Pending 상태로 유지**
|
||||
```bash
|
||||
# PodGroup 상태 확인 - minMember 충족 여부
|
||||
kubectl describe podgroup <podgroup-name>
|
||||
# Queue 리소스 capacity 확인
|
||||
kubectl get queue training-queue -o yaml
|
||||
# Volcano scheduler 로그 확인
|
||||
kubectl logs -n volcano-system -l app=volcano-scheduler
|
||||
```
|
||||
|
||||
### Kubeflow Trainer 관련
|
||||
|
||||
**TrainJob이 생성되지 않음**
|
||||
```bash
|
||||
# Trainer controller 로그 확인
|
||||
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer
|
||||
# TrainJob 이벤트 확인
|
||||
kubectl describe trainjob <trainjob-name>
|
||||
```
|
||||
|
||||
**ClusterTrainingRuntime을 찾을 수 없음**
|
||||
```bash
|
||||
# 런타임 목록 확인
|
||||
kubectl get clustertrainingruntimes
|
||||
# 런타임이 없으면 재설치
|
||||
kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
|
||||
```
|
||||
|
||||
### 연동 관련
|
||||
|
||||
**TrainJob Pod가 default-scheduler로 스케줄링됨**
|
||||
- `schedulerName: volcano` 설정이 ClusterTrainingRuntime의 pod spec에 있는지 확인
|
||||
- TrainJob이 올바른 runtimeRef를 참조하는지 확인
|
||||
|
||||
**PodGroup이 자동 생성되지 않음**
|
||||
- Volcano controller 로그 확인
|
||||
- Pod에 `scheduling.volcano.sh/queue-name` annotation이 있는지 확인
|
||||
|
||||
### 로그 수집
|
||||
|
||||
```bash
|
||||
# 전체 로그 수집
|
||||
kubectl logs -n volcano-system -l app=volcano-scheduler --tail=100
|
||||
kubectl logs -n volcano-system -l app=volcano-controller --tail=100
|
||||
kubectl logs -n volcano-system -l app=volcano-admission --tail=100
|
||||
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer --tail=100
|
||||
```
|
||||
Binary file not shown.
Binary file not shown.
|
|
@ -0,0 +1,52 @@
|
|||
# Kubeflow Trainer Helm Chart Custom Values
|
||||
# Chart version: 0.0.0-sha-48e7a93
|
||||
# Reference: charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz (default values)
|
||||
|
||||
# JobSet dependency - install if not already present
|
||||
jobset:
|
||||
install: true
|
||||
fullnameOverride: jobset
|
||||
|
||||
image:
|
||||
registry: ghcr.io
|
||||
repository: kubeflow/trainer/trainer-controller-manager
|
||||
pullPolicy: IfNotPresent
|
||||
|
||||
manager:
|
||||
replicas: 1
|
||||
resources:
|
||||
limits:
|
||||
cpu: 200m
|
||||
memory: 512Mi
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
runAsNonRoot: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
config:
|
||||
health:
|
||||
healthProbeBindAddress: ":8081"
|
||||
metrics:
|
||||
bindAddress: ":8443"
|
||||
secureServing: true
|
||||
webhook:
|
||||
port: 9443
|
||||
leaderElection:
|
||||
leaderElect: true
|
||||
resourceName: "trainer.kubeflow.org"
|
||||
controller:
|
||||
groupKindConcurrency:
|
||||
trainJob: 5
|
||||
trainingRuntime: 1
|
||||
clusterTrainingRuntime: 1
|
||||
certManagement:
|
||||
enable: true
|
||||
|
||||
webhook:
|
||||
failurePolicy: Fail
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
# Volcano <-> Kubeflow Trainer Integration Configuration Sample
|
||||
# This file contains examples for integrating Volcano scheduler with Kubeflow TrainJob.
|
||||
|
||||
---
|
||||
# 1. Volcano Queue for training workloads
|
||||
apiVersion: scheduling.volcano.sh/v1beta1
|
||||
kind: Queue
|
||||
metadata:
|
||||
name: training-queue
|
||||
spec:
|
||||
weight: 1
|
||||
reclaimable: true
|
||||
capability:
|
||||
# Adjust based on your cluster capacity
|
||||
cpu: "100"
|
||||
memory: "512Gi"
|
||||
nvidia.com/gpu: "16"
|
||||
|
||||
---
|
||||
# 2. Example TrainJob with Volcano PodGroup scheduling
|
||||
# TrainJob that uses Volcano as the scheduler via podGroupPolicy
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: TrainJob
|
||||
metadata:
|
||||
name: example-distributed-training
|
||||
namespace: default
|
||||
spec:
|
||||
runtimeRef:
|
||||
name: torch-distributed
|
||||
trainer:
|
||||
image: docker.io/your-org/your-training-image:latest
|
||||
numNodes: 2
|
||||
numProcPerNode: "auto"
|
||||
resourcesPerNode:
|
||||
requests:
|
||||
nvidia.com/gpu: "4"
|
||||
cpu: "16"
|
||||
memory: "64Gi"
|
||||
limits:
|
||||
nvidia.com/gpu: "4"
|
||||
cpu: "16"
|
||||
memory: "64Gi"
|
||||
|
||||
---
|
||||
# 3. Volcano PodGroup for gang scheduling (auto-created by integration)
|
||||
# This is a reference example - Volcano controller auto-creates PodGroups
|
||||
# when scheduler annotation is set.
|
||||
apiVersion: scheduling.volcano.sh/v1beta1
|
||||
kind: PodGroup
|
||||
metadata:
|
||||
name: example-distributed-training
|
||||
namespace: default
|
||||
spec:
|
||||
# MinMember ensures all pods are scheduled together (gang scheduling)
|
||||
minMember: 2
|
||||
queue: training-queue
|
||||
priorityClassName: normal-priority
|
||||
minResources:
|
||||
cpu: "32"
|
||||
memory: "128Gi"
|
||||
nvidia.com/gpu: "8"
|
||||
|
||||
---
|
||||
# 4. PriorityClass for training workloads
|
||||
apiVersion: scheduling.k8s.io/v1
|
||||
kind: PriorityClass
|
||||
metadata:
|
||||
name: training-priority
|
||||
value: 100
|
||||
globalDefault: false
|
||||
description: "Priority class for distributed training workloads"
|
||||
|
||||
---
|
||||
# 5. ClusterTrainingRuntime with Volcano scheduler annotation
|
||||
# Use this runtime to automatically schedule TrainJobs with Volcano
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
name: torch-distributed-volcano
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torch
|
||||
spec:
|
||||
mlPolicy:
|
||||
torch:
|
||||
numProcPerNode: 1
|
||||
numNodes: 1
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
annotations:
|
||||
# Use Volcano as the scheduler
|
||||
scheduling.volcano.sh/queue-name: training-queue
|
||||
spec:
|
||||
schedulerName: volcano
|
||||
containers:
|
||||
- name: node
|
||||
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
|
||||
resources:
|
||||
requests:
|
||||
nvidia.com/gpu: "1"
|
||||
limits:
|
||||
nvidia.com/gpu: "1"
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
# Volcano Helm Chart Custom Values
|
||||
# Chart version: 1.14.0
|
||||
# Reference: charts/volcano/volcano-1.14.0.tgz (default values)
|
||||
|
||||
basic:
|
||||
image_pull_policy: "IfNotPresent"
|
||||
image_tag_version: "v1.14.0"
|
||||
|
||||
custom:
|
||||
# Enable admission webhook
|
||||
admission_enable: true
|
||||
admission_replicas: 1
|
||||
|
||||
# Enable controller
|
||||
controller_enable: true
|
||||
controller_replicas: 1
|
||||
|
||||
# Enable scheduler
|
||||
scheduler_enable: true
|
||||
scheduler_replicas: 1
|
||||
|
||||
# Enable leader election for HA
|
||||
leader_elect_enable: false
|
||||
|
||||
# Scheduler config override for GPU workloads
|
||||
# Includes gang scheduling, binpack for GPU locality, and topology-aware scheduling
|
||||
scheduler_config_override: |
|
||||
actions: "enqueue, allocate, backfill"
|
||||
tiers:
|
||||
- plugins:
|
||||
- name: priority
|
||||
- name: gang
|
||||
enablePreemptable: false
|
||||
- name: conformance
|
||||
- plugins:
|
||||
- name: overcommit
|
||||
- name: drf
|
||||
enablePreemptable: false
|
||||
- name: predicates
|
||||
- name: proportion
|
||||
- name: nodeorder
|
||||
- name: binpack
|
||||
arguments:
|
||||
binpack.weight: 10
|
||||
binpack.resources: nvidia.com/gpu
|
||||
binpack.resources.nvidia.com/gpu.weight: 5
|
||||
|
||||
# API rate limits
|
||||
scheduler_kube_api_qps: 2000
|
||||
scheduler_kube_api_burst: 2000
|
||||
controller_kube_api_qps: 50
|
||||
controller_kube_api_burst: 100
|
||||
|
|
@ -0,0 +1,453 @@
|
|||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: deepspeed
|
||||
name: deepspeed-distributed
|
||||
spec:
|
||||
mlPolicy:
|
||||
mpi:
|
||||
mpiImplementation: OpenMPI
|
||||
numProcPerNode: 1
|
||||
runLauncherAsNode: true
|
||||
sshAuthMountPath: /home/mpiuser/.ssh
|
||||
numNodes: 1
|
||||
template:
|
||||
spec:
|
||||
network:
|
||||
publishNotReadyAddresses: true
|
||||
replicatedJobs:
|
||||
- name: launcher
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
|
||||
name: node
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
- name: node
|
||||
template:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- -De
|
||||
- -f
|
||||
- /home/mpiuser/.sshd_config
|
||||
command:
|
||||
- /usr/sbin/sshd
|
||||
image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
|
||||
name: node
|
||||
readinessProbe:
|
||||
initialDelaySeconds: 5
|
||||
tcpSocket:
|
||||
port: 2222
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
successPolicy:
|
||||
operator: All
|
||||
targetReplicatedJobs:
|
||||
- launcher
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: jax
|
||||
name: jax-distributed
|
||||
spec:
|
||||
mlPolicy:
|
||||
jax: {}
|
||||
numNodes: 1
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- image: nvcr.io/nvidia/jax:25.10-py3
|
||||
name: node
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: mlx
|
||||
name: mlx-distributed
|
||||
spec:
|
||||
mlPolicy:
|
||||
mpi:
|
||||
mpiImplementation: OpenMPI
|
||||
numProcPerNode: 1
|
||||
runLauncherAsNode: true
|
||||
sshAuthMountPath: /home/mpiuser/.ssh
|
||||
numNodes: 1
|
||||
template:
|
||||
spec:
|
||||
network:
|
||||
publishNotReadyAddresses: true
|
||||
replicatedJobs:
|
||||
- name: launcher
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
|
||||
name: node
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
- name: node
|
||||
template:
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- args:
|
||||
- -De
|
||||
- -f
|
||||
- /home/mpiuser/.sshd_config
|
||||
command:
|
||||
- /usr/sbin/sshd
|
||||
image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
|
||||
name: node
|
||||
readinessProbe:
|
||||
initialDelaySeconds: 5
|
||||
tcpSocket:
|
||||
port: 2222
|
||||
securityContext:
|
||||
runAsUser: 1000
|
||||
successPolicy:
|
||||
operator: All
|
||||
targetReplicatedJobs:
|
||||
- launcher
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torch
|
||||
name: torch-distributed
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch:
|
||||
numProcPerNode: auto
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- image: pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime
|
||||
name: node
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torchtune
|
||||
name: torchtune-llama3.2-1b
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch:
|
||||
numProcPerNode: auto
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: dataset-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://tatsu-lab/alpaca
|
||||
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
|
||||
name: dataset-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-1b
|
||||
- name: model-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://meta-llama/Llama-3.2-1B-Instruct
|
||||
image: ghcr.io/kubeflow/trainer/model-initializer:latest
|
||||
name: model-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-1b
|
||||
- dependsOn:
|
||||
- name: dataset-initializer
|
||||
status: Complete
|
||||
- name: model-initializer
|
||||
status: Complete
|
||||
name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- tune
|
||||
- run
|
||||
- --rdzv_endpoint=localhost:29500
|
||||
- full_finetune_distributed
|
||||
- --config
|
||||
- llama3_2/1B_full
|
||||
- dataset=torchtune.datasets.instruct_dataset
|
||||
- dataset.source=parquet
|
||||
- dataset.data_dir=/workspace/dataset/data
|
||||
- output_dir=/workspace/output
|
||||
- tokenizer.path=/workspace/model/original/tokenizer.model
|
||||
- checkpointer.checkpoint_dir=/workspace/model
|
||||
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
|
||||
name: node
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-1b
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torchtune
|
||||
name: torchtune-llama3.2-3b
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch:
|
||||
numProcPerNode: auto
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: dataset-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://tatsu-lab/alpaca
|
||||
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
|
||||
name: dataset-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-3b
|
||||
- name: model-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://meta-llama/Llama-3.2-3B-Instruct
|
||||
image: ghcr.io/kubeflow/trainer/model-initializer:latest
|
||||
name: model-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-3b
|
||||
- dependsOn:
|
||||
- name: dataset-initializer
|
||||
status: Complete
|
||||
- name: model-initializer
|
||||
status: Complete
|
||||
name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- tune
|
||||
- run
|
||||
- --rdzv_endpoint=localhost:29500
|
||||
- full_finetune_distributed
|
||||
- --config
|
||||
- llama3_2/3B_full
|
||||
- dataset=torchtune.datasets.instruct_dataset
|
||||
- dataset.source=parquet
|
||||
- dataset.data_dir=/workspace/dataset/data
|
||||
- output_dir=/workspace/output
|
||||
- tokenizer.path=/workspace/model/original/tokenizer.model
|
||||
- checkpointer.checkpoint_dir=/workspace/model
|
||||
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
|
||||
name: node
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-llama3.2-3b
|
||||
---
|
||||
apiVersion: trainer.kubeflow.org/v1alpha1
|
||||
kind: ClusterTrainingRuntime
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/framework: torchtune
|
||||
name: torchtune-qwen2.5-1.5b
|
||||
spec:
|
||||
mlPolicy:
|
||||
numNodes: 1
|
||||
torch:
|
||||
numProcPerNode: auto
|
||||
template:
|
||||
spec:
|
||||
replicatedJobs:
|
||||
- name: dataset-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://tatsu-lab/alpaca
|
||||
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
|
||||
name: dataset-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-qwen2.5-1.5b
|
||||
- name: model-initializer
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: STORAGE_URI
|
||||
value: hf://Qwen/Qwen2.5-1.5B-Instruct
|
||||
image: ghcr.io/kubeflow/trainer/model-initializer:latest
|
||||
name: model-initializer
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-qwen2.5-1.5b
|
||||
- dependsOn:
|
||||
- name: dataset-initializer
|
||||
status: Complete
|
||||
- name: model-initializer
|
||||
status: Complete
|
||||
name: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
trainer.kubeflow.org/trainjob-ancestor-step: trainer
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- command:
|
||||
- tune
|
||||
- run
|
||||
- --rdzv_endpoint=localhost:29500
|
||||
- full_finetune_distributed
|
||||
- --config
|
||||
- qwen2_5/1.5B_full
|
||||
- dataset=torchtune.datasets.instruct_dataset
|
||||
- dataset.source=parquet
|
||||
- dataset.data_dir=/workspace/dataset/data
|
||||
- output_dir=/workspace/output
|
||||
- tokenizer.path=/workspace/model/vocab.json
|
||||
- tokenizer.merges_file=/workspace/model/merges.txt
|
||||
- checkpointer.checkpoint_dir=/workspace/model
|
||||
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
|
||||
name: node
|
||||
resources:
|
||||
limits:
|
||||
nvidia.com/gpu: 2
|
||||
volumeMounts:
|
||||
- mountPath: /workspace
|
||||
name: initializer
|
||||
volumes:
|
||||
- name: initializer
|
||||
persistentVolumeClaim:
|
||||
claimName: torchtune-qwen2.5-1.5b
|
||||
Loading…
Reference in New Issue