selee
/
kubeflow_trainer_with_volcano
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial setup: Volcano + Kubeflow Trainer installation environment 

- Add Volcano Helm chart v1.14.0 (tgz)
- Add Kubeflow Trainer Helm chart sha-48e7a93 (tgz)
- Add ClusterTrainingRuntime manifests (runtimes.yaml)
- Add custom values for Volcano (gang scheduling, binpack for GPU)
- Add custom values for Kubeflow Trainer
- Add Volcano-Trainer integration config sample (Queue, PodGroup, PriorityClass)
- Add comprehensive installation manual (README.md)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
selee 2026-02-09 15:21:10 +09:00
commit 66e93c2035
8 changed files with 1034 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored Normal file
View File

@ -0,0 +1 @@
.env

369
README.md Normal file
View File

@ -0,0 +1,369 @@
# Volcano + Kubeflow Trainer 설치 매뉴얼
멀티노드 GPU 분산 학습을 위한 Volcano Scheduler와 Kubeflow Trainer v2 설치 가이드.
## 환경 정보
| 항목 | 값 |
|------|-----|
| Kubernetes | v1.34.3 |
| Helm | v4.0.4 |
| 노드 구성 | Master 3 + Worker 13 (총 16) |
| Volcano | v1.14.0 |
| Kubeflow Trainer | sha-48e7a93 |
| JobSet (의존성) | v0.10.1 |
## 디렉토리 구조
```
kubeflow_trainer_with_volcano/
├── .env # 환경변수 (git 제외)
├── .gitignore
├── README.md # 이 문서
├── charts/
│ ├── volcano/
│ │ └── volcano-1.14.0.tgz # Volcano Helm chart
│ └── kubeflow-trainer-0.0.0-sha-48e7a93.tgz # Kubeflow Trainer Helm chart
├── manifests/
│ └── kubeflow-trainer/
│ └── runtimes/
│ └── runtimes.yaml # ClusterTrainingRuntime manifests
└── configs/
├── volcano-values.yaml # Volcano 커스텀 values
├── kubeflow-trainer-values.yaml # Kubeflow Trainer 커스텀 values
└── volcano-trainjob-integration.yaml # Volcano-Trainer 연동 설정 샘플
```
## 사전 요구사항
- Kubernetes 클러스터 (v1.26+)
- Helm v3 이상
- `kubectl` 클러스터 접근 설정 완료
- GPU 노드에 NVIDIA device plugin 설치 완료
- 클러스터에 기존 Volcano/Kubeflow CRD가 없는 상태
```bash
# 사전 확인
kubectl version
helm version
kubectl get nodes
nvidia-smi # GPU 노드에서 확인
```
---
## Task 2: Volcano 설치
Volcano는 Kubernetes를 위한 배치 스케줄링 시스템으로, gang scheduling을 통해 분산 학습 Pod들이 동시에 스케줄링되도록 보장합니다.
### 2.1 네임스페이스 생성
```bash
kubectl create namespace volcano-system
```
### 2.2 Helm으로 Volcano 설치 (로컬 chart 사용)
```bash
helm install volcano charts/volcano/volcano-1.14.0.tgz \
-n volcano-system \
-f configs/volcano-values.yaml \
--wait --timeout 5m
```
### 2.3 설치 확인
```bash
# Pod 상태 확인
kubectl get pods -n volcano-system
# 기대 출력:
# volcano-admission-xxxxx 1/1 Running
# volcano-controllers-xxxxx 1/1 Running
# volcano-scheduler-xxxxx 1/1 Running
# CRD 확인
kubectl get crd | grep volcano
# 기대 출력:
# commands.bus.volcano.sh
# jobs.batch.volcano.sh
# podgroups.scheduling.volcano.sh
# queues.scheduling.volcano.sh
# 기본 Queue 확인
kubectl get queue -n volcano-system
```
### 2.4 Volcano 삭제 (필요시)
```bash
helm uninstall volcano -n volcano-system
kubectl delete namespace volcano-system
```
---
## Task 3: Kubeflow Trainer 설치
Kubeflow Trainer v2는 TrainJob CRD를 통해 분산 학습 워크로드를 관리합니다.
### 3.1 네임스페이스 생성
```bash
kubectl create namespace kubeflow-trainer
```
### 3.2 Helm으로 Kubeflow Trainer 설치 (로컬 chart 사용)
```bash
helm install kubeflow-trainer charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz \
-n kubeflow-trainer \
-f configs/kubeflow-trainer-values.yaml \
--wait --timeout 5m
```
### 3.3 ClusterTrainingRuntime 설치
```bash
kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
```
### 3.4 설치 확인
```bash
# Trainer controller 확인
kubectl get pods -n kubeflow-trainer
# 기대 출력:
# kubeflow-trainer-controller-manager-xxxxx 1/1 Running
# CRD 확인
kubectl get crd | grep trainer
# 기대 출력:
# clustertrainingruntimes.trainer.kubeflow.org
# trainjobs.trainer.kubeflow.org
# trainingruntimes.trainer.kubeflow.org
# ClusterTrainingRuntime 확인
kubectl get clustertrainingruntimes
# 기대 출력:
# deepspeed-distributed
# torch-distributed
# ...
```
### 3.5 Kubeflow Trainer 삭제 (필요시)
```bash
kubectl delete -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
helm uninstall kubeflow-trainer -n kubeflow-trainer
kubectl delete namespace kubeflow-trainer
```
---
## Task 4: Volcano ↔ Kubeflow Trainer 연동 설정
Volcano 스케줄러를 Kubeflow TrainJob과 연동하여 gang scheduling, queue 기반 리소스 관리를 활성화합니다.
### 4.1 Training Queue 생성
```bash
kubectl apply -f - <<EOF
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
name: training-queue
spec:
weight: 1
reclaimable: true
capability:
cpu: "100"
memory: "512Gi"
nvidia.com/gpu: "16"
EOF
```
Queue 상태 확인:
```bash
kubectl get queue training-queue -o yaml
```
### 4.2 PriorityClass 생성
```bash
kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: training-priority
value: 100
globalDefault: false
description: "Priority class for distributed training workloads"
EOF
```
### 4.3 Volcano 연동 ClusterTrainingRuntime 생성
Volcano 스케줄러를 사용하는 커스텀 런타임을 생성합니다.
```bash
kubectl apply -f - <<EOF
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
name: torch-distributed-volcano
labels:
trainer.kubeflow.org/framework: torch
spec:
mlPolicy:
torch:
numProcPerNode: 1
numNodes: 1
template:
spec:
replicatedJobs:
- name: node
template:
spec:
template:
metadata:
annotations:
scheduling.volcano.sh/queue-name: training-queue
spec:
schedulerName: volcano
containers:
- name: node
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
EOF
```
또는 전체 연동 설정 샘플 적용:
```bash
kubectl apply -f configs/volcano-trainjob-integration.yaml
```
### 4.4 연동 테스트
Volcano 스케줄러를 사용하는 TrainJob 제출:
```bash
kubectl apply -f - <<EOF
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
name: test-volcano-integration
namespace: default
spec:
runtimeRef:
name: torch-distributed-volcano
trainer:
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
numNodes: 2
numProcPerNode: "1"
resourcesPerNode:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
EOF
```
### 4.5 연동 확인
```bash
# TrainJob 상태 확인
kubectl get trainjob test-volcano-integration
# PodGroup 자동 생성 확인 (Volcano가 관리)
kubectl get podgroup
# Pod 스케줄러 확인 (schedulerName이 volcano인지)
kubectl get pods -l trainer.kubeflow.org/trainjob-name=test-volcano-integration \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.schedulerName}{"\n"}{end}'
# Volcano Queue 사용량 확인
kubectl get queue training-queue
```
---
## 검증 체크리스트
- [ ] Volcano Pod 3개 Running (admission, controller, scheduler)
- [ ] Volcano CRD 생성됨 (jobs, podgroups, queues 등)
- [ ] Kubeflow Trainer controller Running
- [ ] Kubeflow Trainer CRD 생성됨 (trainjobs, trainingruntimes 등)
- [ ] ClusterTrainingRuntime 목록 확인 (deepspeed-distributed, torch-distributed 등)
- [ ] Training Queue 생성됨
- [ ] 테스트 TrainJob이 Volcano 스케줄러로 스케줄링됨
---
## 트러블슈팅
### Volcano 관련
**Volcano admission webhook 타임아웃**
```bash
# webhook 인증서 확인
kubectl get secret volcano-admission-secret -n volcano-system
# admission pod 로그 확인
kubectl logs -n volcano-system -l app=volcano-admission
```
**Pod가 Pending 상태로 유지**
```bash
# PodGroup 상태 확인 - minMember 충족 여부
kubectl describe podgroup <podgroup-name>
# Queue 리소스 capacity 확인
kubectl get queue training-queue -o yaml
# Volcano scheduler 로그 확인
kubectl logs -n volcano-system -l app=volcano-scheduler
```
### Kubeflow Trainer 관련
**TrainJob이 생성되지 않음**
```bash
# Trainer controller 로그 확인
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer
# TrainJob 이벤트 확인
kubectl describe trainjob <trainjob-name>
```
**ClusterTrainingRuntime을 찾을 수 없음**
```bash
# 런타임 목록 확인
kubectl get clustertrainingruntimes
# 런타임이 없으면 재설치
kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
```
### 연동 관련
**TrainJob Pod가 default-scheduler로 스케줄링됨**
- `schedulerName: volcano` 설정이 ClusterTrainingRuntime의 pod spec에 있는지 확인
- TrainJob이 올바른 runtimeRef를 참조하는지 확인
**PodGroup이 자동 생성되지 않음**
- Volcano controller 로그 확인
- Pod에 `scheduling.volcano.sh/queue-name` annotation이 있는지 확인
### 로그 수집
```bash
# 전체 로그 수집
kubectl logs -n volcano-system -l app=volcano-scheduler --tail=100
kubectl logs -n volcano-system -l app=volcano-controller --tail=100
kubectl logs -n volcano-system -l app=volcano-admission --tail=100
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer --tail=100
```

BIN
charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz Normal file
View File

Binary file not shown.

BIN
charts/volcano/volcano-1.14.0.tgz Normal file
View File

Binary file not shown.

52
configs/kubeflow-trainer-values.yaml Normal file
View File

@ -0,0 +1,52 @@
# Kubeflow Trainer Helm Chart Custom Values
# Chart version: 0.0.0-sha-48e7a93
# Reference: charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz (default values)
# JobSet dependency - install if not already present
jobset:
install: true
fullnameOverride: jobset
image:
registry: ghcr.io
repository: kubeflow/trainer/trainer-controller-manager
pullPolicy: IfNotPresent
manager:
replicas: 1
resources:
limits:
cpu: 200m
memory: 512Mi
requests:
cpu: 100m
memory: 256Mi
securityContext:
allowPrivilegeEscalation: false
runAsNonRoot: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
config:
health:
healthProbeBindAddress: ":8081"
metrics:
bindAddress: ":8443"
secureServing: true
webhook:
port: 9443
leaderElection:
leaderElect: true
resourceName: "trainer.kubeflow.org"
controller:
groupKindConcurrency:
trainJob: 5
trainingRuntime: 1
clusterTrainingRuntime: 1
certManagement:
enable: true
webhook:
failurePolicy: Fail

107
configs/volcano-trainjob-integration.yaml Normal file
View File

@ -0,0 +1,107 @@
# Volcano <-> Kubeflow Trainer Integration Configuration Sample
# This file contains examples for integrating Volcano scheduler with Kubeflow TrainJob.
---
# 1. Volcano Queue for training workloads
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
name: training-queue
spec:
weight: 1
reclaimable: true
capability:
# Adjust based on your cluster capacity
cpu: "100"
memory: "512Gi"
nvidia.com/gpu: "16"
---
# 2. Example TrainJob with Volcano PodGroup scheduling
# TrainJob that uses Volcano as the scheduler via podGroupPolicy
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
name: example-distributed-training
namespace: default
spec:
runtimeRef:
name: torch-distributed
trainer:
image: docker.io/your-org/your-training-image:latest
numNodes: 2
numProcPerNode: "auto"
resourcesPerNode:
requests:
nvidia.com/gpu: "4"
cpu: "16"
memory: "64Gi"
limits:
nvidia.com/gpu: "4"
cpu: "16"
memory: "64Gi"
---
# 3. Volcano PodGroup for gang scheduling (auto-created by integration)
# This is a reference example - Volcano controller auto-creates PodGroups
# when scheduler annotation is set.
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
name: example-distributed-training
namespace: default
spec:
# MinMember ensures all pods are scheduled together (gang scheduling)
minMember: 2
queue: training-queue
priorityClassName: normal-priority
minResources:
cpu: "32"
memory: "128Gi"
nvidia.com/gpu: "8"
---
# 4. PriorityClass for training workloads
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: training-priority
value: 100
globalDefault: false
description: "Priority class for distributed training workloads"
---
# 5. ClusterTrainingRuntime with Volcano scheduler annotation
# Use this runtime to automatically schedule TrainJobs with Volcano
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
name: torch-distributed-volcano
labels:
trainer.kubeflow.org/framework: torch
spec:
mlPolicy:
torch:
numProcPerNode: 1
numNodes: 1
template:
spec:
replicatedJobs:
- name: node
template:
spec:
template:
metadata:
annotations:
# Use Volcano as the scheduler
scheduling.volcano.sh/queue-name: training-queue
spec:
schedulerName: volcano
containers:
- name: node
image: ghcr.io/kubeflow/trainer/torch-runtime:latest
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"

52
configs/volcano-values.yaml Normal file
View File

@ -0,0 +1,52 @@
# Volcano Helm Chart Custom Values
# Chart version: 1.14.0
# Reference: charts/volcano/volcano-1.14.0.tgz (default values)
basic:
image_pull_policy: "IfNotPresent"
image_tag_version: "v1.14.0"
custom:
# Enable admission webhook
admission_enable: true
admission_replicas: 1
# Enable controller
controller_enable: true
controller_replicas: 1
# Enable scheduler
scheduler_enable: true
scheduler_replicas: 1
# Enable leader election for HA
leader_elect_enable: false
# Scheduler config override for GPU workloads
# Includes gang scheduling, binpack for GPU locality, and topology-aware scheduling
scheduler_config_override: |
actions: "enqueue, allocate, backfill"
tiers:
- plugins:
- name: priority
- name: gang
enablePreemptable: false
- name: conformance
- plugins:
- name: overcommit
- name: drf
enablePreemptable: false
- name: predicates
- name: proportion
- name: nodeorder
- name: binpack
arguments:
binpack.weight: 10
binpack.resources: nvidia.com/gpu
binpack.resources.nvidia.com/gpu.weight: 5
# API rate limits
scheduler_kube_api_qps: 2000
scheduler_kube_api_burst: 2000
controller_kube_api_qps: 50
controller_kube_api_burst: 100

453
manifests/kubeflow-trainer/runtimes/runtimes.yaml Normal file
View File

@ -0,0 +1,453 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: deepspeed
name: deepspeed-distributed
spec:
mlPolicy:
mpi:
mpiImplementation: OpenMPI
numProcPerNode: 1
runLauncherAsNode: true
sshAuthMountPath: /home/mpiuser/.ssh
numNodes: 1
template:
spec:
network:
publishNotReadyAddresses: true
replicatedJobs:
- name: launcher
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
name: node
securityContext:
runAsUser: 1000
- name: node
template:
spec:
template:
spec:
containers:
- args:
- -De
- -f
- /home/mpiuser/.sshd_config
command:
- /usr/sbin/sshd
image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
name: node
readinessProbe:
initialDelaySeconds: 5
tcpSocket:
port: 2222
securityContext:
runAsUser: 1000
successPolicy:
operator: All
targetReplicatedJobs:
- launcher
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: jax
name: jax-distributed
spec:
mlPolicy:
jax: {}
numNodes: 1
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: nvcr.io/nvidia/jax:25.10-py3
name: node
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: mlx
name: mlx-distributed
spec:
mlPolicy:
mpi:
mpiImplementation: OpenMPI
numProcPerNode: 1
runLauncherAsNode: true
sshAuthMountPath: /home/mpiuser/.ssh
numNodes: 1
template:
spec:
network:
publishNotReadyAddresses: true
replicatedJobs:
- name: launcher
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
name: node
securityContext:
runAsUser: 1000
- name: node
template:
spec:
template:
spec:
containers:
- args:
- -De
- -f
- /home/mpiuser/.sshd_config
command:
- /usr/sbin/sshd
image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
name: node
readinessProbe:
initialDelaySeconds: 5
tcpSocket:
port: 2222
securityContext:
runAsUser: 1000
successPolicy:
operator: All
targetReplicatedJobs:
- launcher
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: torch
name: torch-distributed
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime
name: node
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: torchtune
name: torchtune-llama3.2-1b
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: dataset-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://tatsu-lab/alpaca
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
name: dataset-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-1b
- name: model-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://meta-llama/Llama-3.2-1B-Instruct
image: ghcr.io/kubeflow/trainer/model-initializer:latest
name: model-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-1b
- dependsOn:
- name: dataset-initializer
status: Complete
- name: model-initializer
status: Complete
name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- command:
- tune
- run
- --rdzv_endpoint=localhost:29500
- full_finetune_distributed
- --config
- llama3_2/1B_full
- dataset=torchtune.datasets.instruct_dataset
- dataset.source=parquet
- dataset.data_dir=/workspace/dataset/data
- output_dir=/workspace/output
- tokenizer.path=/workspace/model/original/tokenizer.model
- checkpointer.checkpoint_dir=/workspace/model
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
name: node
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-1b
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: torchtune
name: torchtune-llama3.2-3b
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: dataset-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://tatsu-lab/alpaca
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
name: dataset-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-3b
- name: model-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://meta-llama/Llama-3.2-3B-Instruct
image: ghcr.io/kubeflow/trainer/model-initializer:latest
name: model-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-3b
- dependsOn:
- name: dataset-initializer
status: Complete
- name: model-initializer
status: Complete
name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- command:
- tune
- run
- --rdzv_endpoint=localhost:29500
- full_finetune_distributed
- --config
- llama3_2/3B_full
- dataset=torchtune.datasets.instruct_dataset
- dataset.source=parquet
- dataset.data_dir=/workspace/dataset/data
- output_dir=/workspace/output
- tokenizer.path=/workspace/model/original/tokenizer.model
- checkpointer.checkpoint_dir=/workspace/model
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
name: node
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-llama3.2-3b
---
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
labels:
trainer.kubeflow.org/framework: torchtune
name: torchtune-qwen2.5-1.5b
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: dataset-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://tatsu-lab/alpaca
image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
name: dataset-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-qwen2.5-1.5b
- name: model-initializer
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
spec:
template:
spec:
containers:
- env:
- name: STORAGE_URI
value: hf://Qwen/Qwen2.5-1.5B-Instruct
image: ghcr.io/kubeflow/trainer/model-initializer:latest
name: model-initializer
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-qwen2.5-1.5b
- dependsOn:
- name: dataset-initializer
status: Complete
- name: model-initializer
status: Complete
name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- command:
- tune
- run
- --rdzv_endpoint=localhost:29500
- full_finetune_distributed
- --config
- qwen2_5/1.5B_full
- dataset=torchtune.datasets.instruct_dataset
- dataset.source=parquet
- dataset.data_dir=/workspace/dataset/data
- output_dir=/workspace/output
- tokenizer.path=/workspace/model/vocab.json
- tokenizer.merges_file=/workspace/model/merges.txt
- checkpointer.checkpoint_dir=/workspace/model
image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
name: node
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- mountPath: /workspace
name: initializer
volumes:
- name: initializer
persistentVolumeClaim:
claimName: torchtune-qwen2.5-1.5b