commit 66e93c2035c7e070cbb8f6770bae49abda6fb3ac
Author: selee <selee@inje-private.com>
Date:   Mon Feb 9 15:21:10 2026 +0900

    Initial setup: Volcano + Kubeflow Trainer installation environment
    
    - Add Volcano Helm chart v1.14.0 (tgz)
    - Add Kubeflow Trainer Helm chart sha-48e7a93 (tgz)
    - Add ClusterTrainingRuntime manifests (runtimes.yaml)
    - Add custom values for Volcano (gang scheduling, binpack for GPU)
    - Add custom values for Kubeflow Trainer
    - Add Volcano-Trainer integration config sample (Queue, PodGroup, PriorityClass)
    - Add comprehensive installation manual (README.md)
    
    Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..4c49bd7
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1 @@
+.env
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..042b246
--- /dev/null
+++ b/README.md
@@ -0,0 +1,369 @@
+# Volcano + Kubeflow Trainer 설치 매뉴얼
+
+멀티노드 GPU 분산 학습을 위한 Volcano Scheduler와 Kubeflow Trainer v2 설치 가이드.
+
+## 환경 정보
+
+| 항목 | 값 |
+|------|-----|
+| Kubernetes | v1.34.3 |
+| Helm | v4.0.4 |
+| 노드 구성 | Master 3 + Worker 13 (총 16) |
+| Volcano | v1.14.0 |
+| Kubeflow Trainer | sha-48e7a93 |
+| JobSet (의존성) | v0.10.1 |
+
+## 디렉토리 구조
+
+```
+kubeflow_trainer_with_volcano/
+├── .env                                          # 환경변수 (git 제외)
+├── .gitignore
+├── README.md                                     # 이 문서
+├── charts/
+│   ├── volcano/
+│   │   └── volcano-1.14.0.tgz                    # Volcano Helm chart
+│   └── kubeflow-trainer-0.0.0-sha-48e7a93.tgz    # Kubeflow Trainer Helm chart
+├── manifests/
+│   └── kubeflow-trainer/
+│       └── runtimes/
+│           └── runtimes.yaml                     # ClusterTrainingRuntime manifests
+└── configs/
+    ├── volcano-values.yaml                       # Volcano 커스텀 values
+    ├── kubeflow-trainer-values.yaml              # Kubeflow Trainer 커스텀 values
+    └── volcano-trainjob-integration.yaml         # Volcano-Trainer 연동 설정 샘플
+```
+
+## 사전 요구사항
+
+- Kubernetes 클러스터 (v1.26+)
+- Helm v3 이상
+- `kubectl` 클러스터 접근 설정 완료
+- GPU 노드에 NVIDIA device plugin 설치 완료
+- 클러스터에 기존 Volcano/Kubeflow CRD가 없는 상태
+
+```bash
+# 사전 확인
+kubectl version
+helm version
+kubectl get nodes
+nvidia-smi  # GPU 노드에서 확인
+```
+
+---
+
+## Task 2: Volcano 설치
+
+Volcano는 Kubernetes를 위한 배치 스케줄링 시스템으로, gang scheduling을 통해 분산 학습 Pod들이 동시에 스케줄링되도록 보장합니다.
+
+### 2.1 네임스페이스 생성
+
+```bash
+kubectl create namespace volcano-system
+```
+
+### 2.2 Helm으로 Volcano 설치 (로컬 chart 사용)
+
+```bash
+helm install volcano charts/volcano/volcano-1.14.0.tgz \
+  -n volcano-system \
+  -f configs/volcano-values.yaml \
+  --wait --timeout 5m
+```
+
+### 2.3 설치 확인
+
+```bash
+# Pod 상태 확인
+kubectl get pods -n volcano-system
+
+# 기대 출력:
+# volcano-admission-xxxxx     1/1   Running
+# volcano-controllers-xxxxx   1/1   Running
+# volcano-scheduler-xxxxx     1/1   Running
+
+# CRD 확인
+kubectl get crd | grep volcano
+
+# 기대 출력:
+# commands.bus.volcano.sh
+# jobs.batch.volcano.sh
+# podgroups.scheduling.volcano.sh
+# queues.scheduling.volcano.sh
+
+# 기본 Queue 확인
+kubectl get queue -n volcano-system
+```
+
+### 2.4 Volcano 삭제 (필요시)
+
+```bash
+helm uninstall volcano -n volcano-system
+kubectl delete namespace volcano-system
+```
+
+---
+
+## Task 3: Kubeflow Trainer 설치
+
+Kubeflow Trainer v2는 TrainJob CRD를 통해 분산 학습 워크로드를 관리합니다.
+
+### 3.1 네임스페이스 생성
+
+```bash
+kubectl create namespace kubeflow-trainer
+```
+
+### 3.2 Helm으로 Kubeflow Trainer 설치 (로컬 chart 사용)
+
+```bash
+helm install kubeflow-trainer charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz \
+  -n kubeflow-trainer \
+  -f configs/kubeflow-trainer-values.yaml \
+  --wait --timeout 5m
+```
+
+### 3.3 ClusterTrainingRuntime 설치
+
+```bash
+kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
+```
+
+### 3.4 설치 확인
+
+```bash
+# Trainer controller 확인
+kubectl get pods -n kubeflow-trainer
+
+# 기대 출력:
+# kubeflow-trainer-controller-manager-xxxxx   1/1   Running
+
+# CRD 확인
+kubectl get crd | grep trainer
+
+# 기대 출력:
+# clustertrainingruntimes.trainer.kubeflow.org
+# trainjobs.trainer.kubeflow.org
+# trainingruntimes.trainer.kubeflow.org
+
+# ClusterTrainingRuntime 확인
+kubectl get clustertrainingruntimes
+
+# 기대 출력:
+# deepspeed-distributed
+# torch-distributed
+# ...
+```
+
+### 3.5 Kubeflow Trainer 삭제 (필요시)
+
+```bash
+kubectl delete -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
+helm uninstall kubeflow-trainer -n kubeflow-trainer
+kubectl delete namespace kubeflow-trainer
+```
+
+---
+
+## Task 4: Volcano ↔ Kubeflow Trainer 연동 설정
+
+Volcano 스케줄러를 Kubeflow TrainJob과 연동하여 gang scheduling, queue 기반 리소스 관리를 활성화합니다.
+
+### 4.1 Training Queue 생성
+
+```bash
+kubectl apply -f - <<EOF
+apiVersion: scheduling.volcano.sh/v1beta1
+kind: Queue
+metadata:
+  name: training-queue
+spec:
+  weight: 1
+  reclaimable: true
+  capability:
+    cpu: "100"
+    memory: "512Gi"
+    nvidia.com/gpu: "16"
+EOF
+```
+
+Queue 상태 확인:
+```bash
+kubectl get queue training-queue -o yaml
+```
+
+### 4.2 PriorityClass 생성
+
+```bash
+kubectl apply -f - <<EOF
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata:
+  name: training-priority
+value: 100
+globalDefault: false
+description: "Priority class for distributed training workloads"
+EOF
+```
+
+### 4.3 Volcano 연동 ClusterTrainingRuntime 생성
+
+Volcano 스케줄러를 사용하는 커스텀 런타임을 생성합니다.
+
+```bash
+kubectl apply -f - <<EOF
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  name: torch-distributed-volcano
+  labels:
+    trainer.kubeflow.org/framework: torch
+spec:
+  mlPolicy:
+    torch:
+      numProcPerNode: 1
+    numNodes: 1
+  template:
+    spec:
+      replicatedJobs:
+      - name: node
+        template:
+          spec:
+            template:
+              metadata:
+                annotations:
+                  scheduling.volcano.sh/queue-name: training-queue
+              spec:
+                schedulerName: volcano
+                containers:
+                - name: node
+                  image: ghcr.io/kubeflow/trainer/torch-runtime:latest
+                  resources:
+                    requests:
+                      nvidia.com/gpu: "1"
+                    limits:
+                      nvidia.com/gpu: "1"
+EOF
+```
+
+또는 전체 연동 설정 샘플 적용:
+```bash
+kubectl apply -f configs/volcano-trainjob-integration.yaml
+```
+
+### 4.4 연동 테스트
+
+Volcano 스케줄러를 사용하는 TrainJob 제출:
+
+```bash
+kubectl apply -f - <<EOF
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: TrainJob
+metadata:
+  name: test-volcano-integration
+  namespace: default
+spec:
+  runtimeRef:
+    name: torch-distributed-volcano
+  trainer:
+    image: ghcr.io/kubeflow/trainer/torch-runtime:latest
+    numNodes: 2
+    numProcPerNode: "1"
+    resourcesPerNode:
+      requests:
+        nvidia.com/gpu: "1"
+      limits:
+        nvidia.com/gpu: "1"
+EOF
+```
+
+### 4.5 연동 확인
+
+```bash
+# TrainJob 상태 확인
+kubectl get trainjob test-volcano-integration
+
+# PodGroup 자동 생성 확인 (Volcano가 관리)
+kubectl get podgroup
+
+# Pod 스케줄러 확인 (schedulerName이 volcano인지)
+kubectl get pods -l trainer.kubeflow.org/trainjob-name=test-volcano-integration \
+  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.schedulerName}{"\n"}{end}'
+
+# Volcano Queue 사용량 확인
+kubectl get queue training-queue
+```
+
+---
+
+## 검증 체크리스트
+
+- [ ] Volcano Pod 3개 Running (admission, controller, scheduler)
+- [ ] Volcano CRD 생성됨 (jobs, podgroups, queues 등)
+- [ ] Kubeflow Trainer controller Running
+- [ ] Kubeflow Trainer CRD 생성됨 (trainjobs, trainingruntimes 등)
+- [ ] ClusterTrainingRuntime 목록 확인 (deepspeed-distributed, torch-distributed 등)
+- [ ] Training Queue 생성됨
+- [ ] 테스트 TrainJob이 Volcano 스케줄러로 스케줄링됨
+
+---
+
+## 트러블슈팅
+
+### Volcano 관련
+
+**Volcano admission webhook 타임아웃**
+```bash
+# webhook 인증서 확인
+kubectl get secret volcano-admission-secret -n volcano-system
+# admission pod 로그 확인
+kubectl logs -n volcano-system -l app=volcano-admission
+```
+
+**Pod가 Pending 상태로 유지**
+```bash
+# PodGroup 상태 확인 - minMember 충족 여부
+kubectl describe podgroup <podgroup-name>
+# Queue 리소스 capacity 확인
+kubectl get queue training-queue -o yaml
+# Volcano scheduler 로그 확인
+kubectl logs -n volcano-system -l app=volcano-scheduler
+```
+
+### Kubeflow Trainer 관련
+
+**TrainJob이 생성되지 않음**
+```bash
+# Trainer controller 로그 확인
+kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer
+# TrainJob 이벤트 확인
+kubectl describe trainjob <trainjob-name>
+```
+
+**ClusterTrainingRuntime을 찾을 수 없음**
+```bash
+# 런타임 목록 확인
+kubectl get clustertrainingruntimes
+# 런타임이 없으면 재설치
+kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
+```
+
+### 연동 관련
+
+**TrainJob Pod가 default-scheduler로 스케줄링됨**
+- `schedulerName: volcano` 설정이 ClusterTrainingRuntime의 pod spec에 있는지 확인
+- TrainJob이 올바른 runtimeRef를 참조하는지 확인
+
+**PodGroup이 자동 생성되지 않음**
+- Volcano controller 로그 확인
+- Pod에 `scheduling.volcano.sh/queue-name` annotation이 있는지 확인
+
+### 로그 수집
+
+```bash
+# 전체 로그 수집
+kubectl logs -n volcano-system -l app=volcano-scheduler --tail=100
+kubectl logs -n volcano-system -l app=volcano-controller --tail=100
+kubectl logs -n volcano-system -l app=volcano-admission --tail=100
+kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer --tail=100
+```
diff --git a/charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz b/charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz
new file mode 100644
index 0000000..91d2bfd
Binary files /dev/null and b/charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz differ
diff --git a/charts/volcano/volcano-1.14.0.tgz b/charts/volcano/volcano-1.14.0.tgz
new file mode 100644
index 0000000..2885c44
Binary files /dev/null and b/charts/volcano/volcano-1.14.0.tgz differ
diff --git a/configs/kubeflow-trainer-values.yaml b/configs/kubeflow-trainer-values.yaml
new file mode 100644
index 0000000..4bd3bf3
--- /dev/null
+++ b/configs/kubeflow-trainer-values.yaml
@@ -0,0 +1,52 @@
+# Kubeflow Trainer Helm Chart Custom Values
+# Chart version: 0.0.0-sha-48e7a93
+# Reference: charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz (default values)
+
+# JobSet dependency - install if not already present
+jobset:
+  install: true
+  fullnameOverride: jobset
+
+image:
+  registry: ghcr.io
+  repository: kubeflow/trainer/trainer-controller-manager
+  pullPolicy: IfNotPresent
+
+manager:
+  replicas: 1
+  resources:
+    limits:
+      cpu: 200m
+      memory: 512Mi
+    requests:
+      cpu: 100m
+      memory: 256Mi
+  securityContext:
+    allowPrivilegeEscalation: false
+    runAsNonRoot: true
+    capabilities:
+      drop:
+      - ALL
+    seccompProfile:
+      type: RuntimeDefault
+  config:
+    health:
+      healthProbeBindAddress: ":8081"
+    metrics:
+      bindAddress: ":8443"
+      secureServing: true
+    webhook:
+      port: 9443
+    leaderElection:
+      leaderElect: true
+      resourceName: "trainer.kubeflow.org"
+    controller:
+      groupKindConcurrency:
+        trainJob: 5
+        trainingRuntime: 1
+        clusterTrainingRuntime: 1
+    certManagement:
+      enable: true
+
+webhook:
+  failurePolicy: Fail
diff --git a/configs/volcano-trainjob-integration.yaml b/configs/volcano-trainjob-integration.yaml
new file mode 100644
index 0000000..5960612
--- /dev/null
+++ b/configs/volcano-trainjob-integration.yaml
@@ -0,0 +1,107 @@
+# Volcano <-> Kubeflow Trainer Integration Configuration Sample
+# This file contains examples for integrating Volcano scheduler with Kubeflow TrainJob.
+
+---
+# 1. Volcano Queue for training workloads
+apiVersion: scheduling.volcano.sh/v1beta1
+kind: Queue
+metadata:
+  name: training-queue
+spec:
+  weight: 1
+  reclaimable: true
+  capability:
+    # Adjust based on your cluster capacity
+    cpu: "100"
+    memory: "512Gi"
+    nvidia.com/gpu: "16"
+
+---
+# 2. Example TrainJob with Volcano PodGroup scheduling
+# TrainJob that uses Volcano as the scheduler via podGroupPolicy
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: TrainJob
+metadata:
+  name: example-distributed-training
+  namespace: default
+spec:
+  runtimeRef:
+    name: torch-distributed
+  trainer:
+    image: docker.io/your-org/your-training-image:latest
+    numNodes: 2
+    numProcPerNode: "auto"
+    resourcesPerNode:
+      requests:
+        nvidia.com/gpu: "4"
+        cpu: "16"
+        memory: "64Gi"
+      limits:
+        nvidia.com/gpu: "4"
+        cpu: "16"
+        memory: "64Gi"
+
+---
+# 3. Volcano PodGroup for gang scheduling (auto-created by integration)
+# This is a reference example - Volcano controller auto-creates PodGroups
+# when scheduler annotation is set.
+apiVersion: scheduling.volcano.sh/v1beta1
+kind: PodGroup
+metadata:
+  name: example-distributed-training
+  namespace: default
+spec:
+  # MinMember ensures all pods are scheduled together (gang scheduling)
+  minMember: 2
+  queue: training-queue
+  priorityClassName: normal-priority
+  minResources:
+    cpu: "32"
+    memory: "128Gi"
+    nvidia.com/gpu: "8"
+
+---
+# 4. PriorityClass for training workloads
+apiVersion: scheduling.k8s.io/v1
+kind: PriorityClass
+metadata:
+  name: training-priority
+value: 100
+globalDefault: false
+description: "Priority class for distributed training workloads"
+
+---
+# 5. ClusterTrainingRuntime with Volcano scheduler annotation
+# Use this runtime to automatically schedule TrainJobs with Volcano
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  name: torch-distributed-volcano
+  labels:
+    trainer.kubeflow.org/framework: torch
+spec:
+  mlPolicy:
+    torch:
+      numProcPerNode: 1
+    numNodes: 1
+  template:
+    spec:
+      replicatedJobs:
+      - name: node
+        template:
+          spec:
+            template:
+              metadata:
+                annotations:
+                  # Use Volcano as the scheduler
+                  scheduling.volcano.sh/queue-name: training-queue
+              spec:
+                schedulerName: volcano
+                containers:
+                - name: node
+                  image: ghcr.io/kubeflow/trainer/torch-runtime:latest
+                  resources:
+                    requests:
+                      nvidia.com/gpu: "1"
+                    limits:
+                      nvidia.com/gpu: "1"
diff --git a/configs/volcano-values.yaml b/configs/volcano-values.yaml
new file mode 100644
index 0000000..27de39a
--- /dev/null
+++ b/configs/volcano-values.yaml
@@ -0,0 +1,52 @@
+# Volcano Helm Chart Custom Values
+# Chart version: 1.14.0
+# Reference: charts/volcano/volcano-1.14.0.tgz (default values)
+
+basic:
+  image_pull_policy: "IfNotPresent"
+  image_tag_version: "v1.14.0"
+
+custom:
+  # Enable admission webhook
+  admission_enable: true
+  admission_replicas: 1
+
+  # Enable controller
+  controller_enable: true
+  controller_replicas: 1
+
+  # Enable scheduler
+  scheduler_enable: true
+  scheduler_replicas: 1
+
+  # Enable leader election for HA
+  leader_elect_enable: false
+
+  # Scheduler config override for GPU workloads
+  # Includes gang scheduling, binpack for GPU locality, and topology-aware scheduling
+  scheduler_config_override: |
+    actions: "enqueue, allocate, backfill"
+    tiers:
+    - plugins:
+      - name: priority
+      - name: gang
+        enablePreemptable: false
+      - name: conformance
+    - plugins:
+      - name: overcommit
+      - name: drf
+        enablePreemptable: false
+      - name: predicates
+      - name: proportion
+      - name: nodeorder
+      - name: binpack
+        arguments:
+          binpack.weight: 10
+          binpack.resources: nvidia.com/gpu
+          binpack.resources.nvidia.com/gpu.weight: 5
+
+  # API rate limits
+  scheduler_kube_api_qps: 2000
+  scheduler_kube_api_burst: 2000
+  controller_kube_api_qps: 50
+  controller_kube_api_burst: 100
diff --git a/manifests/kubeflow-trainer/runtimes/runtimes.yaml b/manifests/kubeflow-trainer/runtimes/runtimes.yaml
new file mode 100644
index 0000000..be4d80f
--- /dev/null
+++ b/manifests/kubeflow-trainer/runtimes/runtimes.yaml
@@ -0,0 +1,453 @@
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: deepspeed
+  name: deepspeed-distributed
+spec:
+  mlPolicy:
+    mpi:
+      mpiImplementation: OpenMPI
+      numProcPerNode: 1
+      runLauncherAsNode: true
+      sshAuthMountPath: /home/mpiuser/.ssh
+    numNodes: 1
+  template:
+    spec:
+      network:
+        publishNotReadyAddresses: true
+      replicatedJobs:
+      - name: launcher
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
+                  name: node
+                  securityContext:
+                    runAsUser: 1000
+      - name: node
+        template:
+          spec:
+            template:
+              spec:
+                containers:
+                - args:
+                  - -De
+                  - -f
+                  - /home/mpiuser/.sshd_config
+                  command:
+                  - /usr/sbin/sshd
+                  image: ghcr.io/kubeflow/trainer/deepspeed-runtime:latest
+                  name: node
+                  readinessProbe:
+                    initialDelaySeconds: 5
+                    tcpSocket:
+                      port: 2222
+                  securityContext:
+                    runAsUser: 1000
+      successPolicy:
+        operator: All
+        targetReplicatedJobs:
+        - launcher
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: jax
+  name: jax-distributed
+spec:
+  mlPolicy:
+    jax: {}
+    numNodes: 1
+  template:
+    spec:
+      replicatedJobs:
+      - name: node
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - image: nvcr.io/nvidia/jax:25.10-py3
+                  name: node
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: mlx
+  name: mlx-distributed
+spec:
+  mlPolicy:
+    mpi:
+      mpiImplementation: OpenMPI
+      numProcPerNode: 1
+      runLauncherAsNode: true
+      sshAuthMountPath: /home/mpiuser/.ssh
+    numNodes: 1
+  template:
+    spec:
+      network:
+        publishNotReadyAddresses: true
+      replicatedJobs:
+      - name: launcher
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
+                  name: node
+                  securityContext:
+                    runAsUser: 1000
+      - name: node
+        template:
+          spec:
+            template:
+              spec:
+                containers:
+                - args:
+                  - -De
+                  - -f
+                  - /home/mpiuser/.sshd_config
+                  command:
+                  - /usr/sbin/sshd
+                  image: ghcr.io/kubeflow/trainer/mlx-runtime:latest
+                  name: node
+                  readinessProbe:
+                    initialDelaySeconds: 5
+                    tcpSocket:
+                      port: 2222
+                  securityContext:
+                    runAsUser: 1000
+      successPolicy:
+        operator: All
+        targetReplicatedJobs:
+        - launcher
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: torch
+  name: torch-distributed
+spec:
+  mlPolicy:
+    numNodes: 1
+    torch:
+      numProcPerNode: auto
+  template:
+    spec:
+      replicatedJobs:
+      - name: node
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - image: pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime
+                  name: node
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: torchtune
+  name: torchtune-llama3.2-1b
+spec:
+  mlPolicy:
+    numNodes: 1
+    torch:
+      numProcPerNode: auto
+  template:
+    spec:
+      replicatedJobs:
+      - name: dataset-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://tatsu-lab/alpaca
+                  image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
+                  name: dataset-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-1b
+      - name: model-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://meta-llama/Llama-3.2-1B-Instruct
+                  image: ghcr.io/kubeflow/trainer/model-initializer:latest
+                  name: model-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-1b
+      - dependsOn:
+        - name: dataset-initializer
+          status: Complete
+        - name: model-initializer
+          status: Complete
+        name: node
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - command:
+                  - tune
+                  - run
+                  - --rdzv_endpoint=localhost:29500
+                  - full_finetune_distributed
+                  - --config
+                  - llama3_2/1B_full
+                  - dataset=torchtune.datasets.instruct_dataset
+                  - dataset.source=parquet
+                  - dataset.data_dir=/workspace/dataset/data
+                  - output_dir=/workspace/output
+                  - tokenizer.path=/workspace/model/original/tokenizer.model
+                  - checkpointer.checkpoint_dir=/workspace/model
+                  image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
+                  name: node
+                  resources:
+                    limits:
+                      nvidia.com/gpu: 2
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-1b
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: torchtune
+  name: torchtune-llama3.2-3b
+spec:
+  mlPolicy:
+    numNodes: 1
+    torch:
+      numProcPerNode: auto
+  template:
+    spec:
+      replicatedJobs:
+      - name: dataset-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://tatsu-lab/alpaca
+                  image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
+                  name: dataset-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-3b
+      - name: model-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://meta-llama/Llama-3.2-3B-Instruct
+                  image: ghcr.io/kubeflow/trainer/model-initializer:latest
+                  name: model-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-3b
+      - dependsOn:
+        - name: dataset-initializer
+          status: Complete
+        - name: model-initializer
+          status: Complete
+        name: node
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - command:
+                  - tune
+                  - run
+                  - --rdzv_endpoint=localhost:29500
+                  - full_finetune_distributed
+                  - --config
+                  - llama3_2/3B_full
+                  - dataset=torchtune.datasets.instruct_dataset
+                  - dataset.source=parquet
+                  - dataset.data_dir=/workspace/dataset/data
+                  - output_dir=/workspace/output
+                  - tokenizer.path=/workspace/model/original/tokenizer.model
+                  - checkpointer.checkpoint_dir=/workspace/model
+                  image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
+                  name: node
+                  resources:
+                    limits:
+                      nvidia.com/gpu: 2
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-llama3.2-3b
+---
+apiVersion: trainer.kubeflow.org/v1alpha1
+kind: ClusterTrainingRuntime
+metadata:
+  labels:
+    trainer.kubeflow.org/framework: torchtune
+  name: torchtune-qwen2.5-1.5b
+spec:
+  mlPolicy:
+    numNodes: 1
+    torch:
+      numProcPerNode: auto
+  template:
+    spec:
+      replicatedJobs:
+      - name: dataset-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: dataset-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://tatsu-lab/alpaca
+                  image: ghcr.io/kubeflow/trainer/dataset-initializer:latest
+                  name: dataset-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-qwen2.5-1.5b
+      - name: model-initializer
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: model-initializer
+          spec:
+            template:
+              spec:
+                containers:
+                - env:
+                  - name: STORAGE_URI
+                    value: hf://Qwen/Qwen2.5-1.5B-Instruct
+                  image: ghcr.io/kubeflow/trainer/model-initializer:latest
+                  name: model-initializer
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-qwen2.5-1.5b
+      - dependsOn:
+        - name: dataset-initializer
+          status: Complete
+        - name: model-initializer
+          status: Complete
+        name: node
+        template:
+          metadata:
+            labels:
+              trainer.kubeflow.org/trainjob-ancestor-step: trainer
+          spec:
+            template:
+              spec:
+                containers:
+                - command:
+                  - tune
+                  - run
+                  - --rdzv_endpoint=localhost:29500
+                  - full_finetune_distributed
+                  - --config
+                  - qwen2_5/1.5B_full
+                  - dataset=torchtune.datasets.instruct_dataset
+                  - dataset.source=parquet
+                  - dataset.data_dir=/workspace/dataset/data
+                  - output_dir=/workspace/output
+                  - tokenizer.path=/workspace/model/vocab.json
+                  - tokenizer.merges_file=/workspace/model/merges.txt
+                  - checkpointer.checkpoint_dir=/workspace/model
+                  image: ghcr.io/kubeflow/trainer/torchtune-trainer:latest
+                  name: node
+                  resources:
+                    limits:
+                      nvidia.com/gpu: 2
+                  volumeMounts:
+                  - mountPath: /workspace
+                    name: initializer
+                volumes:
+                - name: initializer
+                  persistentVolumeClaim:
+                    claimName: torchtune-qwen2.5-1.5b