kubeflow_trainer_with_volcano/README.md

Volcano + Kubeflow Trainer 설치 매뉴얼

멀티노드 GPU 분산 학습을 위한 Volcano Scheduler와 Kubeflow Trainer v2 설치 가이드.

환경 정보

항목	값
Kubernetes	v1.34.3
Helm	v4.0.4
노드 구성	Master 3 + Worker 13 (총 16)
Volcano	v1.14.0
Kubeflow Trainer	sha-48e7a93
JobSet (의존성)	v0.10.1

디렉토리 구조

kubeflow_trainer_with_volcano/
├── .env                                          # 환경변수 (git 제외)
├── .gitignore
├── README.md                                     # 이 문서
├── charts/
│   ├── volcano/
│   │   └── volcano-1.14.0.tgz                    # Volcano Helm chart
│   └── kubeflow-trainer-0.0.0-sha-48e7a93.tgz    # Kubeflow Trainer Helm chart
├── manifests/
│   └── kubeflow-trainer/
│       └── runtimes/
│           └── runtimes.yaml                     # ClusterTrainingRuntime manifests
└── configs/
    ├── volcano-values.yaml                       # Volcano 커스텀 values
    ├── kubeflow-trainer-values.yaml              # Kubeflow Trainer 커스텀 values
    └── volcano-trainjob-integration.yaml         # Volcano-Trainer 연동 설정 샘플

사전 요구사항

• Kubernetes 클러스터 (v1.26+)
• Helm v3 이상
• kubectl 클러스터 접근 설정 완료
• GPU 노드에 NVIDIA device plugin 설치 완료
• 클러스터에 기존 Volcano/Kubeflow CRD가 없는 상태

# 사전 확인
kubectl version
helm version
kubectl get nodes
nvidia-smi  # GPU 노드에서 확인

---

Task 2: Volcano 설치

Volcano는 Kubernetes를 위한 배치 스케줄링 시스템으로, gang scheduling을 통해 분산 학습 Pod들이 동시에 스케줄링되도록 보장합니다.

2.1 네임스페이스 생성

kubectl create namespace volcano-system

2.2 Helm으로 Volcano 설치 (로컬 chart 사용)

helm install volcano charts/volcano/volcano-1.14.0.tgz \
  -n volcano-system \
  -f configs/volcano-values.yaml \
  --wait --timeout 5m

2.3 설치 확인

# Pod 상태 확인
kubectl get pods -n volcano-system

# 기대 출력:
# volcano-admission-xxxxx     1/1   Running
# volcano-controllers-xxxxx   1/1   Running
# volcano-scheduler-xxxxx     1/1   Running

# CRD 확인
kubectl get crd | grep volcano

# 기대 출력:
# commands.bus.volcano.sh
# jobs.batch.volcano.sh
# podgroups.scheduling.volcano.sh
# queues.scheduling.volcano.sh

# 기본 Queue 확인
kubectl get queue -n volcano-system

2.4 Volcano 삭제 (필요시)

helm uninstall volcano -n volcano-system
kubectl delete namespace volcano-system

---

Task 3: Kubeflow Trainer 설치

Kubeflow Trainer v2는 TrainJob CRD를 통해 분산 학습 워크로드를 관리합니다.

3.1 네임스페이스 생성

kubectl create namespace kubeflow-trainer

3.2 Helm으로 Kubeflow Trainer 설치 (로컬 chart 사용)

helm install kubeflow-trainer charts/kubeflow-trainer-0.0.0-sha-48e7a93.tgz \
  -n kubeflow-trainer \
  -f configs/kubeflow-trainer-values.yaml \
  --wait --timeout 5m

3.3 ClusterTrainingRuntime 설치

kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml

3.4 설치 확인

# Trainer controller 확인
kubectl get pods -n kubeflow-trainer

# 기대 출력:
# kubeflow-trainer-controller-manager-xxxxx   1/1   Running

# CRD 확인
kubectl get crd | grep trainer

# 기대 출력:
# clustertrainingruntimes.trainer.kubeflow.org
# trainjobs.trainer.kubeflow.org
# trainingruntimes.trainer.kubeflow.org

# ClusterTrainingRuntime 확인
kubectl get clustertrainingruntimes

# 기대 출력:
# deepspeed-distributed
# torch-distributed
# ...

3.5 Kubeflow Trainer 삭제 (필요시)

kubectl delete -f manifests/kubeflow-trainer/runtimes/runtimes.yaml
helm uninstall kubeflow-trainer -n kubeflow-trainer
kubectl delete namespace kubeflow-trainer

---

Task 4: Volcano ↔ Kubeflow Trainer 연동 설정

Volcano 스케줄러를 Kubeflow TrainJob과 연동하여 gang scheduling, queue 기반 리소스 관리를 활성화합니다.

4.1 Training Queue 생성

kubectl apply -f - <<EOF
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: training-queue
spec:
  weight: 1
  reclaimable: true
  capability:
    cpu: "100"
    memory: "512Gi"
    nvidia.com/gpu: "16"
EOF

Queue 상태 확인:

kubectl get queue training-queue -o yaml

4.2 PriorityClass 생성

kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: training-priority
value: 100
globalDefault: false
description: "Priority class for distributed training workloads"
EOF

4.3 Volcano 연동 ClusterTrainingRuntime 생성

Volcano 스케줄러를 사용하는 커스텀 런타임을 생성합니다.

kubectl apply -f - <<EOF
apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: torch-distributed-volcano
  labels:
    trainer.kubeflow.org/framework: torch
spec:
  mlPolicy:
    torch:
      numProcPerNode: 1
    numNodes: 1
  template:
    spec:
      replicatedJobs:
      - name: node
        template:
          spec:
            template:
              metadata:
                annotations:
                  scheduling.volcano.sh/queue-name: training-queue
              spec:
                schedulerName: volcano
                containers:
                - name: node
                  image: ghcr.io/kubeflow/trainer/torch-runtime:latest
                  resources:
                    requests:
                      nvidia.com/gpu: "1"
                    limits:
                      nvidia.com/gpu: "1"
EOF

또는 전체 연동 설정 샘플 적용:

kubectl apply -f configs/volcano-trainjob-integration.yaml

4.4 연동 테스트

Volcano 스케줄러를 사용하는 TrainJob 제출:

kubectl apply -f - <<EOF
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: test-volcano-integration
  namespace: default
spec:
  runtimeRef:
    name: torch-distributed-volcano
  trainer:
    image: ghcr.io/kubeflow/trainer/torch-runtime:latest
    numNodes: 2
    numProcPerNode: "1"
    resourcesPerNode:
      requests:
        nvidia.com/gpu: "1"
      limits:
        nvidia.com/gpu: "1"
EOF

4.5 연동 확인

# TrainJob 상태 확인
kubectl get trainjob test-volcano-integration

# PodGroup 자동 생성 확인 (Volcano가 관리)
kubectl get podgroup

# Pod 스케줄러 확인 (schedulerName이 volcano인지)
kubectl get pods -l trainer.kubeflow.org/trainjob-name=test-volcano-integration \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.schedulerName}{"\n"}{end}'

# Volcano Queue 사용량 확인
kubectl get queue training-queue

---

검증 체크리스트

• Volcano Pod 3개 Running (admission, controller, scheduler)
• Volcano CRD 생성됨 (jobs, podgroups, queues 등)
• Kubeflow Trainer controller Running
• Kubeflow Trainer CRD 생성됨 (trainjobs, trainingruntimes 등)
• ClusterTrainingRuntime 목록 확인 (deepspeed-distributed, torch-distributed 등)
• Training Queue 생성됨
• 테스트 TrainJob이 Volcano 스케줄러로 스케줄링됨

---

트러블슈팅

Volcano 관련

Volcano admission webhook 타임아웃

# webhook 인증서 확인
kubectl get secret volcano-admission-secret -n volcano-system
# admission pod 로그 확인
kubectl logs -n volcano-system -l app=volcano-admission

Pod가 Pending 상태로 유지

# PodGroup 상태 확인 - minMember 충족 여부
kubectl describe podgroup <podgroup-name>
# Queue 리소스 capacity 확인
kubectl get queue training-queue -o yaml
# Volcano scheduler 로그 확인
kubectl logs -n volcano-system -l app=volcano-scheduler

Kubeflow Trainer 관련

TrainJob이 생성되지 않음

# Trainer controller 로그 확인
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer
# TrainJob 이벤트 확인
kubectl describe trainjob <trainjob-name>

ClusterTrainingRuntime을 찾을 수 없음

# 런타임 목록 확인
kubectl get clustertrainingruntimes
# 런타임이 없으면 재설치
kubectl apply -f manifests/kubeflow-trainer/runtimes/runtimes.yaml

연동 관련

TrainJob Pod가 default-scheduler로 스케줄링됨

• schedulerName: volcano 설정이 ClusterTrainingRuntime의 pod spec에 있는지 확인
• TrainJob이 올바른 runtimeRef를 참조하는지 확인

PodGroup이 자동 생성되지 않음

• Volcano controller 로그 확인
• Pod에 scheduling.volcano.sh/queue-name annotation이 있는지 확인

로그 수집

# 전체 로그 수집
kubectl logs -n volcano-system -l app=volcano-scheduler --tail=100
kubectl logs -n volcano-system -l app=volcano-controller --tail=100
kubectl logs -n volcano-system -l app=volcano-admission --tail=100
kubectl logs -n kubeflow-trainer -l app.kubernetes.io/name=kubeflow-trainer --tail=100