vm 내에 k8s cluster 설치

https://daaa0555.tistory.com/478

https://kubevirt.io/user-guide/storage/disks_and_volumes/

PVC 생성

vm에 연결할 pvc를 생성한다.

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: writable-disk
  namespace: kubevirt
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi  # 원하는 크기로 설정
EOF

SSH Public key 생성

kubectl create secret generic my-pub-key --from-file=key1=id_rsa.pub -n kubevirt

KubeVirt로 VM 생성

cloudInitNoCloud의 runcmd를 통해 pvc 볼륨을 자동으로 /mnt/shared 에 마운트해준다.

kubectl apply -f - <<EOF
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: nested-k8s-vm
  namespace: kubevirt
spec:
  running: true
  template:
    spec:
      domain:
        devices:
          disks:
          - disk:
              bus: virtio
            name: containerdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          - disk:                  # 추가된 쓰기 가능한 디스크
              bus: virtio
            name: writable-disk
        resources:
          requests:
            memory: 4Gi
      terminationGracePeriodSeconds: 0
      accessCredentials:
      - sshPublicKey:
          source:
            secret:
              secretName: my-pub-key
          propagationMethod:
            noCloud: {}
      volumes:
      - name: containerdisk
        containerDisk:
          image: quay.io/containerdisks/ubuntu:22.04
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |
            #cloud-config
            password: ubuntu
            chpasswd: { expire: False }
            runcmd:
              - mkfs.ext4 /dev/vdc
              - mkdir -p /mnt/shared
              - mount /dev/vdc /mnt/shared
              - bash -c "echo '/dev/vdc /mnt/shared ext4 defaults 0 0' >> /etc/fstab"
      - name: writable-disk         # 추가된 디스크 볼륨
        persistentVolumeClaim:
          claimName: writable-disk

EOF

VM 접속

kubectl virt console nested-k8s-vm -n kubevirt

id와 password는 모두 ubuntu

K8s cluster 설치

kubespray를 이용하거나 아래와 같이 설치한다.

VM에 외부에서 접속하기

# Containerd 설치, 기본 설정값으로 config.toml 생성
sudo apt-get update
sudo apt-get install -y containerd
sudo systemctl enable containerd
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# config.toml 파일 수정, containerd 수정사항 적용 및 재실행
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

# kubelet, kubeadm, kubectl 설치
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL <https://pkgs.k8s.io/core:/stable:/v1.32/deb/Release.key> | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] <https://pkgs.k8s.io/core:/stable:/v1.32/deb/> /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

# kubelet 서비스 활성화
sudo systemctl enable --now kubelet

# crictl 설치
VERSION=$(curl -s <https://api.github.com/repos/kubernetes-sigs/cri-tools/releases/latest> | grep tag_name | cut -d '"' -f 4)
curl -LO <https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz>
sudo tar -zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin

# ip_forward를 활성화
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo sysctl -w net.ipv4.ip_forward=1

# Kubernetes 클러스터 초기화
sudo kubeadm init --pod-network-cidr=192.168.0.0/16

# kubeconfig 설정
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 네트워크 플러그인 설치 (예: Calico)
kubectl apply -f <https://docs.projectcalico.org/manifests/calico.yaml>

# [옵션] k9s 설치
K9S_VERSION=$(curl -s <https://api.github.com/repos/derailed/k9s/releases/latest> | grep -Po '"tag_name": "\K.*?(?=")')
curl -LO <https://github.com/derailed/k9s/releases/download/${K9S_VERSION}/k9s_Linux_amd64.tar.gz>
tar -zxvf k9s_Linux_amd64.tar.gz
sudo mv k9s /usr/local/bin/
rm k9s_Linux_amd64.tar.gz

VM에 외부에서 접근하게 하려면 service를 생성해야 한다.

kubectl apply -f - <<EOF 
apiVersion: v1
kind: Service
metadata:
  name: ssh-service
  namespace: kubevirt
spec:
  type: NodePort
  ports:
  - protocol: TCP
    port: 22
    targetPort: 22
  selector:
    vm.kubevirt.io/name: nested-k8s-vm
EOF

ssh로 접속해본다