<div id="readability-page-1" class="page"><div>
 <a href="/m/category">카테고리 없음</a><h3>Kubespray를 활용한 Kubernetes 클러스터 구축</h3>
 </div><cite>김 정출</cite>
 2023. 5. 31. 13:28
 <div>
 <h2 data-ke-size="size26">Kubespray를 활용한 Kubernetes 클러스터 구축</h2>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1000" data-origin-height="1000">
 
 <img src="https://blog.kakaocdn.net/dn/Ef6mZ/btsiadNY87W/HlMPeObmtwUJDHhU6Gz6sK/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="195" height="195" data-origin-width="1000" data-origin-height="1000">
 
 
</figure>
설치하는 환경
<ul data-ke-list-type="disc">
<li>Master VM, Worker VM 3
<ul data-ke-list-type="disc">
<li>OS: Ubuntu 22.04</li>
<li>CPU: 4 core</li>
<li>Mem: 4096MB</li>
<li>N/W: Bridge</li>
</ul>
</li>
</ul>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1780" data-origin-height="1002">
 
 <img src="https://blog.kakaocdn.net/dn/bipym9/btsh9F48oDI/Rk9g3gzcdK5IvJOihQ8Jlk/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" data-origin-width="1780" data-origin-height="1002">
 
 
</figure>
Kubespray로 Kuberenetes 구축 가이드
<ul data-ke-list-type="disc">
<li><a href="https://kubernetes.io/ko/docs/setup/production-environment/tools/kubespray/">https://kubernetes.io/ko/docs/setup/production-environment/tools/kubespray/</a></li>
</ul>

Master 노드에서 pip3 설치를 진행합니다.
<pre id="code_1685507454353" data-ke-language="bash" data-ke-type="codeblock"><code>sudo apt install python3-pip</code></pre>

<h2>ssh key 복사</h2>
ssh key 복사를 진행합니다. ansible을 기본으로 사용하고 있어 master에서 ssh-keygen 이후에 worker vm 3대로 copy를 진행하면 됩니다.
<pre id="code_1685506656875" data-ke-language="bash" data-ke-type="codeblock"><code>ssh-keygen</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1087" data-origin-height="562">
 
 <img src="https://blog.kakaocdn.net/dn/rzov3/btsiaiBI4Px/EktTgjE99wkSv2WE1KE1D1/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="332" data-origin-width="1087" data-origin-height="562">
 
 
</figure>
<pre id="code_1685506663734" data-ke-language="bash" data-ke-type="codeblock"><code>ssh-copy-id 172.16.177.128 # master

ssh-copy-id 172.16.177.129 # worker1

ssh-copy-id 172.16.177.130 # worker2

ssh-copy-id 172.16.177.131 # worker3</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1089" data-origin-height="379">
 
 <img src="https://blog.kakaocdn.net/dn/bRw8zu/btsh4QlKXPk/klpWJDXKdTtfmyUNrVFiHK/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="224" data-origin-width="1089" data-origin-height="379">
 
 
</figure>
<h2>Kubespray 설치</h2>
kubespray 설치를 진행합니다.
<a href="https://github.com/kubernetes-sigs/kubespray">https://github.com/kubernetes-sigs/kubespray</a>
<pre id="code_1685506672152" data-ke-language="bash" data-ke-type="codeblock"><code>mkdir environment &amp;&amp; cd environment

git clone https://github.com/kubernetes-sigs/kubespray

git checkout release-2.22</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1086" data-origin-height="647">
 
 <img src="https://blog.kakaocdn.net/dn/cGoyaw/btsh4O9kfvW/7GuT6vQ7VKAOBKJ4FeHeQk/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="383" data-origin-width="1086" data-origin-height="647">
 
 
</figure>
<pre id="code_1685506680381" data-ke-language="bash" data-ke-type="codeblock"><code>sudo pip3 install -r requirements.txt

cp -r inventory/sample inventory/cluster

declare -a IPS=(172.16.177.129 172.16.177.130 172.16.177.131) # 설치할 Worker node들의 IP를 작성합니다.

CONFIG_FILE=inventory/cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1088" data-origin-height="583">
 
 <img src="https://blog.kakaocdn.net/dn/sE3Jb/btsiacIjZLE/BTnWI80mLPK7TlowFvKKJk/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="344" data-origin-width="1088" data-origin-height="583">
 
 
</figure>
생성된 config file을 확인해봅니다.
<pre id="code_1685506685602" data-ke-language="bash" data-ke-type="codeblock"><code>vi inventory/cluster/hosts.yaml</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1360" data-origin-height="1006">
 
 <img src="https://blog.kakaocdn.net/dn/Bf7Ga/btsh24SytIu/iejqSHXx4715uxKupBO3J1/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="475" data-origin-width="1360" data-origin-height="1006">
 
 
</figure>

addon을 수정하여 필요한 것들을 true로 변경합니다.
<pre id="code_1685506699209" data-ke-language="bash" data-ke-type="codeblock"><code>vi inventory/cluster/group_vars/k8s_cluster/addons.yml

---

# Kubernetes dashboard

# RBAC required. see docs/getting-started.md for access details.

dashboard_enabled: true



# Helm deployment

helm_enabled: true



# Metrics Server deployment

metrics_server_enabled: true

metrics_server_container_port: 10250

metrics_server_kubelet_insecure_tls: true

metrics_server_metric_resolution: 15s

metrics_server_kubelet_preferred_address_types: "InternalIP,ExternalIP,Hostname"

metrics_server_host_network: true

metrics_server_replicas: 1



# Local volume provisioner deployment

local_volume_provisioner_enabled: true



# Nginx ingress controller deployment

ingress_nginx_enabled: true



# Cert manager deployment

cert_manager_enabled: true</code></pre>
Kubernetes 클러스터 설치 진행
<pre id="code_1685506706400" data-ke-language="bash" data-ke-type="codeblock"><code>ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] cluster.yml</code></pre>
설치에 수십 분 정도 소요됩니다.
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1849" data-origin-height="675">
 
 <img src="https://blog.kakaocdn.net/dn/Ck25h/btsh2PnAJo0/eRcrGjeiYf8AewTKp1Qh00/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="235" data-origin-width="1849" data-origin-height="675">
 
 
</figure>

kube 설정 파일을 가져옵니다.
<pre id="code_1685506712791" data-ke-language="bash" data-ke-type="codeblock"><code>mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config</code></pre>
kubectl 명령어 자동 완성
<pre id="code_1685506729120" data-ke-language="bash" data-ke-type="codeblock"><code>echo "source &lt;(kubectl completion zsh)" &gt;&gt; ~/.zshrc</code></pre>
Alias 설정
<pre id="code_1685506757673" data-ke-language="bash" data-ke-type="codeblock"><code>vi ~/.zshrc

alias wk='watch kubectl get'

alias wka='watch kubectl get all'

alias wkgp='watch kubectl get pod'

alias k='kubectl'

alias kg='kubectl get'

alias ke='kubectl edit'

alias kdl='kubectl delete'

alias kd='kubectl describe'

alias kde='kubectl delete'

alias kes='kubectl edit service'

alias ked='kubectl edit deployment'

alias kef='kubectl edit statefulset'

alias kgns='kubectl get namespace'

alias kga='kubectl get all'

alias kgp='kubectl get pod'

alias kgd='kubectl get deployment'

alias kgf='kubectl get statefulset'

alias kgs='kubectl get service'

alias kgns='kubectl get ns'

alias kdp='kubectl describe pod'

alias kds='kubectl describe svc'

alias kdd='kubectl describe deployment'

alias kdf='kubectl describe statefulset'

alias kdnd='kubectl describe node'

alias klf='kubectl logs --follow'

alias kgnd='kubectl get node'

alias ksd='kubectl scale deployment'

alias ksud='kubectl scale deployment --replicas=1'

alias ksdd='kubectl scale deployment --replicas=0'

alias ksf='kubectl scale statefulset'

alias ksuf='kubectl scale statefulset -�~@~Treplicas=1'

alias ksdf='kubectl scale statefulset -�~@~Treplicas=0'

alias krd='kubectl rollout restart deployment'

alias krf='kubectl rollout restart statefulset'




source ~/.zshrc</code></pre>

Kubernetes 클러스터의 노드 정보를 확인합니다.
<pre id="code_1685506764899" data-ke-language="bash" data-ke-type="codeblock"><code>kubectl get node -o wide</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="2048" data-origin-height="228">
 
 <img src="https://blog.kakaocdn.net/dn/OehYE/btsh4OhanDU/J7G7L3zqL7GcRXko37ji20/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="72" data-origin-width="2048" data-origin-height="228">
 
 
</figure>
<pre id="code_1685506772414" data-ke-language="bash" data-ke-type="codeblock"><code>kubectl get pod -nkube-system</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1118" data-origin-height="757">
 
 <img src="https://blog.kakaocdn.net/dn/EUDPL/btsh3kVejpH/irufAXjKH3ju6madY6u2pk/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" data-origin-width="1118" data-origin-height="757">
 
 
</figure>

** Metrics 서버 이슈가 있는 경우 Crashloop back과 더불어 Restart되는 경우
<ul data-ke-list-type="disc">
<li>kubelet이 10250포트를 이미 점유하고 있어 호스트 네트워크 상에 묶여 이미 바인딩된 포트라면서 시작 불가</li>
</ul>
<pre id="code_1685506799690" data-ke-language="bash" data-ke-type="codeblock"><code>kgd -nkube-system</code></pre>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1211" data-origin-height="271">
 
 <img src="https://blog.kakaocdn.net/dn/c4sLvI/btsh23MR7gA/UAI7lQsFU8uCifeYwDG1OK/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="144" data-origin-width="1211" data-origin-height="271">
 
 
</figure>

<pre id="code_1685506804384" data-ke-language="bash" data-ke-type="codeblock"><code>ked -nkube-system metrics-server</code></pre>
metrics 서버의 포트를 10250에서 4443 변경합니다.
<ul data-ke-list-type="disc">
<li>container.args에서 --secure-port=4443</li>
<li>containers.ports에서</li>
</ul>
<figure data-ke-mobilestyle="widthOrigin" data-origin-width="1702" data-origin-height="1127">
 
 <img src="https://blog.kakaocdn.net/dn/bc3QYD/btsh2tLTBy8/hLUahSWSKpAwksCKZ2pG20/img.png" onerror="this.onerror=null; this.src='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png'; this.srcset='//t1.daumcdn.net/tistory_admin/static/images/no-image-v1.png';" width="643" height="425" data-origin-width="1702" data-origin-height="1127">
 
 
</figure>

<h2>Kubespray Kubernetes 업데이트</h2>
<pre id="code_1685506812330" data-ke-language="bash" data-ke-type="codeblock"><code>ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] update-cluster.yml</code></pre>
<h2>Kubespray Kubernetes 삭제</h2>
kubespray를 통해 Kubernetes Cluster 삭제하는 법
<pre id="code_1685506817394" data-ke-language="bash" data-ke-type="codeblock"><code>ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] reset.yml</code></pre>

 
 
 </div></div>

Jeongchul

Kubespray를 활용한 Kubernetes 클러스터 구축
설치하는 환경
Master VM, Worker VM 3
OS: Ubuntu 22.04
CPU: 4 core
Mem: 4096MB
N/W: Bridge
Kubespray로 Kuberenetes 구축 가이드
https://kubernetes.io/ko/docs/setup/production-environment/tools/kubespray/
 
Master 노드에서 pip3 설치를 진행합니다.
sudo apt install python3-pip
 
ssh key 복사
ssh key 복사를 진행합니다. ansible을 기본으로 사용하고 있어 master에서 ssh-keygen 이후에 worker vm 3대로 copy를 진행하면 됩니다.
ssh-keygen
ssh-copy-id 172.16.177.128 # master
ssh-copy-id 172.16.177.129 # worker1
ssh-copy-id 172.16.177.130 # worker2
ssh-copy-id 172.16.177.131 # worker3
Kubespray 설치
kubespray 설치를 진행합니다.
https://github.com/kubernetes-sigs/kubespray
mkdir environment && cd environment
git clone https://github.com/kubernetes-sigs/kubespray
git checkout release-2.22
sudo pip3 install -r requirements.txt
cp -r inventory/sample inventory/cluster
declare -a IPS=(172.16.177.129 172.16.177.130 172.16.177.131) # 설치할 Worker node들의 IP를 작성합니다.
CONFIG_FILE=inventory/cluster/hosts.yaml python3 contrib/inventory_builder/inventory.py ${IPS[@]}
생성된 config file을 확인해봅니다.
vi inventory/cluster/hosts.yaml
 
addon을 수정하여 필요한 것들을 true로 변경합니다.
vi inventory/cluster/group_vars/k8s_cluster/addons.yml
---
# Kubernetes dashboard
# RBAC required. see docs/getting-started.md for access details.
dashboard_enabled: true
# Helm deployment
helm_enabled: true
# Metrics Server deployment
metrics_server_enabled: true
metrics_server_container_port: 10250
metrics_server_kubelet_insecure_tls: true
metrics_server_metric_resolution: 15s
metrics_server_kubelet_preferred_address_types: "InternalIP,ExternalIP,Hostname"
metrics_server_host_network: true
metrics_server_replicas: 1
# Local volume provisioner deployment
local_volume_provisioner_enabled: true
# Nginx ingress controller deployment
ingress_nginx_enabled: true
# Cert manager deployment
cert_manager_enabled: true
Kubernetes 클러스터 설치 진행
ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] cluster.yml
설치에 수십 분 정도 소요됩니다.
 
kube 설정 파일을 가져옵니다.
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl 명령어 자동 완성
echo "source <(kubectl completion zsh)" >> ~/.zshrc
Alias 설정
vi ~/.zshrc
alias wk='watch kubectl get'
alias wka='watch kubectl get all'
alias wkgp='watch kubectl get pod'
alias k='kubectl'
alias kg='kubectl get'
alias ke='kubectl edit'
alias kdl='kubectl delete'
alias kd='kubectl describe'
alias kde='kubectl delete'
alias kes='kubectl edit service'
alias ked='kubectl edit deployment'
alias kef='kubectl edit statefulset'
alias kgns='kubectl get namespace'
alias kga='kubectl get all'
alias kgp='kubectl get pod'
alias kgd='kubectl get deployment'
alias kgf='kubectl get statefulset'
alias kgs='kubectl get service'
alias kgns='kubectl get ns'
alias kdp='kubectl describe pod'
alias kds='kubectl describe svc'
alias kdd='kubectl describe deployment'
alias kdf='kubectl describe statefulset'
alias kdnd='kubectl describe node'
alias klf='kubectl logs --follow'
alias kgnd='kubectl get node'
alias ksd='kubectl scale deployment'
alias ksud='kubectl scale deployment --replicas=1'
alias ksdd='kubectl scale deployment --replicas=0'
alias ksf='kubectl scale statefulset'
alias ksuf='kubectl scale statefulset -�~@~Treplicas=1'
alias ksdf='kubectl scale statefulset -�~@~Treplicas=0'
alias krd='kubectl rollout restart deployment'
alias krf='kubectl rollout restart statefulset'
source ~/.zshrc
 
Kubernetes 클러스터의 노드 정보를 확인합니다.
kubectl get node -o wide
kubectl get pod -nkube-system
 
** Metrics 서버 이슈가 있는 경우 Crashloop back과 더불어 Restart되는 경우
kubelet이 10250포트를 이미 점유하고 있어 호스트 네트워크 상에 묶여 이미 바인딩된 포트라면서 시작 불가
kgd -nkube-system
 
ked -nkube-system metrics-server
metrics 서버의 포트를 10250에서 4443 변경합니다.
container.args에서 --secure-port=4443
containers.ports에서
 
Kubespray Kubernetes 업데이트
ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] update-cluster.yml
Kubespray Kubernetes 삭제
kubespray를 통해 Kubernetes Cluster 삭제하는 법
ansible-playbook -i inventory/cluster/hosts.yaml --become --become-user=root --extra-vars ansible_sudo_pass=[SUDO_PASSWORD] reset.yml

Kubespray를 활용한 Kubernetes 클러스터 구축

댓글