기록's

개요

Kubernetes API 서버를 KUBECONIFG를 통한 접근을 하기 위해 단일 서버로 설정해서 접근 할 수도 있지만,

API 서버를 LB로 묶어서 접근 관리 할수있도록 하기 위한 설정이다. 이렇게 묶어서 설정 및 관리하면, 여러 관리 도구와의 결합 구성시 고가용성 유지할 수 있도록 해준다.

• HAproxy서버에서 Kubernetes Master Node에 대해 tcp LB 설정을 해준다.

    #----------------------------------------------
    # test kube-apiserver Settings
    #----------------------------------------------
    defaults kube-tcp
        log global
        mode tcp
        maxconn 65535
        timeout connect  4s
        timeout server  15s
        timeout client  15s
        #timeout tunnel    2m
  
    # ---------------- ea-kube-master-front frontend --------------------
    frontend ea-kube-master-front
        bind 192.168.0.101:6443 name kube-tcp
        description test Kube Apiserver Front
        option tcplog
        default_backend ea-kube-dev-master-backend
    # ---------------- ea-kube-master-backend --------------------
    backend ea-kube-dev-master-backend
    mode tcp
    balance roundrobin
    option tcp-check
    option tcplog
    server sri-ea-kube-dev1 192.168.0.74:6443 check
    server sri-ea-kube-dev2 192.168.0.75:6443 check
    server sri-ea-kube-dev3 192.168.0.76:6443 check

  ---

• bind :6443 포트, Backend 또한 6443 포트로 진행

• Kubernetes를 컨트롤 할 서버(중앙 관리하는 MNG 서버나, ArgoCD 서버 등등) 에서 KUBECONFIG 설정 진행

    apiVersion: v1
    clusters:
    - cluster:
        certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMvakNDQWVhZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJek1EZ3hNVEEzTURBek5sb1hEVE16TURnd09EQTNNREF6Tmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUV6CnJkTE01Qk10RFI1UXhjUXZ3M0RvVjlHdHRYY3duM2xla0JiaDY1QmdpVUdoa05RajJRa0FNdS9BeWZGZTVPMWoKMUo2STZ3dkl1WUhralJTS040bitoU09ReVlRZ1dBSjFLOWRsZzZpZ3hFS0NhaUpqZm9XUjlpKytheDhFTUNHVQpUM2sxVy8xVmhNOVFWY2JMaWtoSHgyVndUaXlOMWJGRzVGVjlISWNVa0daV3laS2NuRDRrU0RhajU4bkRqNVhTCktqUnprMVlkbnhYSzRuNmQxVVpvS3pyYjFsYmtmdk5sR29PSjRTS2RqcmVvMlpKc1J4dE9YR1o0aHR3dm5ndG0KMFo3VzhoWHMrcmRwOEFXQnYvY2wzMEk0OFVnZzA1U2dKbXN2ZEE1cklkdjJZc01nU2FUTnBnekwwZTgvUEE0UgpxNnZ5SWlRTnk4V2ZpY3NpZytzQ0F3RUFBYU5aTUZjd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZLeXJ5dHFua2l5Uytqc09YK1AvSFRlemhBZGVNQlVHQTFVZEVRUU8KTUF5Q0NtdDFZbVZ5Ym1WMFpYTXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ0Y2VjZNczRJSGhieTFuNHp0cgpEcVJZODBjWlBRcjdsVWszUHNMaW5QSlRIT1dwTFdGVkMzYU1TYndpdWNGaUttenVOcUNLWHpwR3g1RzRwLzlhCnM3YVZNbUV3Vjl4NFp0RkNmMkVDUVhXeHkvWVppeFFrdDNpNTBrMU5wT0hmTU5NZ1BXMHhoamt1QS9DMzRXKzUKdUN0RnZUT3NyQ014ZGtuak13bElGWnZ4OHlXTjZLbUsxUWFMRGFtUVBRQk1IQVEvM2l0S2d4Y0ZNQldKQWcxUApmd2ZGZHpQWUVvOVZEbW1LQkI4RnlEYk1wNmd0MDRsVlNJbnFXUDdFcjR3MkZpNnhWYkJadHpJYVNkWFFHNkZRCkl2NTFNc2c4QVFmQ05DSkxkV1c3M3N2dTNGNklZMkVETmUxWVlvOXAvN2Z1ZDRxckNLbGZBdnBTcThFdkRjVzMKL1BjPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
        server: https://ea-kube-dev.test.co.kr:6443
      name: ea.cluster.local
    contexts:
    - context:
        cluster: ea.cluster.local
        user: default
      name: ea-kube
    current-context: ea-kube
    kind: Config
    preferences: {}
    users:
    - name: default
      user:
        token: eyJhbGciOiJSUzI1NiIsImtpZCI6IktxOERHZHdCMFFKSmkxTW9USzBGTndWalZJX2h2NDhVSTI5MEZNdDZPREkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImE4MjE0YWJmLTA0NDctNDhlMC05MjEzLTdkMjFhZDFhOTM0YSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.hOT7Q-lMb_2bV11YQ895DOde2TuWDRnjJOv1H2QYy65tfEdVjZGZmjWzOefVnihxjjhYYP5VKl4G7OfX6GQrDoR0ZW_bB1ABikePblJmCbaf4kL2GlQtrUHQeDwg9BFCXwzUzIPFBzn-6gst_cPJF3XMPMVqbjixrCUe5fJYmouDiKckLjxINNCvUB5wlAon1F20QFeiCU8X5rteJl-GYZApRbPYpuacl3Tea9tmdESBbG17kBjquTSTvwbB6GeTwgPrs4ek8Q_Hs3bIUlGEtCHXeyja8so-kKb7bWr_LThFLxVhqoIuvpi2TACOvAmdLrM-RSknD6HN6ghGCPdVUg

  ---

• Kubernetes Context Swiching 진행 후 kubectl 명령어 수행

    kubectl get pod

  ---

• 하지만 아래와 같이 에러가 발생한다.

    Unable to connect to the server: x509: certificate is valid for ea-kube-dev.test.co.kr

  ---

  이 경우는 보통 인증서가 잘못되었거나 아니면 해당 인증서에 대해 유효한 서버가 아니기 때문에 에러가 발생한다.

  우리는 haproxy에서 tcp로 접근하기 때문에 별도 인증서를 세팅 하지 않았기 때문에 서버에 대해 추가해야 한다.

• 먼저 현재 인증서에 접근하는 서버가 리스트에 있는지 확인

    openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text
  

  ---

• output에서 Alternative Name을 살펴보면 접근 하고자 하는 ea-kube-dev.test.co.kr 서버가 없는 것을 확인 할 수 있다.

     X509v3 Subject Alternative Name:
                    DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.ea-dev.cluster.local, DNS:lb-apiserver.kubernetes.local, DNS:localhost, DNS:sri-ea-kube-dev1, DNS:sri-ea-kube-dev2, DNS:sri-ea-kube-dev3, IP Address:10.233.0.1, IP Address:192.168.0.74, IP Address:127.0.0.1, IP Address:192.168.0.75, IP Address:192.168.0.76

  ---

• 서버를 추가한 후 인증서를 재생성 해야 한다.

    # 인증서 백업
    # 별도 서비스를 내리지 않고 진행한다. 기존 파일이 있으면 반영 되지 않는다.
    mv /etc/kubernetes/pki/apiserver.key /home/hkpark/.
    mv /etc/kubernetes/pki/apiserver.crt /home/hkpark/.
  
    # Config Map 수정
    kubectl -n kube-system edit configmap kubeadm-config
  
    data:
      ClusterConfiguration: |
        apiServer:
          certSANs:
          - kubernetes
          - kubernetes.default
          - kubernetes.default.svc
          - kubernetes.default.svc.ea-dev.cluster.local
          - 10.233.0.1
          - localhost
          - 127.0.0.1
          - sri-ea-kube-dev1
          - sri-ea-kube-dev2
          - sri-ea-kube-dev3
          - lb-apiserver.kubernetes.local
          - 192.168.0.74
          - 192.168.0.75
          - 192.168.0.76
          - ea-kube-dev.test.co.kr
          - 192.168.0.101
  
    # certSANs 리스트에 서버를 추가한다. (도메인 이름 ea-kube-dev.test.co.kr, IP주소까지 해주면 좋을듯)

  ---

• 인증서 재생성

    # 추가 하고자 하는 서버만 파라미터에 추가해서 수행하면 안되고, 현재 등록된 서버 모두 함께 --apiserver-cert-extra-sans 파라미터에 추가하자
  
    kubeadm init phase certs apiserver --apiserver-cert-extra-sans=kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.ea.cluster.local,10.233.0.1,localhost,127.0.0.1,sri-ea-kube-master1,sri-ea-kube-master2,sri-ea-kube-master3,lb-apiserver.kubernetes.local,10.0.2.64,10.0.2.65,10.0.2.66,sri-ea-kube-master1.test.co.kr,sri-ea-kube-master2.test.co.kr,sri-ea-kube-master3.test.co.kr,ea-kube.test.co.kr,10.0.2.190
  
    # 해당 인증서 생성은 Master Node 전체에서 수행 (ex: kube-master1, kube-master2, kube-master3)

  ---

• apiserver pod 리스타트

    kubectl delete pod kube-apiserver-sri-ea-kube-dev1 kube-apiserver-sri-ea-kube-dev2 kube-apiserver-sri-ea-kube-dev3 -n kube-system

  ---

• 정상 확인

    root # kubectl get ns
    NAME                                                STATUS   AGE
    akhq                                                Active   44d
    apigateway-893-review-develop-3zknud                Active   42d
    apigateway-893-review-feature-bl-ltxs10             Active   42d
    apigateway-893-review-v1-x-sdiidw                   Active   42d
    apigateway-893-staging                              Active   41d
    authorization-server-886-review-authorizat-fl7x7y   Active   55d
    authorization-server-886-review-develop-3zknud      Active   42d
    authorization-server-886-review-v1-x-sdiidw         Active   44d
    authorization-server-886-staging                    Active   41d
    bootadmin-server-912-staging                        Active   28d
    cattle-fleet-clusters-system                        Active   56d
    cattle-fleet-local-system                           Active   56d
    cattle-fleet-system                                 Active   56d
    cattle-global-data                                  Active   56d
    cattle-global-nt                                    Active   56d

  ---

• Cert Renew 작업에서 다시 원복 되지 않는 것으로 확인하였다.

    /usr/local/bin/kubeadm certs renew all

  ---