Overview

Istio를 통해 Traffic을 제어 할 수 있다.

다음은 Istio를 통해서도 Canary를 구현 할 수 있다. Canary 또한 Traffic을 통제하여 신버전으로 Traffic을 적게 가져가는 식으로 하면 된다.

Weighted 기반 라우팅

1. Demo Application 구성

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-simulator
    spec:
      selector:
        matchLabels:
          app: position-simulator
      replicas: 1
      template:
        metadata:
          labels:
            app: position-simulator
        spec:
          containers:
          - name: position-simulator
            image: richardchesterwood/istio-fleetman-position-simulator:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-tracker
    spec:
      selector:
        matchLabels:
          app: position-tracker
      replicas: 1
      template:
        metadata:
          labels:
            app: position-tracker
        spec:
          containers:
          - name: position-tracker
            image: richardchesterwood/istio-fleetman-position-tracker:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: api-gateway
    spec:
      selector:
        matchLabels:
          app: api-gateway
      replicas: 1
      template:
        metadata:
          labels:
            app: api-gateway
        spec:
          containers:
          - name: api-gateway
            image: richardchesterwood/istio-fleetman-api-gateway:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: webapp
    spec:
      selector:
        matchLabels:
          app: webapp
      replicas: 1
      template:
        metadata:
          labels:
            app: webapp
        spec:
          containers:
          - name: webapp
            image: richardchesterwood/istio-fleetman-webapp-angular:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: vehicle-telemetry
    spec:
      selector:
        matchLabels:
          app: vehicle-telemetry
      replicas: 1
      template:
        metadata:
          labels:
            app: vehicle-telemetry
        spec:
          containers:
          - name: vehicle-telemtry
            image: richardchesterwood/istio-fleetman-vehicle-telemetry:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 1
      template:
        metadata:
          labels:
            app: staff-service
            version: safe      # 구버전에 대한 Application Label이다. Safe가 기존 운영하던 Stable한 버전이라고 친다.
        spec:
          containers:
          - image: richardchesterwood/istio-fleetman-staff-service:6-placeholder
            name: staff-service
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service-risky-version
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 1
      template:
        metadata:
          labels:
            app: staff-service
            version: risky       # 신버전에 대한 Application에 대한 Deploy이다. 신버전에 대해서 version: risky 라는 Label로 지정하였다.
        spec:
          containers:
          - name: staff-service
            image: richardchesterwood/istio-fleetman-staff-service:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-webapp
    spec:
      selector:
        app: webapp
      ports:
        - appProtocol: http
          name: http
          port: 80
          protocol: TCP
          targetPort: 80
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-position-tracker
    spec:
      selector:
        app: position-tracker
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-api-gateway
    spec:
      selector:
        app: api-gateway
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-vehicle-telemetry
    spec:
      selector:
        app: vehicle-telemetry
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-staff-service
    spec:
      selector:
        app: staff-service
      ports:
        - name: http
          port: 8080
      type: ClusterIP
   

   ---

   2. Demo Application 웹 접속을 위한 GW, VS 구성

      apiVersion: networking.istio.io/v1beta1
      kind: Gateway
      metadata:
      name: fleetman-gw
      spec:
      selector:
       istio: ingressgateway
      servers:
      - hosts:
       - "kiali-mng-dev.test.co.kr"
       port:
         number: 80
         name: http
         protocol: HTTP

   ---

   apiVersion: networking.istio.io/v1beta1
   kind: VirtualService
   metadata:
   name: fleetman-vs
   spec:
   hosts:

   • "kiali-mng-dev.test.co.kr"
     gateways:

   • fleetman-gw
     http:

   • match:

     • port: 80
       route:
     • destination:
       host: fleetman-webapp
       port:

       number: 80

     ---

     Kiali 에서보면 구조는 아래와 같다.

     staff-service에 보면 Risky 버전과 Safe 버전 2가지가 공존하는 것을 볼 수 있다.

     3. Demo Web 접속
     • 해당 웹어플리케이션은 배송 추적 시스템이다.

2. 신버전과 구버전간의 트래픽 비율 조정

    kind: VirtualService
    apiVersion: networking.istio.io/v1alpha3
    metadata:
      name: canary-vs  # 이 부분은 그냥 이름 넣어주면 된다.
      namespace: default
    spec:
      hosts:
        - fleetman-staff-service.default.svc.cluster.local  # 라우팅 규칙을 적용할 서비스 DNS 이름 입니다. (내부 용 FQDN으로 처리)
      http:
        - route:
            - destination:
                host: fleetman-staff-service.default.svc.cluster.local # 타겟 DNS 이름
                subset: safe-group  # Destination Rule의 이름
              weight: 90   # 비율 지정
            - destination:
                host: fleetman-staff-service.default.svc.cluster.local # 타겟 DNS 이름
                subset: risky-group  # Destination Rule의 이름
              weight: 10
    ---
    kind: DestinationRule       # 각 Subset에 포함될 파드 정의
    apiVersion: networking.istio.io/v1alpha3
    metadata:
      name: canary-dr
      namespace: default
    spec:
      host: fleetman-staff-service # 서비스
      subsets:
        - labels:   # Selector
            version: safe # 파드를 Label과 함께 찾는다.
          name: safe-group
        - labels:
            version: risky
          name: risky-group

   ---

3. 확인

4. 실제 City Truck의 운전자 사진이 Risk 버전에서는 제대로 표시가 안되는 것을 확인

5. 브라우저에서는 Caching 때문에 제대로 확인이 안될 수 있는데 curl로 확인 해보자

    root # while true; do curl http://kiali-mng-dev.test.co.kr/api/vehicles/driver/City%20Truck; echo; sleep 0.5; done
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
    {"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}

   ---

   "1.jpg" 가 10% 확율 치고는 자주 나온거 같긴하지만 얼추 그래도 맞는 것 같다.

Session Affinity

Istio에서 Consistent Hash 라는 것을 이용하여 Session을 유지 해주는 기능을 제공한다.

LB에서 Hash 알고리즘을 이용하여 Client로부터 받은 데이터를 Hashing 한 후 데이터를 파드에 전송하는데 Hash 값을 이용하여 Sticky Session을 유지한다

Session 관리로 여러가지 방식을 지원한다. Header, Cookie, SourceIP 등

여기서는 SourceIP를 통해 테스트 진행 해 보았다.

위에 테스트한 데모 어플리케이션이 Version에 따라 safe와 risky 버전을 나눠서 90:10 비율로 처리 했는데, Consistent Hash를 사용하여 처음 접근 했던 세션이 그대로 유지 되는지 테스트

kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: session-vs
  namespace: default
spec:
  hosts:
    - fleetman-staff-service.default.svc.cluster.local
  http:
    - route:
        - destination:
            host: fleetman-staff-service.default.svc.cluster.local
            subset: all-staff-service-pods
          # weight: weight 옵션을 같이 사용 할 수 없다.
---
kind: DestinationRule
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: session-dr
  namespace: default
spec:
  host: fleetman-staff-service
  trafficPolicy:
    loadBalancer:
      consistentHash:
        useSourceIp: true   # useSourceIp: true로 설정하면 접속한 SourceIP으로 세션을 유지 해준다.
  subsets:
    - labels:
        app: staff-service
      name: all-staff-service-pods

처음 접속 했을 때 Stable 버전으로 접속되서 다른 Truck들도 사진이 잘 나온다. 반면에 Risky 버전으로 접속이 됐으면 계속 사진이 없는 Truck만 조회 된다.

Prefix기반 라우팅

1. Demo Application 구성

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-simulator
    spec:
      selector:
        matchLabels:
          app: position-simulator
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: position-simulator
        spec:
          containers:
          - name: position-simulator
            image: richardchesterwood/istio-fleetman-position-simulator:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-tracker
    spec:
      selector:
        matchLabels:
          app: position-tracker
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: position-tracker
        spec:
          containers:
          - name: position-tracker
            image: richardchesterwood/istio-fleetman-position-tracker:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: api-gateway
    spec:
      selector:
        matchLabels:
          app: api-gateway
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: api-gateway
        spec:
          containers:
          - name: api-gateway
            image: richardchesterwood/istio-fleetman-api-gateway:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: webapp
    spec:
      selector:
        matchLabels:
          app: webapp
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: webapp
            version: original
        spec:
          containers:
          - name: webapp
            image: richardchesterwood/istio-fleetman-webapp-angular:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: webapp-experimental
    spec:
      selector:
        matchLabels:
          app: webapp
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: webapp
            version: experimental     # labels로 version: experimental 이름으로 canary와 동일한 신규버전을 만들었다.
        spec:
          containers:
          - name: webapp
            image: richardchesterwood/istio-fleetman-webapp-angular:6-experimental
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: vehicle-telemetry
    spec:
      selector:
        matchLabels:
          app: vehicle-telemetry
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: vehicle-telemetry
        spec:
          containers:
          - name: vehicle-telemtry
            image: richardchesterwood/istio-fleetman-vehicle-telemetry:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 0
      template: # template for the pods
        metadata:
          labels:
            app: staff-service
            version: safe               # version: safe로 기존 stable 버전이다.
        spec:
          containers:
          - name: staff-service
            image: richardchesterwood/istio-fleetman-staff-service:6-placeholder
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service-risky-version
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: staff-service
            version: risky              # version: risky가 신규 버전의 Canry 이다.
        spec:
          containers:
          - name: staff-service
            image: richardchesterwood/istio-fleetman-staff-service:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-webapp
    spec:
      # This defines which pods are going to be represented by this Service
      # The service becomes a network endpoint for either other services
      # or maybe external users to connect to (eg browser)
      selector:
        app: webapp
      ports:
        - name: http
          port: 80
          nodePort: 30080
      type: NodePort
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-position-tracker
    spec:
      # This defines which pods are going to be represented by this Service
      # The service becomes a network endpoint for either other services
      # or maybe external users to connect to (eg browser)
      selector:
        app: position-tracker
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-api-gateway
    spec:
      selector:
        app: api-gateway
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-vehicle-telemetry
    spec:
      selector:
        app: vehicle-telemetry
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-staff-service
    spec:
      selector:
        app: staff-service
      ports:
        - name: http
          port: 8080
      type: ClusterIP

   ---

2. Virtaul Service, Gateway, Destination Rule

    apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      name: ingress-gateway-configuration
    spec:
      selector:
        istio: ingressgateway
      servers:
      - port:
          number: 80
          name: http
          protocol: HTTP
        hosts:
        - "*"
    ---
    kind: VirtualService
    apiVersion: networking.istio.io/v1alpha3
    metadata:
      name: fleetman-webapp
      namespace: default
    spec:
      hosts:
        - "*"
      gateways:
        - ingress-gateway-configuration
      http:
        - match:  # Prefix에 대한 설정
          - uri:  # IF /experimental 이거나
              prefix: "/experimental"
          - uri:  # OR /canary 이면
              prefix: "/canary"
          route: # THEN destination rule의 experimental로 라우팅 해라.
          - destination:
              host: fleetman-webapp
              subset: experimental
        - match:  # / Prefix면 DR의 original로 기존 버전으로 라우팅
          - uri :
              prefix: "/"
          route:
          - destination:
              host: fleetman-webapp
              subset: original
    ---
    kind: DestinationRule
    apiVersion: networking.istio.io/v1alpha3
    metadata:
      name: fleetman-webapp
      namespace: default
    spec:
      host: fleetman-webapp
      subsets:
        - labels:
            version: original # labels.version: original로 라우팅
          name: original     # Subset Name이 Original이면
        - labels:
            version: experimental # labels.version: experimental로 라우팅
          name: experimental # Subset Name이 experimental이면

   ---

3. 테스트

    root # while true; do curl -s http://kiali-mng-dev.test.co.kr/experimental | grep title; sleep 0.5; done
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>

   ---

   /experimental로 하면 신규버전으로 "Fleet Management Istio Premium Enterprise Edition" 만 표시되어야 하는데 라운드로빈으로 왔다갔다 호출 되는거 같다.

   이 부분은 Istio-Ingressgateway로 트래픽을 제대로 타지 못했거나 K8s 서비스 오브젝트를 통해 접근 했을 경우 또는 다른 이슈가 있는 듯하다.

   Haproxy와 Ingressgateway service의 설정은 딱히 문제가 없는 듯 하다.

   확인 해보니 기존에 테스트로 생성해둔 GW, VS가 동일한 호스트와 포트로 사용중인게 있어서 문제였다 (관련해서 kiali를 통해 발견했다. 스크린샷은 찍지못함)

   관련해서 제거 한 후 다시 테스트

    root # while true; do curl -s http://kiali-mng-dev.test.co.kr/experimental | grep title; sleep 0.5; done
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>
      <title>Fleet Management Istio Premium Enterprise Edition</title>

   ---

    root # while true; do curl -s http://kiali-mng-dev.test.co.kr/ | grep title; sleep 0.5; done
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
      <title>Fleet Management</title>
   

   ---

   신규 버전과 구 버전의 Prefix로 라우팅이 잘 되는듯 하다.

SubDomain으로 라우팅

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway-configuration
spec:
  selector:
    istio: ingressgateway #
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "kiali-mng-dev.test.co.kr"
    - "prometheus-mng-dev.test.co.kr"
    #- "*.kiali-mng-dev.test.co.kr"  이런식으로 와일드카드로도 설정 가능
    #- "*.test.co.kr"
---
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  hosts:
    - "kiali-mng-dev.test.co.kr"
  gateways:
    - ingress-gateway-configuration
  http:
    - route:
      - destination:
          host: fleetman-webapp
          subset: original
---
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: fleetman-webapp-experiment
  namespace: default
spec:
  hosts:
    - "prometheus-mng-dev.test.co.kr"
  gateways:
    - ingress-gateway-configuration
  http:
      - route:
        - destination:
            host: fleetman-webapp
            subset: experimental
---
kind: DestinationRule
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  host: fleetman-webapp
  subsets:
    - labels:
        version: original
      name: original
    - labels:
        version: experimental
      name: experimental

JWT 기반 라우팅은 아직 알파단계 조만간 나올 것으로 보인다.

Overview

Circuit break를 해결하는 방식은 기존에도 있었으며, 그 중 대표적으로 hystirx라는 라이브러리를 통해서 해결할 수 있었다. (넷플릭스가 개발하였으나 현재는 더 이상 업데이트가 없으며, 기존 기능에 대한 운영만 지원)

그러나 hystrix는 개별 마이크로서비스의 내부 코드에 이를(circuit break 함수) 반영해야만 하는 번거로움이 있으며, JVM기반의 어플리케이션만 지원하므로 go/python 등으로 개발된 마이크로서비스에는 적용할 수 없는 문제가 있다.

Istio는 마이크로서비스 외부의 proxy(envoy)를 이용하여 모든 네트워크를 제어하하는데, curcuit breker도 적용 가능하다. 즉, 마이크로서비스의 코드 변경없이 어떤 마이크로서비스에도 적용할 수 있는 장점이 있다

1. Demo Applications 배포

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-simulator
    spec:
      selector:
        matchLabels:
          app: position-simulator
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: position-simulator
        spec:
          containers:
          - name: position-simulator
            image: richardchesterwood/istio-fleetman-position-simulator:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: position-tracker
    spec:
      selector:
        matchLabels:
          app: position-tracker
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: position-tracker
        spec:
          containers:
          - name: position-tracker
            image: richardchesterwood/istio-fleetman-position-tracker:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: api-gateway
    spec:
      selector:
        matchLabels:
          app: api-gateway
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: api-gateway
        spec:
          containers:
          - name: api-gateway
            image: richardchesterwood/istio-fleetman-api-gateway:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            command: ["java","-Xmx50m","-jar","webapp.jar"]
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: webapp
    spec:
      selector:
        matchLabels:
          app: webapp
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: webapp
            version: original
        spec:
          containers:
          - name: webapp
            image: richardchesterwood/istio-fleetman-webapp-angular:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: vehicle-telemetry
    spec:
      selector:
        matchLabels:
          app: vehicle-telemetry
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: vehicle-telemetry
        spec:
          containers:
          - name: vehicle-telemtry
            image: richardchesterwood/istio-fleetman-vehicle-telemetry:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: staff-service
            version: safe
        spec:
          containers:
          - name: staff-service
            image: richardchesterwood/istio-fleetman-staff-service:6
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: staff-service-risky-version
    spec:
      selector:
        matchLabels:
          app: staff-service
      replicas: 1
      template: # template for the pods
        metadata:
          labels:
            app: staff-service
            version: risky
        spec:
          containers:
          - name: staff-service
            image: richardchesterwood/istio-fleetman-staff-service:6-bad    # 해당 소스가 장애가 가지고 있는 소스이고 Risky로 배포 될 예정이다.
            env:
            - name: SPRING_PROFILES_ACTIVE
              value: production-microservice
            imagePullPolicy: Always
            ports:
            - containerPort: 8080
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-webapp
    spec:
      # This defines which pods are going to be represented by this Service
      # The service becomes a network endpoint for either other services
      # or maybe external users to connect to (eg browser)
      selector:
        app: webapp
      ports:
        - name: http
          port: 80
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-position-tracker
    spec:
      # This defines which pods are going to be represented by this Service
      # The service becomes a network endpoint for either other services
      # or maybe external users to connect to (eg browser)
      selector:
        app: position-tracker
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-api-gateway
    spec:
      selector:
        app: api-gateway
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-vehicle-telemetry
    spec:
      selector:
        app: vehicle-telemetry
      ports:
        - name: http
          port: 8080
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: fleetman-staff-service
    spec:
      selector:
        app: staff-service
      ports:
        - name: http
          port: 8080
      type: ClusterIP

   ---

2. Gw, Vs 구성

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway-configuration
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "kiali-mng-dev.saraminhr.co.kr"   # Domain name of the external website
---
# All traffic routed to the fleetman-webapp service
# No DestinationRule needed as we aren't doing any subsets, load balancing or outlier detection.
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  hosts:      # which incoming host are we applying the proxy rules to???
    - "kiali-mng-dev.saraminhr.co.kr"
  gateways:
    - ingress-gateway-configuration
  http:
    - route:
      - destination:
          host: fleetman-webapp

3. 확인

문제가 있는 Risky와 같이 배포를 했더니 브라우저에서 확인 해보면 한번씩 500에러가 발생한다.

4. curl로 확인

root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001459
connect:       0.002182
appconnect:    0.000000
pretransfer:   0.002226
redirect:      0.000000
starttransfer: 0.019133
--------------------------------------
total:         0.019139
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001552
connect:       0.002251
appconnect:    0.000000
pretransfer:   0.002260
redirect:      0.000000
starttransfer: 0.019725
--------------------------------------
total:         0.019842
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001496
connect:       0.002103
appconnect:    0.000000
pretransfer:   0.002477
redirect:      0.000000
starttransfer: 0.022399
--------------------------------------
total:         0.022466
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}namelookup:    0.001412
connect:       0.002050
appconnect:    0.000000
pretransfer:   0.002138
redirect:      0.000000
starttransfer: 1.285805
--------------------------------------
total:         1.285837
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"timestamp":"2023-11-07T07:49:21.555+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}namelookup:    0.001339
connect:       0.001931
appconnect:    0.000000
pretransfer:   0.001974
redirect:      0.000000
starttransfer: 5.003001
--------------------------------------
total:         5.003088

• 한번씩 실패나기도 하면서 지연도 있는것 같다.
• 예거에서도 보면 다른 서비스에서도 4초 이상 지연이 발생했다.
• kiali에서 확인 해보면 Risky 하나로 전체적으로 지연 발생하는 것으로 보인다.

5. Circuit Breaker 설정

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: circuit-breaker-for-the-entire-default-namespace
spec:
  host: "fleetman-staff-service.default.svc.cluster.local"
  trafficPolicy:
    outlierDetection: # Circuit Breakers가 작동하는 기준 설정
      consecutive5xxErrors: 2
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 100

[consecutiveErrors]
연속적인 에러가 몇번까지 발생해야 circuit breaker를 동작시킬 것인지 결정
여기서는 연속 2번 오류가 발생하면 circuit breaker 동작 (테스트 환경으로 횟수를 낮췄다.)

[interval]
interval에서 지정한 시간 내에 consecutiveError 횟수 만큼 에러가 발생하는 경우 circuit breaker 동작
즉, 10초 내에 2번의 연속적인 오류가 발생하면 circuit breaker 동작

[baseEjectionTime]
차단한 호스트를 얼마 동안 로드밸런서 pool에서 제외할 것인가?
즉, 얼마나 오래 circuit breaker를 해당 호스트에게 적용할지 시간을 결정

[maxEjectionPercent]
네트워크를 차단할 최대 host의 비율. 즉, 최대 몇 %까지 차단할 것인지 설정
현재 구성은 2개의 pod가 있으므로, 100%인 경우 2개 모두 차단이 가능하다
10%인 경우 차단이 불가능해 보이는데(1개가 50%이므로),
envoy에서는 circuit breaker가 발동되었으나,
10%에 해당하지 않아서 차단할 호스트가 없으면
강제적으로 해당 호스트를 차단하도록 설정한다

6. 확인

서킷 브레이커가 동작 중이면 번개 표시로 나타남

• curl로 동작 확인

while true; do curl http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck; echo; sleep 0.5; done
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"timestamp":"2023-11-07T08:39:50.949+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"timestamp":"2023-11-07T08:39:53.483+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
^C

처음에 2번 에러가 나면서 서킷 브레이커가 동작하게 되면서 더이상 에러가 발생 안하는 모습을 볼수 있었다.

• 웹브라우저에서도 지연없이 사진도 잘 불러와지는 것을 확인 할 수 있었다.
• 전체 서비스에 서킷브레이커를 동작 시키고 싶다면 전역 설정이 있다.