서킷 브레이킹

Overview

Circuit break를 해결하는 방식은 기존에도 있었으며, 그 중 대표적으로 hystirx라는 라이브러리를 통해서 해결할 수 있었다. (넷플릭스가 개발하였으나 현재는 더 이상 업데이트가 없으며, 기존 기능에 대한 운영만 지원)

그러나 hystrix는 개별 마이크로서비스의 내부 코드에 이를(circuit break 함수) 반영해야만 하는 번거로움이 있으며, JVM기반의 어플리케이션만 지원하므로 go/python 등으로 개발된 마이크로서비스에는 적용할 수 없는 문제가 있다.

Istio는 마이크로서비스 외부의 proxy(envoy)를 이용하여 모든 네트워크를 제어하하는데, curcuit breker도 적용 가능하다. 즉, 마이크로서비스의 코드 변경없이 어떤 마이크로서비스에도 적용할 수 있는 장점이 있다

Demo Applications 배포

 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: position-simulator
 spec:
   selector:
     matchLabels:
       app: position-simulator
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: position-simulator
     spec:
       containers:
       - name: position-simulator
         image: richardchesterwood/istio-fleetman-position-simulator:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         command: ["java","-Xmx50m","-jar","webapp.jar"]
         imagePullPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: position-tracker
 spec:
   selector:
     matchLabels:
       app: position-tracker
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: position-tracker
     spec:
       containers:
       - name: position-tracker
         image: richardchesterwood/istio-fleetman-position-tracker:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         command: ["java","-Xmx50m","-jar","webapp.jar"]
         imagePullPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: api-gateway
 spec:
   selector:
     matchLabels:
       app: api-gateway
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: api-gateway
     spec:
       containers:
       - name: api-gateway
         image: richardchesterwood/istio-fleetman-api-gateway:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         command: ["java","-Xmx50m","-jar","webapp.jar"]
         imagePullPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: webapp
 spec:
   selector:
     matchLabels:
       app: webapp
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: webapp
         version: original
     spec:
       containers:
       - name: webapp
         image: richardchesterwood/istio-fleetman-webapp-angular:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         imagePullPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: vehicle-telemetry
 spec:
   selector:
     matchLabels:
       app: vehicle-telemetry
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: vehicle-telemetry
     spec:
       containers:
       - name: vehicle-telemtry
         image: richardchesterwood/istio-fleetman-vehicle-telemetry:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         imagePullPolicy: Always
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: staff-service
 spec:
   selector:
     matchLabels:
       app: staff-service
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: staff-service
         version: safe
     spec:
       containers:
       - name: staff-service
         image: richardchesterwood/istio-fleetman-staff-service:6
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         imagePullPolicy: Always
         ports:
         - containerPort: 8080
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   name: staff-service-risky-version
 spec:
   selector:
     matchLabels:
       app: staff-service
   replicas: 1
   template: # template for the pods
     metadata:
       labels:
         app: staff-service
         version: risky
     spec:
       containers:
       - name: staff-service
         image: richardchesterwood/istio-fleetman-staff-service:6-bad    # 해당 소스가 장애가 가지고 있는 소스이고 Risky로 배포 될 예정이다.
         env:
         - name: SPRING_PROFILES_ACTIVE
           value: production-microservice
         imagePullPolicy: Always
         ports:
         - containerPort: 8080
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: fleetman-webapp
 spec:
   # This defines which pods are going to be represented by this Service
   # The service becomes a network endpoint for either other services
   # or maybe external users to connect to (eg browser)
   selector:
     app: webapp
   ports:
     - name: http
       port: 80
   type: ClusterIP
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: fleetman-position-tracker
 spec:
   # This defines which pods are going to be represented by this Service
   # The service becomes a network endpoint for either other services
   # or maybe external users to connect to (eg browser)
   selector:
     app: position-tracker
   ports:
     - name: http
       port: 8080
   type: ClusterIP
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: fleetman-api-gateway
 spec:
   selector:
     app: api-gateway
   ports:
     - name: http
       port: 8080
   type: ClusterIP
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: fleetman-vehicle-telemetry
 spec:
   selector:
     app: vehicle-telemetry
   ports:
     - name: http
       port: 8080
   type: ClusterIP
 ---
 apiVersion: v1
 kind: Service
 metadata:
   name: fleetman-staff-service
 spec:
   selector:
     app: staff-service
   ports:
     - name: http
       port: 8080
   type: ClusterIP

Gw, Vs 구성

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: ingress-gateway-configuration
spec:
  selector:
    istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "kiali-mng-dev.saraminhr.co.kr"   # Domain name of the external website
---
# All traffic routed to the fleetman-webapp service
# No DestinationRule needed as we aren't doing any subsets, load balancing or outlier detection.
kind: VirtualService
apiVersion: networking.istio.io/v1alpha3
metadata:
  name: fleetman-webapp
  namespace: default
spec:
  hosts:      # which incoming host are we applying the proxy rules to???
    - "kiali-mng-dev.saraminhr.co.kr"
  gateways:
    - ingress-gateway-configuration
  http:
    - route:
      - destination:
          host: fleetman-webapp

확인

문제가 있는 Risky와 같이 배포를 했더니 브라우저에서 확인 해보면 한번씩 500에러가 발생한다.

curl로 확인

root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001459
connect:       0.002182
appconnect:    0.000000
pretransfer:   0.002226
redirect:      0.000000
starttransfer: 0.019133
--------------------------------------
total:         0.019139
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001552
connect:       0.002251
appconnect:    0.000000
pretransfer:   0.002260
redirect:      0.000000
starttransfer: 0.019725
--------------------------------------
total:         0.019842
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}namelookup:    0.001496
connect:       0.002103
appconnect:    0.000000
pretransfer:   0.002477
redirect:      0.000000
starttransfer: 0.022399
--------------------------------------
total:         0.022466
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}namelookup:    0.001412
connect:       0.002050
appconnect:    0.000000
pretransfer:   0.002138
redirect:      0.000000
starttransfer: 1.285805
--------------------------------------
total:         1.285837
[SARAMIN] root@sri-mng-kube-dev1:/usr/local/src/istio
04:49 오후
root # curl -w @curl.txt http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck
{"timestamp":"2023-11-07T07:49:21.555+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}namelookup:    0.001339
connect:       0.001931
appconnect:    0.000000
pretransfer:   0.001974
redirect:      0.000000
starttransfer: 5.003001
--------------------------------------
total:         5.003088

한번씩 실패나기도 하면서 지연도 있는것 같다.
예거에서도 보면 다른 서비스에서도 4초 이상 지연이 발생했다.
kiali에서 확인 해보면 Risky 하나로 전체적으로 지연 발생하는 것으로 보인다.

Circuit Breaker 설정

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: circuit-breaker-for-the-entire-default-namespace
spec:
  host: "fleetman-staff-service.default.svc.cluster.local"
  trafficPolicy:
    outlierDetection: # Circuit Breakers가 작동하는 기준 설정
      consecutive5xxErrors: 2
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 100

[consecutiveErrors]
연속적인 에러가 몇번까지 발생해야 circuit breaker를 동작시킬 것인지 결정
여기서는 연속 2번 오류가 발생하면 circuit breaker 동작 (테스트 환경으로 횟수를 낮췄다.)

[interval]
interval에서 지정한 시간 내에 consecutiveError 횟수 만큼 에러가 발생하는 경우 circuit breaker 동작
즉, 10초 내에 2번의 연속적인 오류가 발생하면 circuit breaker 동작

[baseEjectionTime]
차단한 호스트를 얼마 동안 로드밸런서 pool에서 제외할 것인가?
즉, 얼마나 오래 circuit breaker를 해당 호스트에게 적용할지 시간을 결정

[maxEjectionPercent]
네트워크를 차단할 최대 host의 비율. 즉, 최대 몇 %까지 차단할 것인지 설정
현재 구성은 2개의 pod가 있으므로, 100%인 경우 2개 모두 차단이 가능하다
10%인 경우 차단이 불가능해 보이는데(1개가 50%이므로),
envoy에서는 circuit breaker가 발동되었으나,
10%에 해당하지 않아서 차단할 호스트가 없으면
강제적으로 해당 호스트를 차단하도록 설정한다

확인

서킷 브레이커가 동작 중이면 번개 표시로 나타남

curl로 동작 확인

while true; do curl http://kiali-mng-dev.saraminhr.co.kr/api/vehicles/driver/City%20Truck; echo; sleep 0.5; done
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/placeholder.png"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"timestamp":"2023-11-07T08:39:50.949+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"timestamp":"2023-11-07T08:39:53.483+0000","status":500,"error":"Internal Server Error","message":"status 502 reading RemoteStaffMicroserviceCalls#getDriverFor(String)","path":"//vehicles/driver/City%20Truck"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
{"name":"Pam Parry","photo":"https://rac-istio-course-images.s3.amazonaws.com/1.jpg"}
^C

처음에 2번 에러가 나면서 서킷 브레이커가 동작하게 되면서 더이상 에러가 발생 안하는 모습을 볼수 있었다.

웹브라우저에서도 지연없이 사진도 잘 불러와지는 것을 확인 할 수 있었다.
전체 서비스에 서킷브레이커를 동작 시키고 싶다면 전역 설정이 있다.

Mutual TLS(mTLS) with Istio (1)	2024.01.02
Istio Traffic Management 트래픽 통제하기 (0)	2023.11.10

티스토리툴바

기록's