Jacob Baek's home

InternalTrafficPolicy

Jacob_baek — Tue, 25 Nov 2025 17:54:26 +0900

https://kubernetes.io/docs/concepts/services-networking/service-traffic-policy/

연결하려는 pod가 동일 node에 없는 경우 연결자체가 안된다.

## internaltrafficpolicy = local and pods are not in same node.
root@aks-nodepool1-23236778-vmss000002:/# iptables-save | grep -i nginx
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc has no local endpoints" -j DROP
## internaltrafficpolicy = cluster
root@aks-nodepool1-23236778-vmss000002:/# iptables-save | grep -i nginx
-A KUBE-SEP-VKFWAD6C5GW7XJNY -s 10.240.0.50/32 -m comment --comment "default/nginx-sample-svc" -j KUBE-MARK-MASQ
-A KUBE-SEP-VKFWAD6C5GW7XJNY -p tcp -m comment --comment "default/nginx-sample-svc" -m tcp -j DNAT --to-destination 10.240.0.50:8080
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-SVC-JT67RD6F3OETQGP2
-A KUBE-SVC-JT67RD6F3OETQGP2 -d 10.0.130.191/32 ! -i azv+ -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-MARK-MASQ
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.50:8080" -j KUBE-SEP-VKFWAD6C5GW7XJNY

만약 동일 node에 통신을 하려는 두 pod가 있는 경우는

## internaltrafficpolicy = local and pods are in same node. 
root@aks-nodepool1-23236778-vmss000002:/# iptables-save |grep nginx
-A KUBE-SEP-HLUNARSNCDYYPHUV -s 10.240.0.68/32 -m comment --comment "default/nginx-sample-svc" -j KUBE-MARK-MASQ
-A KUBE-SEP-HLUNARSNCDYYPHUV -p tcp -m comment --comment "default/nginx-sample-svc" -m tcp -j DNAT --to-destination 10.240.0.68:8080
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-SVL-JT67RD6F3OETQGP2
-A KUBE-SVL-JT67RD6F3OETQGP2 -d 10.0.130.191/32 ! -i azv+ -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-MARK-MASQ
-A KUBE-SVL-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.68:8080" -j KUBE-SEP-HLUNARSNCDYYPHUV

보통 아래와 같은 service endpoint 에 매칭되는 rule이 존재하며 nodeIP:port로 지정되어 있다.
(여기서 10.240.0.41 는 nginx-sample pod가 동작되고 있는 node이다.)

-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.41:8080" -j KUBE-SEP-I4JGEQAI6S5SYXV7

실제 pod를 4개로 늘린 경우 아래와 같이 가중치가 붙는 형태로 rule이 생성된다.
(여기서 10.240.0.73 이 실제 확인중인 node의 ip이다. 즉, 동일 node는 가중치가 없고 바로 연결을 지향한다.)

-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.148:8080" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-44HLQTUCEZ7VEBKD
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.29:8080" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-7CLJKZYKU4YYJTUQ
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.41:8080" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-I4JGEQAI6S5SYXV7
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.73:8080" -j KUBE-SEP-PD4DNTIBFNIHTOXJ

envoy gateway api controller

Jacob_baek — Mon, 17 Nov 2025 14:36:18 +0900

Background

https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/

Envoy controller

https://gateway.envoyproxy.io/docs/tasks/quickstart/

Installation

https://gateway.envoyproxy.io/docs/tasks/quickstart/

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.6.0 -n envoy-gateway-system --create-namespace

https://gateway.envoyproxy.io/latest/install/install-yaml/

quick start and learn how it works

$ kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.6.0/quickstart.yaml -n default
gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created

following the below service, you can access URL using external-ip

$ kubectl get svc -n envoy-gateway-system -l app.kubernetes.io/instance=eg
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                            AGE
envoy-gateway   ClusterIP   10.0.173.35   <none>        18000/TCP,18001/TCP,18002/TCP,19001/TCP,9443/TCP   76m
$ kubectl get svc -n envoy-gateway-system -l app.kubernetes.io/name=envoy
NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
envoy-default-eg-e41e7b31   LoadBalancer   10.0.42.162   x.x.x.x        80:30436/TCP   72m
$ kubectl get gateway
NAME   CLASS   ADDRESS        PROGRAMMED   AGE
eg     eg      x.x.x.x        True         74m

ingress와의 차이점

	ingress	gateway api
component	ingress	gateway / httproute
additional features	annotation	CRD로 추가 제공

기본 생성시에는 ingress와 비교해보았을때에는 gateway 와 httproute만 제공되면 서비스가 가능하다.

gateway : listener 역할 수행
httproute : ingress에서 path 로 ingress내에 선언하였던 부분을 분리하여 설정할수 있다.
그외에도 다양한 crd를 활용할 수 있다.

$ kubectl get crd | grep gateway
backends.gateway.envoyproxy.io                        2025-11-21T04:43:10Z
backendtlspolicies.gateway.networking.k8s.io          2025-11-21T04:43:08Z
backendtrafficpolicies.gateway.envoyproxy.io          2025-11-21T04:43:11Z
clienttrafficpolicies.gateway.envoyproxy.io           2025-11-21T04:43:12Z
envoyextensionpolicies.gateway.envoyproxy.io          2025-11-21T04:43:12Z
envoypatchpolicies.gateway.envoyproxy.io              2025-11-21T04:43:13Z
envoyproxies.gateway.envoyproxy.io                    2025-11-21T04:43:14Z
gatewayclasses.gateway.networking.k8s.io              2025-11-21T04:43:08Z
gateways.gateway.networking.k8s.io                    2025-11-21T04:43:08Z
grpcroutes.gateway.networking.k8s.io                  2025-11-21T04:43:08Z
httproutefilters.gateway.envoyproxy.io                2025-11-21T04:43:15Z
httproutes.gateway.networking.k8s.io                  2025-11-21T04:43:09Z
referencegrants.gateway.networking.k8s.io             2025-11-21T04:43:08Z
securitypolicies.gateway.envoyproxy.io                2025-11-21T04:43:16Z
tcproutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
tlsroutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
udproutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
xbackendtrafficpolicies.gateway.networking.x-k8s.io   2025-11-21T04:43:08Z
xlistenersets.gateway.networking.x-k8s.io             2025-11-21T04:43:08Z
xmeshes.gateway.networking.x-k8s.io                   2025-11-21T04:43:07Z

configuration

envoy-gateway.yaml

Usages

whitelist

$ curl -H "Host: www.example.com" xxx.xxx.xxx.xxx
RBAC: access denied

spec:
  authorization:
    defaultAction: Deny
    rules:
    - action: Allow
      principal:
        clientCIDRs:
        - 218.238.135.0/24
        - 4.194.122.0/24

AKS-MCP

Jacob_baek — Mon, 18 Aug 2025 11:26:55 +0900

What is AKS MCP

You can operate your AKS cluster using AI.

There are 15 functions

Prerequiste

VSCode
MCP binary (https://github.com/Azure/aks-mcp/releases)

I used aks-mcp binary on WSL.

How to use

vscode and github copilot

github copilot

configure tools on github copilot

mcp.json file

Test result

you should input your subscription.

claude desktop

claude_desktop_config.json file.

restart claude desktop and you can see below Local MCP servers
(I recommend to Exit button on the menu)

Test result

doesn't need subscription id.

References

how to check ACR login user

Jacob_baek — Mon, 4 Aug 2025 14:24:28 +0900

Refresh token을 사용하여 아래와 같은 JWT 형태의 token을 decode 해볼수 있다.

JWT=$(az acr login -n <ACRName> -t --query refreshToken -o tsv)
jq -R 'split(".") | .[0],.[1] | @base64d | fromjson' <<< $(echo "$JWT")

References

https://prefetch.net/blog/2020/07/14/decoding-json-web-tokens-jwts-from-the-linux-command-line/

ingress-nginx

Jacob_baek — Mon, 14 Jul 2025 11:09:52 +0900

아래와 같은 에러가 발생되는 경우

Error from server (BadRequest): error when creating "nginx-with-svc-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configuration

다음과 같은 방식으로 ingress-nginx를 업데이트 해줘야 한다.
helm 으로 ingress-nginx 를 배포하는 경우 아래와 같은 annotations-risk-level 과 allowSnippetAnnotations 설정추가가 필요하다.

helm upgrade --install ingress-nginx ingress-nginx \
 --repo [https://kubernetes.github.io/ingress-nginx](https://kubernetes.github.io/ingress-nginx) \
 --namespace ingress-nginx --create-namespace \
 --set controller.allowSnippetAnnotations=true \
 --set controller.config.annotations-risk-level=Critical \
 --set controller.image.tag=v1.10.1 \
 --set controller.service.annotations."service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path"=/healthz

https://ellie.wtf/notes/ingress-nginx-risky-annotations

Entra ID token

Jacob_baek — Wed, 26 Feb 2025 10:20:06 +0900

Azure infra 상에서 동작되는 app에서 azure infra의 resource를 사용하거나 접근해야할 경우 token 기반으로 접근이 이루어지는 경우들이 있다. 해당 token은 Entra ID에서 발급받고 관리되어진다. 이와 같은 경우 token에 대한 이해가 없다면 동작방식을 이해하는데 어려움이 따를수 있다.

Tokens

제공되는 token의 종류는 총 3가지로 아래와 같다.

Access Token : Oauth2 용(즉, 허가용)
Refresh Token : Access Token 재발급을 위한 token
ID token : OIDC 용 (즉, 인증용)

https://learn.microsoft.com/en-us/entra/identity-platform/security-tokens

Token configuration

Access token lifecycle : 기본으로 60 ~ 90 min (https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens#token-lifetime)
Refresh token lifecycle :
- 사용할때마다 갱신 되며 최대는 90 days (https://learn.microsoft.com/en-us/entra/identity-platform/configurable-token-lifetimes#refresh-and-session-token-lifetime-policy-properties)
- access token을 1시간마다 재발급 받는 과정에서 refresh token도 새로 발급됨 : https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-lifetime

Refresh token expire time은 지정불가
https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-timeouts

default token revocation 기간

https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-revocation

2021-01-30 이후로 default로만 제공되며 이전에는 refresh token 설정이 가능했다.
현재는 conditional access 정책으로 관리하도록 한다.
https://learn.microsoft.com/en-us/entra/identity-platform/configurable-token-lifetimes#refresh-and-session-token-lifetime-policy-properties

다음과 같은 conditional access 정책에 따라 인증을 해제할수 있다.

https://learn.microsoft.com/en-us/entra/identity/conditional-access/howto-conditional-access-session-lifetime

Refresh Token Expire

최대 수명 : 90일
(Refresh Token이 rotation을 계속 하더라도 90일 이후에는 재로그인이 필요)
idle Timeout : 14일
(Refresh Token 사용 안하고 14일 지나면 expire)
- 여기서 사용을 안한다는 의미는 cli나 sdk 등을 통한 인증과정이 한번이라도 포함된 명령 수행이 없는 경우를 의미해.

Refresh token이 만료되지 않은 경우 cli/sdk 등을 통한 인증과정을 같이 수행하는 명령이 수행되어지는 경우
Access Token이 만료되어 있다면 재발급이 이루어진다. 즉, CLI / SDK / Application 이 idle 상태로 인증을 수행하는 과정이 없는 경우라면 재발급이 발생되지 않는다.

Refresh tokens replace themselves with a fresh token upon every use
https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-lifetime

References

https://www.cswrld.com/2024/06/microsoft-entra-id-token-lifetime-and-revocation/

fluentbit with azure blob storage

Jacob_baek — Tue, 27 Aug 2024 13:32:00 +0900

installation

fluentbit install using helm chart

$ helm repo add fluent https://fluent.github.io/helm-charts
$ kubectl create ns logging
$ helm upgrade --install fluent-bit fluent/fluent-bit -n logging

create storage account and blob container

$ az storage account create -n fluentbitteststor -g fluentbittest-rg -l koreacentral --sku Standard_LRS

config for azure blob

$ kubectl edit configmap fluent-bit -n logging
...
    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

https://docs.fluentbit.io/manual/pipeline/outputs/azure_blob

full configmap

  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

    [INPUT]
        Name tail
        Path /var/log/*.log
        Tag system.*

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

restart daemonset

After modifying the fluent-bit configmap, run rollout like below.

$ kubectl rollout restart ds fluent-bit -n logging
daemonset.apps/fluent-bit restarted

storage browser

kubernetes authentication with cURL using service account

Jacob_baek — Tue, 2 Apr 2024 16:44:55 +0900

kubernetes service account를 사용하여 cURL로 kubernetes에 접근하는 방법을 알아보자.

아래와 같은 yaml을 통해 SA(service account) role, rolebinding 그리고 secret을 생성한다.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: onlypods
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curltest
  namespace: default
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: onlyreadpods
  namespace: default
subjects:
- kind: ServiceAccount
  name:  curltest
roleRef:
  kind: Role
  name: onlypods
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: curltest-secret
  annotations:
    kubernetes.io/service-account.name: curltest

참고로 1.24 이후부터 service account를 생성하더라도 secret을 자동으로 만들지 않기에 직접 생성을 해주어야 한다.

이후 kubectl 명령 수행이 가능한 환경이라면 가능하다면 아래와 같은 TOKEN과 ca.crt를 가져와 cURL을 통핸 pod 정보를 가져올수 있다.

#!/bin/bash

APISERVER=$(kubectl config view -o jsonpath='{.clusters[].cluster.server}')

TOKEN=$(kubectl get secret curltest-secret -o jsonpath='{.data.token}' | base64 -d)

kubectl get secret curltest-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > test-ca.crt

curl -H "Authorization: Bearer $TOKEN" --cacert kube-ca.crt "$APISERVER/api/v1/namespaces/default/pods"

실제 아래와 같은 결과가 출력된다.

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
  ...

Reference

https://stackoverflow.com/questions/55415867/curl-kubernetes-with-serivceaccount-token-it-always-returns-unauthorized

Retina

Jacob_baek — Fri, 22 Mar 2024 17:42:51 +0900

Introduce Retina

네트워크 트래픽(capture)과 metric 수집하고 저장하는것을 도와주는 도구이다.

Metrics (eBPF 프로그램을 주입하여 지속가능한 형태로 저장)
Captures (일시적인 tcpdump 형태이며 custom 하게 지정이 가능함)

Retina 와 관련된 주요 링크들

Architecture

https://retina.sh/docs/intro#extendable-architecture

Install

공식적인 helm chart를 제공하고 있지는 않다.
이를 사용하기 위해서는 retina 공식 github 저장소에서 clone 받은 source를 이용하여

$ make helm-install
$ ## 혹은 operator를 통한 설치를 하고자 한다면,,
$ make helm-install-with-operator

실제 배포시 아래와 같은 형태로 helm install이 진행된다.

helm upgrade --install retina ./deploy/manifests/controller/helm/retina/ \
        --namespace kube-system \
        --set image.repository=ghcr.io/microsoft/retina/retina-agent \
        --set image.initRepository=ghcr.io/microsoft/retina/retina-init \
        --set image.tag=v0.0.1 \
        --set operator.tag=v0.0.1 \
        --set image.pullPolicy=Always \
        --set logLevel=info \
        --set os.windows=true \
        --set operator.enabled=true \
        --set operator.enableRetinaEndpoint=true \
        --set operator.repository=ghcr.io/microsoft/retina/retina-operator \
        --skip-crds \
        --set enabledPlugin_linux="\[dropreason\,packetforward\,linuxutil\,dns\,packetparser\]"

아래와 같은 deployment 및 daemonset이 배포되어진다.

$ k get ds -n kube-system -l k8s-app=retina
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
retina-agent       3         3         2       3            2           kubernetes.io/os=linux     3m17s
retina-agent-win   0         0         0       0            0           kubernetes.io/os=windows   3m17s

monitoring

additionalscrapconfigs를 추가한다.

$ cat retina-values.yaml
prometheus:
  prometheusSpec:
    additionalScrapeConfigs: |
      - job_name: "retina-pods"
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_container_name]
            action: keep
            regex: retina(.*)
          - source_labels:
              [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            separator: ":"
            regex: ([^:]+)(?::\d+)?
            target_label: __address__
            replacement: ${1}:${2}
            action: replace
          - source_labels: [__meta_kubernetes_pod_node_name]
            action: replace
            target_label: instance
        metric_relabel_configs:
          - source_labels: [__name__]
            action: keep
            regex: (.*)

https://github.com/microsoft/retina/blob/main/deploy/prometheus/values.yaml

$ helm upgrade -f retina-values.yaml kube-prom-stack prometheus-community/kube-prometheus-stack -n monitoring

How to use

retina binary

아래 링크를 따라 kubectl-retian binary를 설치한다.
(krew로 설치하거나 download를 받아 PATH가 지정된 곳에 넣어주어도 된다.)

https://retina.sh/docs/installation/cli

job을 통해 capture를 수행하기에 별도의 CRD 설치가 필요하지는 않다.
만약 blob의 token이 포함된 SAS등의 민감정보로 직접 명령을 수행하는것이 어렵다면
CRD를 통해서도 capture가 가능하다. (https://retina.sh/docs/captures/)
다시 말하자면, kubectl-retian binary 설치만으로 kubectl 명령으로 capture 및 blob에 저장이 가능하다.

hostpath로 capture 파일 생성은 아래와 같은 명령으로 수행 가능하다.

$ kubectl retina capture create --name capturetestjacob0 --namespace default --pod-selectors="app=nginx-sample"  --host-path /mnt/capture
ts=2024-06-04T10:58:53.947+0900 level=info caller=capture/create.go:243 msg="The capture duration is set to 1m0s"
ts=2024-06-04T10:58:53.947+0900 level=info caller=capture/create.go:289 msg="The capture file max size is set to 100MB"
ts=2024-06-04T10:58:53.947+0900 level=info caller=utils/capture_image.go:56 msg="Using capture workload image ghcr.io/microsoft/retina/retina-agent:v0.0.11 with version determined by CLI version"
ts=2024-06-04T10:58:53.948+0900 level=info caller=capture/crd_to_job.go:201 msg="HostPath is not empty" HostPath=/mnt/capture
ts=2024-06-04T10:58:54.083+0900 level=info caller=capture/crd_to_job.go:876 msg="The Parsed tcpdump filter is \"\""
ts=2024-06-04T10:58:54.109+0900 level=info caller=capture/create.go:369 msg="Packet capture job is created" namespace=default capture job=capturetestjacob0-2xbnf
ts=2024-06-04T10:58:54.109+0900 level=info caller=capture/create.go:125 msg="Please manually delete all capture jobs"
NAMESPACE   CAPTURE NAME        JOBS                      COMPLETIONS   AGE
default     capturetestjacob0   capturetestjacob0-2xbnf   0/1           1s

Azure Storage Account 의 blob에 capture 파일을 업로드 할수 있다.

$ $ kubectl retina capture create --namespace default --pod-selectors="app=nginx-sample" --blob-upload="https://azstorageaccountname.blob.core.windows.net/captureblob?sp=raw&st=xxxxxxxxxxxx" --name capturetestbyjacob1
ts=2024-06-04T10:55:48.130+0900 level=info caller=capture/create.go:243 msg="The capture duration is set to 1m0s"
ts=2024-06-04T10:55:48.130+0900 level=info caller=capture/create.go:289 msg="The capture file max size is set to 100MB"
ts=2024-06-04T10:55:48.264+0900 level=info caller=utils/capture_image.go:56 msg="Using capture workload image ghcr.io/microsoft/retina/retina-agent:v0.0.11 with version determined by CLI version"
ts=2024-06-04T10:55:48.266+0900 level=info caller=capture/crd_to_job.go:224 msg="BlobUpload is not empty"
ts=2024-06-04T10:55:48.331+0900 level=info caller=capture/crd_to_job.go:876 msg="The Parsed tcpdump filter is \"\""
ts=2024-06-04T10:55:48.349+0900 level=info caller=capture/create.go:369 msg="Packet capture job is created" namespace=default capture job=capturetestbyjacob1-2cz48
ts=2024-06-04T10:55:48.349+0900 level=info caller=capture/create.go:125 msg="Please manually delete all capture jobs"
ts=2024-06-04T10:55:48.349+0900 level=info caller=capture/create.go:127 msg="Please manually delete capture secret" namespace=default secret name=capture-blob-upload-secretsjvwh
NAMESPACE   CAPTURE NAME          JOBS                        COMPLETIONS   AGE
default     capturetestbyjacob1   capturetestbyjacob1-2cz48   0/1           0s

완료된후 kubectl-retina 에서 제공하는 capture list parameter로 확인 가능하며 job으로도 볼수 있다.

$ kubectl retina capture list
NAMESPACE   CAPTURE NAME           JOBS                         COMPLETIONS   AGE
default     retina-capture-2vqh6   retina-capture-2vqh6-cf4kg   1/1           4m30s
default     retina-capture-mb862   retina-capture-mb862-jzns9   1/1           2m16s
default     retina-capture-tckbw   retina-capture-tckbw-swlt7   1/1           4m42s

$ kubectl get jobs
NAME                         COMPLETIONS   DURATION   AGE
retina-capture-2vqh6-cf4kg   1/1           70s        4m17s
retina-capture-mb862-jzns9   1/1           71s        2m3s
retina-capture-tckbw-swlt7   1/1           80s        4m29s

위와 같이 capture 가 가능하며 실제로 storageaccount에서 확인해보면 아래와 같은 tar 파일이 생성된것을 확인할 수 있다.

azure storageaccount 및 hostpath, PVC를 통해 capture 된 tar 파일을 받을수도 있으며
s3 규격을 맞춘 storage 의 경우 지원이 된다.

https://retina.sh/docs/CRDs/Capture#fields

metric

아래 plugin을 추가하여 다양한
https://retina.sh/docs/metrics/plugins/packetforward

기본적으로 아래와 같은 metric이

root@aks-nodepool1-26826537-vmss00000A:/# curl 10.224.0.4:18080/metrics
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.0899e-05
go_gc_duration_seconds{quantile="0.25"} 6.75e-05
go_gc_duration_seconds{quantile="0.5"} 0.0001773
go_gc_duration_seconds{quantile="0.75"} 0.000183199
go_gc_duration_seconds{quantile="1"} 0.000394699
go_gc_duration_seconds_sum 0.000980497
go_gc_duration_seconds_count 6
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 30
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.21.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 4.1930528e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 6.376612e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.551113e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 198985
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 5.046112e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 4.1930528e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 8.97024e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.4802048e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 368726
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 2.53952e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 5.3772288e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7111824243566203e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 567711
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 482496
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 505176
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 8.2977432e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 611673
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 753664
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 753664
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 6.2255626e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 9
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1.4
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 52
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.11316992e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71118218107e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.338843136e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP rest_client_request_duration_seconds Request latency in seconds. Broken down by verb, and host.
# TYPE rest_client_request_duration_seconds histogram
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="0.005"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="0.025"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="0.1"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="0.25"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="0.5"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="1"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="2"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="4"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="8"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="15"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="30"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="60"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET",le="+Inf"} 2
rest_client_request_duration_seconds_sum{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET"} 0.039509378
rest_client_request_duration_seconds_count{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/api",verb="GET"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="0.005"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="0.025"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="0.1"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="0.25"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="0.5"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="1"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="2"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="4"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="8"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="15"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="30"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="60"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET",le="+Inf"} 2
rest_client_request_duration_seconds_sum{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET"} 0.021485134
rest_client_request_duration_seconds_count{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/apis",verb="GET"} 2
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="0.005"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="0.025"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="0.1"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="0.25"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="0.5"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="1"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="2"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="4"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="8"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="15"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="30"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="60"} 1
rest_client_request_duration_seconds_bucket{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET"} 0.001631895
rest_client_request_duration_seconds_count{host="https://karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443/version",verb="GET"} 1
# HELP rest_client_request_size_bytes Request size in bytes. Broken down by verb and host.
# TYPE rest_client_request_size_bytes histogram
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="64"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="256"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="512"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1024"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="4096"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="16384"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="65536"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="262144"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1.048576e+06"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="4.194304e+06"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1.6777216e+07"} 5
rest_client_request_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="+Inf"} 5
rest_client_request_size_bytes_sum{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET"} 0
rest_client_request_size_bytes_count{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET"} 5
# HELP rest_client_requests_total Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",method="GET"} 5
# HELP rest_client_response_size_bytes Response size in bytes. Broken down by verb and host.
# TYPE rest_client_response_size_bytes histogram
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="64"} 0
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="256"} 0
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="512"} 1
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1024"} 1
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="4096"} 1
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="16384"} 3
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="65536"} 5
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="262144"} 5
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1.048576e+06"} 5
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="4.194304e+06"} 5
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="1.6777216e+07"} 5
rest_client_response_size_bytes_bucket{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET",le="+Inf"} 5
rest_client_response_size_bytes_sum{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET"} 84242
rest_client_response_size_bytes_count{host="karpenter2-karpenter2test-6fb462-rcp6iqyi.hcp.koreacentral.azmk8s.io:443",verb="GET"} 5

drop_count
drop_bytes
forward_count
forward_bytes
tcp_state
tcp_connection_remote
tcp
https://github.com/microsoft/retina/blob/9f455ca581af8b5355386d55ed0a70e2ee86ef20/pkg/utils/metric_names.go#L14-L35

capture

azure blob storage로 capture 파일을 전달

https://retina.sh/docs/captures/#example-1

kubectl-retina

CLI 도구로 사용이 가능하며 사용시 kubernetes 의 Node 단에 capture된 패킷 덤프 파일이 남겨지게 된다.

$ kubectl-retina
Usage:
   [command]

Available Commands:
  capture     Retina Capture - capture network traffic
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  version     Show version

Flags:
  -h, --help   help for this command

Use " [command] --help" for more information about a command.

https://retina.sh/docs/captures/cli

node 가 3개일 경우 3개의 job이 생성

$ kubectl retina capture list --namespace capture
NAMESPACE   CAPTURE NAME           JOBS                                                                               COMPLETIONS   AGE
capture     retina-capture-n4h8m   retina-capture-n4h8m-ffkpp,retina-capture-n4h8m-q75gj,retina-capture-n4h8m-zwfpp   3/3           5h40m

$ k get job -A
NAMESPACE   NAME                         COMPLETIONS   DURATION   AGE
capture     retina-capture-n4h8m-ffkpp   1/1           84s        5h40m
capture     retina-capture-n4h8m-q75gj   1/1           84s        5h40m
capture     retina-capture-n4h8m-zwfpp   1/1           87s        5h40m
$ k get pod -n capture
NAME                               READY   STATUS      RESTARTS   AGE
retina-capture-n4h8m-ffkpp-x8kr9   0/1     Completed   0          5h40m
retina-capture-n4h8m-q75gj-tddqc   0/1     Completed   0          5h40m
retina-capture-n4h8m-zwfpp-nf5dz   0/1     Completed   0          5h40m

실제 노드로 접근(e.g. node-shell)하여 확인해보면 다음과 같은 경로에 tar 파일이 존재한다.

root@aks-nodepool1-26826537-vmss000006:/mnt/capture# ls -al
total 464
drwxr-xr-x 2 root root   4096 Mar 22 03:03 .
drwxr-xr-x 4 root root   4096 Mar 22 03:01 ..
-rw-r--r-- 1 root root 464849 Mar 22 03:03 retina-capture-n4h8m-aks-nodepool1-26826537-vmss000006-20240322030207UTC.tar.gz

custom build

https://retina.sh/docs/contributing/developing#environment-config
사전에 llvm 설치 필요.

$ sudo apt install llvm clang -y

이후 아래 명령을 통해 build 를 수행할수 있다.

$ make retina-binary

개인적으로 생각한 이점들

특정 CNI에 종석되어 있지 않다. eBPF를 사용하여 CNI에 제한없이 사용이 가능
packet dump를 하고 이를 다양한 목적지로 보내줄수 있다.
단순 packet dump 만이 아닌 socket 정보 부터 arp, iptables 정보등 다양한 네트워크 단의 정보를 확인해볼수 있도록 해준다.

Reference

https://github.com/microsoft/retina/blob/main/docs/01-Introduction/01-intro.md

how to check certificate with openssl

Jacob_baek — Wed, 29 Nov 2023 10:38:50 +0900

Subject : 소유자의 데이터로 domain 정보가 포함된다.
Issuer : CA를 의미

CA(Certificate Authority)

subject 및 issuer 확인

$ curl -sL https://certs.godaddy.com/repository/gdroot-g2.crt | openssl x509 -subject -noout
subject=C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2
$ curl -sL https://certs.godaddy.com/repository/gdroot-g2.crt | openssl x509 -issuer -noout
issuer=C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", CN = Go Daddy Root Certificate Authority - G2

expire time 확인

$ curl -sL https://certs.godaddy.com/repository/gdroot-g2.crt | openssl x509 -enddate -noout
notAfter=Dec 31 23:59:59 2037 GMT

certificate

subject 및 issuer 확인

$ cat jacobbaek.com/cert.pem  | openssl x509 -subject -noout
subject=CN = *.jacobbaek.com
$ cat jacobbaek.com/cert.pem  | openssl x509 -issuer -noout
issuer=C = US, O = Let's Encrypt, CN = R3

expire time 확인

$ cat jacobbaek.com/cert.pem  | openssl x509 -enddate -noout
notAfter=Jan 15 01:50:45 2024 GMT

certificate

아래는 실제 서비스 되고 있는 url에 직접 요청을 보내서 certificate 정보를 확인하는 방법이다.

server certificate 가져오기

$ SERVERURL="www.google.com"
$ openssl s_client -connect $SERVERURL:443 2>/dev/null </dev/null |  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

subjet 확인

$ SERVERURL="www.google.com"
$ openssl s_client -connect $SERVERURL:443 2>/dev/null </dev/null |  sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -subject -noout

ip와 domain을 포함시켜 요청을 보내 certificate 확인

$ openssl s_client -connect x.x.x.x:443 -servername testdomain.com < /dev/null

References

https://stackoverflow.com/questions/40061263/what-is-ca-certificate-and-why-do-we-need-it