Jacob Baek's home

Why Rootless Docker-in-Docker Fails on Ubuntu 24.04+

Jacob_baek — Wed, 1 Jul 2026 10:45:13 +0900

Issue

With the adoption of Ubuntu 24.04 (Noble) in modern Kubernetes environments, many users have started encountering failures when running rootless Docker-in-Docker (DinD) containers.

A typical error looks like this:

[rootlesskit:parent] error: failed to start the child:2fork/exec /proc/self/exe: operation not permitted

This post explains the root cause, why it only happens on newer Ubuntu versions, and what actually works in practice.

Background: How Rootless DinD Works

docker:dind-rootless is designed to run Docker without root privileges.
Internally, it relies on:

dockerd-rootless.sh
  → rootlesskit
    → user namespace (userns)
    → mount namespace
    → network namespace
    → re-exec via /proc/self/exe
    → start dockerd

Key requirement:

RootlessKit must create an unprivileged user namespace.

Without this, the Docker daemon never starts.

What Changed in Ubuntu 24.04

Starting from Ubuntu 23.10 and fully enforced in 24.04:

Unprivileged user namespace creation is restricted by AppArmor by default.
https://discourse.ubuntu.com/t/understanding-apparmor-user-namespace-restriction/58007

This behavior is controlled by:

/proc/sys/kernel/apparmor_restrict_unprivileged_userns

When enabled:

Non-root processes cannot freely create user namespaces
Only processes explicitly allowed by AppArmor profiles can do so

Why this worked on Ubuntu 22.04
Nothing in Kubernetes changed — this is a host OS security change

Failure Mechanism

Rootless DinD starts
RootlessKit tries to: unshare(CLONE_NEWUSER)
Kernel checks AppArmor policy
AppArmor denies userns_create
RootlessKit fails when re-executing: fork/exec /proc/self/exe → EPERM

Why privileged: true Does NOT Fix It

Even ifsecurityContext set "privileged:true", it would be failed.

securityContext: 
  privileged: true

Because

privileged: true → grants capabilities
RootlessKit → runs as non-root user
Ubuntu 24.04 restriction → applies to unprivileged user namespace

How to fix it

1. Use Rootful DinD

apiVersion: v1
kind: Pod
metadata:
  name: dind-rootful
spec:
  containers:
    - name: dind
      image: docker:28.0.1-dind 
      securityContext:
        privileged: true

2. Use Ubuntu 22.04 Node

If you can use Ubuntu 22.04 node, it would be option that you can use DinD rootless.

nodeSelector:
  kubernetes.io/os-image: Ubuntu2204

3. Disable restriont (not recommended)

$ sysctl -w kernel.apparmor_restrict_unprivileged_userns=0

4. Custom AppArmor Profile (Advanced / Not Recommended in Managed K8s)

cat <<EOF > /etc/apparmor.d/rootlesskit
profile rootlesskit /usr/bin/rootlesskit flags=(unconfined) {
  userns,
}
EOF

systemctl restart apparmor

Test Result

I run the DinD rootless container on Ubuntu 22/24 OS.
The result is different.

Ubuntu 22

root@ubuntu22node:/# cat /proc/sys/kernel/unprivileged_userns_clone
1
### by apparmor
root@ubuntu22node:/# cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
cat: /proc/sys/kernel/apparmor_restrict_unprivileged_userns: No such file or directory

Ubuntu 24

root@ubuntu24node:/# cat /proc/sys/kernel/unprivileged_userns_clone
1
root@ubuntu24node:/# cat /proc/sys/kernel/apparmor_restrict_unprivileged_userns
1
root@ubuntu24node:/# sudo dmesg -T | grep -iE 'apparmor|userns|rootlesskit|DENIED' | tail -n 3
[Fri Jun 26 05:08:49 2026] audit: type=1400 audit(1782450529.255:124): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=8815 comm="rootlesskit" capability=21  capname="sys_admin"
[Fri Jun 26 05:09:04 2026] audit: type=1400 audit(1782450544.540:125): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=8860 comm="rootlesskit" requested="userns_create" target="unprivileged_userns"
[Fri Jun 26 05:09:29 2026] audit: type=1400 audit(1782450569.900:126): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=9028 comm="rootlesskit" requested="userns_create" target="unprivileged_userns"

Generating Apparmor custom profile

If you want to make custom profile and apply apparmor profile on your pod it would be good option.

References

AKS Node Disk Usage Analysis

Jacob_baek — Mon, 15 Jun 2026 10:46:24 +0900

Understanding Disk Pressure and Root Causes

Disk pressure on AKS nodes is a common issue in production environments.
While Kubernetes provides basic mechanisms such as image garbage collection, these are often insufficient to resolve real-world disk usage problems.

This post walks through how disk is actually consumed on AKS nodes, what frequently causes disk pressure, and how to systematically analyze it using a diagnostic script.

A helper script is available here:

Node Disk Analyzer

Why Disk Pressure Happens on AKS Nodes

From a Kubernetes perspective, node disk usage is not limited to a single component.
Instead, it is shared across multiple categories collectively known as local ephemeral storage.

In practice, disk pressure is usually caused by the following:

Container images (image cache)
Container writable layers (overlayfs)
Container logs
Pod volumes (such as emptyDir)

Understanding each of these categories is critical for accurate troubleshooting.

Key Disk Usage Categories

1. Container Images

Location: /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs

This is where container images are stored after being pulled.

Typical causes of growth:

Frequent deployments with new image tags
Large image sizes
Stale images not cleaned up

Kubelet manages this area through image garbage collection using:

imageGcHighThreshold
imageGcLowThreshold

However, this only applies to unused images.

2. Container Writable Layer (overlayfs)

Location: /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
This is the writable layer for running containers.

Any file created inside a container (excluding mounted volumes) is stored here.

Typical causes:

Applications writing logs to files instead of stdout
Temporary or cache data inside the container filesystem
High container churn (frequent restarts)

Important characteristics:

Not managed by image garbage collection
Often a major contributor to disk pressure
Difficult to notice without explicit analysis

3. Container Logs

Locations: /var/log/containers, /var/log/pods

These are stdout/stderr logs captured by Kubernetes.

Typical causes:

Verbose logging levels (debug/trace)
Lack of log rotation configuration
High request volume

Mitigation options:

containerLogMaxSizeMB
containerLogMaxFiles

4. Pod Volumes (emptyDir and others)

Location: /var/lib/kubelet/pods

This includes all pod-level data such as:

emptyDir
mounted volume data stored on node disk

Typical use cases:

temporary files
caching
data sharing between containers

Typical issues:

Applications continuously writing data without cleanup
Batch jobs generating files
Sidecars buffering data (e.g., log forwarders)

Unlike overlayfs, this is intentional storage defined by workload configuration.

Why Image GC Alone Is Not Enough

A common misconception is that increasing image GC thresholds will resolve disk pressure.

This is not accurate.

Image garbage collection only affects: container images (content store)

It does not address:

logs
overlayfs usage
emptyDir or pod volume data

In many cases, disk pressure persists even after image cleanup because the majority of usage is outside the image layer.

Approach to Disk Usage Analysis

To properly troubleshoot disk pressure, the goal is to answer:

Which category is consuming the most disk space?

A structured approach includes:

Measure usage per category
Compare relative proportions
Identify dominant contributor
Apply targeted mitigation

Diagnostic Script

To simplify this process, the following analyzer can be deployed to a specific node:

AKS Node Disk Analyzer

The script runs inside a privileged pod and inspects the host filesystem.

It provides:

Per-category disk usage (images, overlay, logs, volumes)
Percentage breakdown
Top contributing directories
Largest files on the node
Classification hints

Example Output Interpretation

A typical output may look like:

Usage Summary (KB)
Image   : 12,000,000
Overlay : 8,000,000
Log     : 2,000,000
Volume  : 500,000
TOTAL   : 22,500,000
Percentage (%)
Image   : 53%
Overlay : 35%
Log     : 8%
Volume  : 2%

How to interpret this

Image dominant (>50%):
- large image cache
- stale images not cleaned
Overlay dominant:
- application writing data inside container filesystem
Log dominant:
- excessive stdout logging
Volume dominant:
- emptyDir or mounted workload data growing

Troubleshooting Guidance by Category

Category	Typical Root Cause	Recommended Action
Image	Stale images, large images	Adjust GC, prune images
Overlay	File writes inside container	Change logging pattern, cleanup temp files
Log	Excessive stdout logging	Tune log rotation, reduce verbosity
Volume	emptyDir or workload-generated files	Add lifecycle cleanup, enforce limits

Best Practices

Always identify the dominant disk consumer before taking action
Do not rely solely on garbage collection
Ensure application-level cleanup policies exist
Configure log rotation proactively
Monitor node disk usage continuously

Conclusion

Disk pressure in AKS is rarely caused by a single factor.
It is the result of how multiple layers in Kubernetes share the same node filesystem.

Accurate troubleshooting requires breaking down disk usage into:

images
overlayfs
logs
volumes

Using a structured approach and proper tooling allows you to identify the root cause quickly and apply the right mitigation strategy.

InternalTrafficPolicy

Jacob_baek — Tue, 25 Nov 2025 17:54:26 +0900

https://kubernetes.io/docs/concepts/services-networking/service-traffic-policy/

연결하려는 pod가 동일 node에 없는 경우 연결자체가 안된다.

## internaltrafficpolicy = local and pods are not in same node.
root@aks-nodepool1-23236778-vmss000002:/# iptables-save | grep -i nginx
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc has no local endpoints" -j DROP
## internaltrafficpolicy = cluster
root@aks-nodepool1-23236778-vmss000002:/# iptables-save | grep -i nginx
-A KUBE-SEP-VKFWAD6C5GW7XJNY -s 10.240.0.50/32 -m comment --comment "default/nginx-sample-svc" -j KUBE-MARK-MASQ
-A KUBE-SEP-VKFWAD6C5GW7XJNY -p tcp -m comment --comment "default/nginx-sample-svc" -m tcp -j DNAT --to-destination 10.240.0.50:8080
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-SVC-JT67RD6F3OETQGP2
-A KUBE-SVC-JT67RD6F3OETQGP2 -d 10.0.130.191/32 ! -i azv+ -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-MARK-MASQ
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.50:8080" -j KUBE-SEP-VKFWAD6C5GW7XJNY

만약 동일 node에 통신을 하려는 두 pod가 있는 경우는

## internaltrafficpolicy = local and pods are in same node. 
root@aks-nodepool1-23236778-vmss000002:/# iptables-save |grep nginx
-A KUBE-SEP-HLUNARSNCDYYPHUV -s 10.240.0.68/32 -m comment --comment "default/nginx-sample-svc" -j KUBE-MARK-MASQ
-A KUBE-SEP-HLUNARSNCDYYPHUV -p tcp -m comment --comment "default/nginx-sample-svc" -m tcp -j DNAT --to-destination 10.240.0.68:8080
-A KUBE-SERVICES -d 10.0.130.191/32 -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-SVL-JT67RD6F3OETQGP2
-A KUBE-SVL-JT67RD6F3OETQGP2 -d 10.0.130.191/32 ! -i azv+ -p tcp -m comment --comment "default/nginx-sample-svc cluster IP" -j KUBE-MARK-MASQ
-A KUBE-SVL-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.68:8080" -j KUBE-SEP-HLUNARSNCDYYPHUV

보통 아래와 같은 service endpoint 에 매칭되는 rule이 존재하며 nodeIP:port로 지정되어 있다.
(여기서 10.240.0.41 는 nginx-sample pod가 동작되고 있는 node이다.)

-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.41:8080" -j KUBE-SEP-I4JGEQAI6S5SYXV7

실제 pod를 4개로 늘린 경우 아래와 같이 가중치가 붙는 형태로 rule이 생성된다.
(여기서 10.240.0.73 이 실제 확인중인 node의 ip이다. 즉, 동일 node는 가중치가 없고 바로 연결을 지향한다.)

-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.148:8080" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-44HLQTUCEZ7VEBKD
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.29:8080" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-7CLJKZYKU4YYJTUQ
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.41:8080" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-I4JGEQAI6S5SYXV7
-A KUBE-SVC-JT67RD6F3OETQGP2 -m comment --comment "default/nginx-sample-svc -> 10.240.0.73:8080" -j KUBE-SEP-PD4DNTIBFNIHTOXJ

envoy gateway api controller

Jacob_baek — Mon, 17 Nov 2025 14:36:18 +0900

Background

https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/

Envoy controller

https://gateway.envoyproxy.io/docs/tasks/quickstart/

Installation

https://gateway.envoyproxy.io/docs/tasks/quickstart/

helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.6.0 -n envoy-gateway-system --create-namespace

https://gateway.envoyproxy.io/latest/install/install-yaml/

quick start and learn how it works

$ kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.6.0/quickstart.yaml -n default
gatewayclass.gateway.networking.k8s.io/eg created
gateway.gateway.networking.k8s.io/eg created
serviceaccount/backend created
service/backend created
deployment.apps/backend created
httproute.gateway.networking.k8s.io/backend created

following the below service, you can access URL using external-ip

$ kubectl get svc -n envoy-gateway-system -l app.kubernetes.io/instance=eg
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                                            AGE
envoy-gateway   ClusterIP   10.0.173.35   <none>        18000/TCP,18001/TCP,18002/TCP,19001/TCP,9443/TCP   76m
$ kubectl get svc -n envoy-gateway-system -l app.kubernetes.io/name=envoy
NAME                        TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
envoy-default-eg-e41e7b31   LoadBalancer   10.0.42.162   x.x.x.x        80:30436/TCP   72m
$ kubectl get gateway
NAME   CLASS   ADDRESS        PROGRAMMED   AGE
eg     eg      x.x.x.x        True         74m

ingress와의 차이점

	ingress	gateway api
component	ingress	gateway / httproute
additional features	annotation	CRD로 추가 제공

기본 생성시에는 ingress와 비교해보았을때에는 gateway 와 httproute만 제공되면 서비스가 가능하다.

gateway : listener 역할 수행
httproute : ingress에서 path 로 ingress내에 선언하였던 부분을 분리하여 설정할수 있다.
그외에도 다양한 crd를 활용할 수 있다.

$ kubectl get crd | grep gateway
backends.gateway.envoyproxy.io                        2025-11-21T04:43:10Z
backendtlspolicies.gateway.networking.k8s.io          2025-11-21T04:43:08Z
backendtrafficpolicies.gateway.envoyproxy.io          2025-11-21T04:43:11Z
clienttrafficpolicies.gateway.envoyproxy.io           2025-11-21T04:43:12Z
envoyextensionpolicies.gateway.envoyproxy.io          2025-11-21T04:43:12Z
envoypatchpolicies.gateway.envoyproxy.io              2025-11-21T04:43:13Z
envoyproxies.gateway.envoyproxy.io                    2025-11-21T04:43:14Z
gatewayclasses.gateway.networking.k8s.io              2025-11-21T04:43:08Z
gateways.gateway.networking.k8s.io                    2025-11-21T04:43:08Z
grpcroutes.gateway.networking.k8s.io                  2025-11-21T04:43:08Z
httproutefilters.gateway.envoyproxy.io                2025-11-21T04:43:15Z
httproutes.gateway.networking.k8s.io                  2025-11-21T04:43:09Z
referencegrants.gateway.networking.k8s.io             2025-11-21T04:43:08Z
securitypolicies.gateway.envoyproxy.io                2025-11-21T04:43:16Z
tcproutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
tlsroutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
udproutes.gateway.networking.k8s.io                   2025-11-21T04:43:08Z
xbackendtrafficpolicies.gateway.networking.x-k8s.io   2025-11-21T04:43:08Z
xlistenersets.gateway.networking.x-k8s.io             2025-11-21T04:43:08Z
xmeshes.gateway.networking.x-k8s.io                   2025-11-21T04:43:07Z

configuration

envoy-gateway.yaml

Usages

whitelist

$ curl -H "Host: www.example.com" xxx.xxx.xxx.xxx
RBAC: access denied

spec:
  authorization:
    defaultAction: Deny
    rules:
    - action: Allow
      principal:
        clientCIDRs:
        - 218.238.135.0/24
        - 4.194.122.0/24

AKS-MCP

Jacob_baek — Mon, 18 Aug 2025 11:26:55 +0900

What is AKS MCP

You can operate your AKS cluster using AI.

There are 15 functions

Prerequiste

VSCode
MCP binary (https://github.com/Azure/aks-mcp/releases)

I used aks-mcp binary on WSL.

How to use

vscode and github copilot

github copilot

configure tools on github copilot

mcp.json file

Test result

you should input your subscription.

claude desktop

claude_desktop_config.json file.

restart claude desktop and you can see below Local MCP servers
(I recommend to Exit button on the menu)

Test result

doesn't need subscription id.

References

how to check ACR login user

Jacob_baek — Mon, 4 Aug 2025 14:24:28 +0900

Refresh token을 사용하여 아래와 같은 JWT 형태의 token을 decode 해볼수 있다.

JWT=$(az acr login -n <ACRName> -t --query refreshToken -o tsv)
jq -R 'split(".") | .[0],.[1] | @base64d | fromjson' <<< $(echo "$JWT")

References

https://prefetch.net/blog/2020/07/14/decoding-json-web-tokens-jwts-from-the-linux-command-line/

ingress-nginx

Jacob_baek — Mon, 14 Jul 2025 11:09:52 +0900

아래와 같은 에러가 발생되는 경우

Error from server (BadRequest): error when creating "nginx-with-svc-ingress.yaml": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: annotation group ConfigurationSnippet contains risky annotation based on ingress configuration

다음과 같은 방식으로 ingress-nginx를 업데이트 해줘야 한다.
helm 으로 ingress-nginx 를 배포하는 경우 아래와 같은 annotations-risk-level 과 allowSnippetAnnotations 설정추가가 필요하다.

helm upgrade --install ingress-nginx ingress-nginx \
 --repo [https://kubernetes.github.io/ingress-nginx](https://kubernetes.github.io/ingress-nginx) \
 --namespace ingress-nginx --create-namespace \
 --set controller.allowSnippetAnnotations=true \
 --set controller.config.annotations-risk-level=Critical \
 --set controller.image.tag=v1.10.1 \
 --set controller.service.annotations."service\\.beta\\.kubernetes\\.io/azure-load-balancer-health-probe-request-path"=/healthz

https://ellie.wtf/notes/ingress-nginx-risky-annotations

Entra ID token

Jacob_baek — Wed, 26 Feb 2025 10:20:06 +0900

Azure infra 상에서 동작되는 app에서 azure infra의 resource를 사용하거나 접근해야할 경우 token 기반으로 접근이 이루어지는 경우들이 있다. 해당 token은 Entra ID에서 발급받고 관리되어진다. 이와 같은 경우 token에 대한 이해가 없다면 동작방식을 이해하는데 어려움이 따를수 있다.

Tokens

제공되는 token의 종류는 총 3가지로 아래와 같다.

Access Token : Oauth2 용(즉, 허가용)
Refresh Token : Access Token 재발급을 위한 token
ID token : OIDC 용 (즉, 인증용)

https://learn.microsoft.com/en-us/entra/identity-platform/security-tokens

Token configuration

Access token lifecycle : 기본으로 60 ~ 90 min (https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens#token-lifetime)
Refresh token lifecycle :
- 사용할때마다 갱신 되며 최대는 90 days (https://learn.microsoft.com/en-us/entra/identity-platform/configurable-token-lifetimes#refresh-and-session-token-lifetime-policy-properties)
- access token을 1시간마다 재발급 받는 과정에서 refresh token도 새로 발급됨 : https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-lifetime

Refresh token expire time은 지정불가
https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-timeouts

default token revocation 기간

https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-revocation

2021-01-30 이후로 default로만 제공되며 이전에는 refresh token 설정이 가능했다.
현재는 conditional access 정책으로 관리하도록 한다.
https://learn.microsoft.com/en-us/entra/identity-platform/configurable-token-lifetimes#refresh-and-session-token-lifetime-policy-properties

다음과 같은 conditional access 정책에 따라 인증을 해제할수 있다.

https://learn.microsoft.com/en-us/entra/identity/conditional-access/howto-conditional-access-session-lifetime

Refresh Token Expire

최대 수명 : 90일
(Refresh Token이 rotation을 계속 하더라도 90일 이후에는 재로그인이 필요)
idle Timeout : 14일
(Refresh Token 사용 안하고 14일 지나면 expire)
- 여기서 사용을 안한다는 의미는 cli나 sdk 등을 통한 인증과정이 한번이라도 포함된 명령 수행이 없는 경우를 의미해.

Refresh token이 만료되지 않은 경우 cli/sdk 등을 통한 인증과정을 같이 수행하는 명령이 수행되어지는 경우
Access Token이 만료되어 있다면 재발급이 이루어진다. 즉, CLI / SDK / Application 이 idle 상태로 인증을 수행하는 과정이 없는 경우라면 재발급이 발생되지 않는다.

Refresh tokens replace themselves with a fresh token upon every use
https://learn.microsoft.com/en-us/entra/identity-platform/refresh-tokens#token-lifetime

References

https://www.cswrld.com/2024/06/microsoft-entra-id-token-lifetime-and-revocation/

fluentbit with azure blob storage

Jacob_baek — Tue, 27 Aug 2024 13:32:00 +0900

installation

fluentbit install using helm chart

$ helm repo add fluent https://fluent.github.io/helm-charts
$ kubectl create ns logging
$ helm upgrade --install fluent-bit fluent/fluent-bit -n logging

create storage account and blob container

$ az storage account create -n fluentbitteststor -g fluentbittest-rg -l koreacentral --sku Standard_LRS

config for azure blob

$ kubectl edit configmap fluent-bit -n logging
...
    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

https://docs.fluentbit.io/manual/pipeline/outputs/azure_blob

full configmap

  custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Daemon Off
        Flush 1
        Log_Level info
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 2020
        Health_Check On

    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        Name systemd
        Tag host.*
        Systemd_Filter _SYSTEMD_UNIT=kubelet.service
        Read_From_Tail On

    [INPUT]
        Name tail
        Path /var/log/*.log
        Tag system.*

    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [OUTPUT]
        name                  azure_blob
        match                 *
        account_name          fluentbitteststor
        shared_key            xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
        path                  k8slogs
        container_name        fluentbittest
        auto_create_container on
        tls                   on

restart daemonset

After modifying the fluent-bit configmap, run rollout like below.

$ kubectl rollout restart ds fluent-bit -n logging
daemonset.apps/fluent-bit restarted

storage browser

kubernetes authentication with cURL using service account

Jacob_baek — Tue, 2 Apr 2024 16:44:55 +0900

kubernetes service account를 사용하여 cURL로 kubernetes에 접근하는 방법을 알아보자.

아래와 같은 yaml을 통해 SA(service account) role, rolebinding 그리고 secret을 생성한다.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: onlypods
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: curltest
  namespace: default
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: onlyreadpods
  namespace: default
subjects:
- kind: ServiceAccount
  name:  curltest
roleRef:
  kind: Role
  name: onlypods
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: curltest-secret
  annotations:
    kubernetes.io/service-account.name: curltest

참고로 1.24 이후부터 service account를 생성하더라도 secret을 자동으로 만들지 않기에 직접 생성을 해주어야 한다.

이후 kubectl 명령 수행이 가능한 환경이라면 가능하다면 아래와 같은 TOKEN과 ca.crt를 가져와 cURL을 통핸 pod 정보를 가져올수 있다.

#!/bin/bash

APISERVER=$(kubectl config view -o jsonpath='{.clusters[].cluster.server}')

TOKEN=$(kubectl get secret curltest-secret -o jsonpath='{.data.token}' | base64 -d)

kubectl get secret curltest-secret -o jsonpath='{.data.ca\.crt}' | base64 -d > test-ca.crt

curl -H "Authorization: Bearer $TOKEN" --cacert kube-ca.crt "$APISERVER/api/v1/namespaces/default/pods"

실제 아래와 같은 결과가 출력된다.

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
  ...

Reference

https://stackoverflow.com/questions/55415867/curl-kubernetes-with-serivceaccount-token-it-always-returns-unauthorized