Gatekeeper Library

이미 만들어놓은 policy들을 가져다 쓸수는 없을까? 라는 의문이 들수 있다.
이를 library라는 항목으로 github내에 yaml 파일들을 제공하고 있다.
이는 kustomize 도구를 통해 배포가 가능하다.

Library

공식 라이브러리로 제공되고 있으며 gatekeeper 의 기본동작방식에 맞게 다음 두가지 방식의 library를 제공한다.

• validation
• mutation
• https://github.com/open-policy-agent/gatekeeper-library/tree/master/library/general

How to use

다음과 같은 순서로 constraint을 만들고 실제 policy로서 동작할수 있도록 해야 한다.

1. library를 이용한 constrinat template 생성
2. 생성된 constraint template을 이용한 constraints 생성 및 적용

library를 이용한 constraint template 생성

아래 작업은 kustomize binary가 필요하다.

간단하게는 아래와 같은 kustomization 파일을 만들어

jacob@laptop:~ $ cat kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/open-policy-agent/gatekeeper-library/library
- custom-constraint-template.yaml
#- constraint.yaml # custom하게 만들어 사용중(혹은 사용할) constraint-template 이 있는 경우 여기에 파일명 추가
jacob@laptop:~ $ kustomize build . | kubectl apply -f -
constrainttemplate.templates.gatekeeper.sh/customct created
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sblockendpointeditdefaultrole created
constrainttemplate.templates.gatekeeper.sh/k8sblockloadbalancer created
constrainttemplate.templates.gatekeeper.sh/k8sblocknodeport created
constrainttemplate.templates.gatekeeper.sh/k8sblockwildcardingress created
constrainttemplate.templates.gatekeeper.sh/k8scontainerephemeralstoragelimit created
constrainttemplate.templates.gatekeeper.sh/k8scontainerlimits created
constrainttemplate.templates.gatekeeper.sh/k8scontainerratios created
constrainttemplate.templates.gatekeeper.sh/k8scontainerrequests created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowanonymous created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedtags created
constrainttemplate.templates.gatekeeper.sh/k8sexternalips created
constrainttemplate.templates.gatekeeper.sh/k8shorizontalpodautoscaler created
constrainttemplate.templates.gatekeeper.sh/k8shttpsonly created
constrainttemplate.templates.gatekeeper.sh/k8simagedigests created
constrainttemplate.templates.gatekeeper.sh/k8spoddisruptionbudget created
constrainttemplate.templates.gatekeeper.sh/k8spspallowedusers created
constrainttemplate.templates.gatekeeper.sh/k8spspallowprivilegeescalationcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspapparmor created
constrainttemplate.templates.gatekeeper.sh/k8spspautomountserviceaccounttokenpod created
constrainttemplate.templates.gatekeeper.sh/k8spspcapabilities created
constrainttemplate.templates.gatekeeper.sh/k8spspflexvolumes created
constrainttemplate.templates.gatekeeper.sh/k8spspforbiddensysctls created
constrainttemplate.templates.gatekeeper.sh/k8spspfsgroup created
constrainttemplate.templates.gatekeeper.sh/k8spsphostfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnamespace created
constrainttemplate.templates.gatekeeper.sh/k8spsphostnetworkingports created
constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created
constrainttemplate.templates.gatekeeper.sh/k8spspprocmount created
constrainttemplate.templates.gatekeeper.sh/k8spspreadonlyrootfilesystem created
constrainttemplate.templates.gatekeeper.sh/k8spspseccomp created
constrainttemplate.templates.gatekeeper.sh/k8spspselinuxv2 created
constrainttemplate.templates.gatekeeper.sh/k8spspvolumetypes created
constrainttemplate.templates.gatekeeper.sh/k8sreplicalimits created
constrainttemplate.templates.gatekeeper.sh/k8srequiredannotations created
constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
constrainttemplate.templates.gatekeeper.sh/k8srequiredprobes created
constrainttemplate.templates.gatekeeper.sh/k8srequiredresources created
constrainttemplate.templates.gatekeeper.sh/k8sstorageclass created
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
constrainttemplate.templates.gatekeeper.sh/k8suniqueserviceselector created
constrainttemplate.templates.gatekeeper.sh/noupdateserviceaccount created
constrainttemplate.templates.gatekeeper.sh/verifydeprecatedapi created

위와 같이 kustomize binary를 사용하여 배포하게 되면 추가한(custom-constraint-template.yaml) yaml을 포함한 crd들이 생성된다.
실제 crd는 gatekeeper의 constrainttemplates 이다.

jacob@laptop:~ $ kubectl get crd -o wide | grep "gatekeeper.sh"
assign.mutations.gatekeeper.sh                                      2023-06-03T11:36:30Z
assignimage.mutations.gatekeeper.sh                                 2023-06-03T11:36:30Z
assignmetadata.mutations.gatekeeper.sh                              2023-06-03T11:36:30Z
configs.config.gatekeeper.sh                                        2023-06-03T11:36:30Z
constraintpodstatuses.status.gatekeeper.sh                          2023-06-03T11:36:30Z
constrainttemplatepodstatuses.status.gatekeeper.sh                  2023-06-03T11:36:31Z
constrainttemplates.templates.gatekeeper.sh                         2023-06-03T11:36:31Z
customct.constraints.gatekeeper.sh                                  2023-06-04T02:23:59Z
expansiontemplate.expansion.gatekeeper.sh                           2023-06-03T11:36:31Z
k8sallowedrepos.constraints.gatekeeper.sh                           2023-06-04T02:23:59Z
k8sblockendpointeditdefaultrole.constraints.gatekeeper.sh           2023-06-04T02:23:59Z
k8sblockloadbalancer.constraints.gatekeeper.sh                      2023-06-04T02:23:59Z
k8sblocknodeport.constraints.gatekeeper.sh                          2023-06-04T02:24:02Z
k8sblockwildcardingress.constraints.gatekeeper.sh                   2023-06-04T02:24:04Z
k8scontainerephemeralstoragelimit.constraints.gatekeeper.sh         2023-06-04T02:24:04Z
k8scontainerlimits.constraints.gatekeeper.sh                        2023-06-04T02:24:06Z
k8scontainerratios.constraints.gatekeeper.sh                        2023-06-04T02:24:08Z
k8scontainerrequests.constraints.gatekeeper.sh                      2023-06-04T02:24:10Z
k8sdisallowanonymous.constraints.gatekeeper.sh                      2023-06-04T02:24:12Z
k8sdisallowedrepos.constraints.gatekeeper.sh                        2023-06-04T02:24:15Z
k8sdisallowedtags.constraints.gatekeeper.sh                         2023-06-04T02:24:15Z
k8sexternalips.constraints.gatekeeper.sh                            2023-06-04T02:24:17Z
k8shorizontalpodautoscaler.constraints.gatekeeper.sh                2023-06-04T02:24:19Z
k8shttpsonly.constraints.gatekeeper.sh                              2023-06-04T02:24:19Z
k8simagedigests.constraints.gatekeeper.sh                           2023-06-04T02:24:21Z
k8spoddisruptionbudget.constraints.gatekeeper.sh                    2023-06-04T02:24:23Z
k8spspallowedusers.constraints.gatekeeper.sh                        2023-06-04T02:24:26Z
k8spspallowprivilegeescalationcontainer.constraints.gatekeeper.sh   2023-06-04T02:24:28Z
k8spspapparmor.constraints.gatekeeper.sh                            2023-06-04T02:24:28Z
k8spspautomountserviceaccounttokenpod.constraints.gatekeeper.sh     2023-06-04T02:24:30Z
k8spspcapabilities.constraints.gatekeeper.sh                        2023-06-04T02:24:32Z
k8spspflexvolumes.constraints.gatekeeper.sh                         2023-06-04T02:24:33Z
k8spspforbiddensysctls.constraints.gatekeeper.sh                    2023-06-04T02:24:34Z
k8spspfsgroup.constraints.gatekeeper.sh                             2023-06-04T02:24:37Z
k8spsphostfilesystem.constraints.gatekeeper.sh                      2023-06-04T02:24:37Z
k8spsphostnamespace.constraints.gatekeeper.sh                       2023-06-04T02:24:39Z
k8spsphostnetworkingports.constraints.gatekeeper.sh                 2023-06-04T02:24:41Z
k8spspprivilegedcontainer.constraints.gatekeeper.sh                 2023-06-04T02:24:43Z
k8spspprocmount.constraints.gatekeeper.sh                           2023-06-04T02:24:45Z
k8spspreadonlyrootfilesystem.constraints.gatekeeper.sh              2023-06-04T02:24:48Z
k8spspseccomp.constraints.gatekeeper.sh                             2023-06-04T02:24:50Z
k8spspselinuxv2.constraints.gatekeeper.sh                           2023-06-04T02:24:50Z
k8spspvolumetypes.constraints.gatekeeper.sh                         2023-06-04T02:24:52Z
k8sreplicalimits.constraints.gatekeeper.sh                          2023-06-04T02:24:52Z
k8srequiredannotations.constraints.gatekeeper.sh                    2023-06-04T02:24:54Z
k8srequiredlabels.constraints.gatekeeper.sh                         2023-06-04T02:24:56Z
k8srequiredprobes.constraints.gatekeeper.sh                         2023-06-04T02:24:57Z
k8srequiredresources.constraints.gatekeeper.sh                      2023-06-04T02:24:59Z
k8sstorageclass.constraints.gatekeeper.sh                           2023-06-04T02:25:01Z
k8suniqueingresshost.constraints.gatekeeper.sh                      2023-06-04T02:25:03Z
k8suniqueserviceselector.constraints.gatekeeper.sh                  2023-06-04T02:25:05Z
modifyset.mutations.gatekeeper.sh                                   2023-06-03T11:36:31Z
mutatorpodstatuses.status.gatekeeper.sh                             2023-06-03T11:36:31Z
noupdateserviceaccount.constraints.gatekeeper.sh                    2023-06-04T02:25:07Z
providers.externaldata.gatekeeper.sh                                2023-06-03T11:36:31Z
verifydeprecatedapi.constraints.gatekeeper.sh                       2023-06-04T02:25:10Z

필요에 따라 아래와 같이 직접 library 파일들을 받아와 변경하여 사용할수도 있다.
공식 라이브러리를 아래와 같이 받아온후

jacob@laptop:~ $ git clone --depth 2 --sparse --no-checkout https://github.com/open-policy-agent/gatekeeper-library.git
Cloning into 'gatekeeper-library'...
remote: Enumerating objects: 1008, done.
remote: Counting objects: 100% (1008/1008), done.
remote: Compressing objects: 100% (660/660), done.
remote: Total 1008 (delta 364), reused 719 (delta 263), pack-reused 0
Receiving objects: 100% (1008/1008), 799.79 KiB | 11.42 MiB/s, done.
Resolving deltas: 100% (364/364), done.
jacob@laptop:~ $ cd gatekeeper-library/
jacob@laptop:~/gatekeeper-library (master)$ git sparse-checkout set library/general
jacob@laptop:~/gatekeeper-library (master)$ git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.

받아온 library를 필요에 따라 수정하고 kustomize로 적용한다.

jacob@laptop:~/gatekeeper-library (master)$ kustomize build . | kubectl apply -f -
constrainttemplate.templates.gatekeeper.sh/k8sallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sblockendpointeditdefaultrole created
constrainttemplate.templates.gatekeeper.sh/k8sblockloadbalancer created
constrainttemplate.templates.gatekeeper.sh/k8sblocknodeport created
constrainttemplate.templates.gatekeeper.sh/k8sblockwildcardingress created
constrainttemplate.templates.gatekeeper.sh/k8scontainerephemeralstoragelimit created
constrainttemplate.templates.gatekeeper.sh/k8scontainerlimits created
constrainttemplate.templates.gatekeeper.sh/k8scontainerratios created
constrainttemplate.templates.gatekeeper.sh/k8scontainerrequests created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowanonymous created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedrepos created
constrainttemplate.templates.gatekeeper.sh/k8sdisallowedtags created
constrainttemplate.templates.gatekeeper.sh/k8sexternalips created
constrainttemplate.templates.gatekeeper.sh/k8shorizontalpodautoscaler created
constrainttemplate.templates.gatekeeper.sh/k8shttpsonly created
constrainttemplate.templates.gatekeeper.sh/k8simagedigests created
constrainttemplate.templates.gatekeeper.sh/k8spoddisruptionbudget created
constrainttemplate.templates.gatekeeper.sh/k8spspautomountserviceaccounttokenpod created
constrainttemplate.templates.gatekeeper.sh/k8sreplicalimits created
constrainttemplate.templates.gatekeeper.sh/k8srequiredannotations created
constrainttemplate.templates.gatekeeper.sh/k8srequiredlabels created
constrainttemplate.templates.gatekeeper.sh/k8srequiredprobes created
constrainttemplate.templates.gatekeeper.sh/k8srequiredresources created
constrainttemplate.templates.gatekeeper.sh/k8sstorageclass created
constrainttemplate.templates.gatekeeper.sh/k8suniqueingresshost created
constrainttemplate.templates.gatekeeper.sh/k8suniqueserviceselector created
constrainttemplate.templates.gatekeeper.sh/noupdateserviceaccount created
constrainttemplate.templates.gatekeeper.sh/verifydeprecatedapi created

이렇게하면 다양한 constrainttemplate과 그에 맞는 crd들을 만들게 된다.
이는 constrainttemplate들만 만들어진것이기에 이를 사용자의 입맛에 맞도록 설정하여 policy로써의 동작이 될수 있도록 해야한다.

생성된 constraint template을 이용한 constraints 생성 및 적용

이제 실제 해당 constrainttemplate을 통한 제어가 어떻게 이루어질지를 정해야한다.
이는 각 library 하단에 보면 sample 들로 constraint 예제가 존재하여 이를 참고하여 생성하면 된다.
몇가지 링크를 남겨두니 이를 참고하여 constraints를 생성 하자.

• validation
  • disallowanonymous
  • httpsonly
  • requiredprobes
  • storageclass
  • ...
• mutation
  • privilege-escalation
  • ...

실제 하나를 예로 들어 직접 적용해보자.
아래는 probe를 강제화하는 K8sRequiredProbes CRD에 대한 constraint 예제이다.

jacob@laptop:~ $ cat constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredProbes
metadata:
  name: must-have-probes
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    probes: ["readinessProbe", "livenessProbe"]
    probeTypes: ["tcpSocket", "httpGet", "exec"]

• https://github.com/open-policy-agent/gatekeeper-library/blob/master/library/general/requiredprobes/samples/must-have-probes/constraint.yaml

이를 적용하고

jacob@laptop:~ $ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper-library/master/library/general/requiredprobes/samples/must-have-probes/constraint.yaml
k8srequiredprobes.constraints.gatekeeper.sh/must-have-probes created

동일 repo내 directory에 존재하는 sample 파일로 테스트를 해보자.
example_allowed.yaml / example_disallowed.yaml 두개 파일을 사용하여 아래와 같이 적용해본 결과 disallowed는 admission webhook에 의해 request deny 된것을 확인할 수 있다.

jacob@laptop:~ $ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper-library/master/library/general/requiredprobes/samples/must-have-probes/example_disallowed.yaml
Error from server (Forbidden): error when creating "https://raw.githubusercontent.com/open-policy-agent/gatekeeper-library/master/library/general/requiredprobes/samples/must-have-probes/example_disallowed.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [must-have-probes] Container <nginx-1> in your <Pod> <test-pod1> has no <livenessProbe>
[must-have-probes] Container <nginx-1> in your <Pod> <test-pod1> has no <readinessProbe>
[must-have-probes] Container <tomcat> in your <Pod> <test-pod1> has no <livenessProbe>
jacob@laptop:~ $ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper-library/master/library/general/requiredprobes/samples/must-have-probes/example_allowed.yaml
pod/test-pod1 created

실제 테스트 했던 두개의 파일을 비교해보면

jacob@laptop:~ $ diff -c example_allowed.yaml example_disallowed.yaml
*** example_allowed.yaml        2023-06-04 11:46:17.007263577 +0900
--- example_disallowed.yaml     2023-06-04 11:46:20.487263163 +0900
***************
*** 4,18 ****
    name: test-pod1
  spec:
    containers:
    - name: tomcat
      image: tomcat
      ports:
      - containerPort: 8080
-     livenessProbe:
-       tcpSocket:
-         port: 80
-       initialDelaySeconds: 5
-       periodSeconds: 10
      readinessProbe:
        tcpSocket:
          port: 8080
--- 4,25 ----
    name: test-pod1
  spec:
    containers:
+   - name: nginx-1
+     image: nginx:1.7.9
+     ports:
+     - containerPort: 80
+     livenessProbe:
+       # tcpSocket:
+       #   port: 80
+       # initialDelaySeconds: 5
+       # periodSeconds: 10
+     volumeMounts:
+     - mountPath: /tmp/cache
+       name: cache-volume
    - name: tomcat
      image: tomcat
      ports:
      - containerPort: 8080
      readinessProbe:
        tcpSocket:
          port: 8080

example_allowed는 readiness/liveness 모두 probe를 포함하여 정상적으로 선언되어 있는 반면에
example_disallowed의 경우 nginx-1는 readiness가 tomcat는 liveness가 정상적으로 선언되지 않아 있어 admission webhook에서 deny 시킨것으로 확인된다.

Summary

library형태로 이미 만들어져 있는 constrainttemplates을 손쉽게 생성하여 사용이 가능했다.
다만 실제 policy로 적용이 되야하는 constraint를 하나씩 확인하고 필요한 정책을 만들어야하는 번거로움(?)이 있었다.
사실 이는 각 항목들을 잘 이해하고 운영/보안상에서 조율이 잘된 설정들을 적용해야하기에 번거로움(?)이 있을지라도
이렇게 샘플을 통한 확인을 하고 설정하기를 권장하는것으로 보인다.

참고사이트

• https://open-policy-agent.github.io/gatekeeper-library/website/