Kubernetes默认提供了一些内置的准入控制器(Admission Webhook),它们负责管理集群中资源对象的创建、更新和删除。准入控制器可以调用Webhook服务,在创建Pod时更改配置(例如注入标签),或者在准入过程中验证Pod的配置,并给出放行
或拒绝
的响应。这种机制允许我们根据自定义逻辑在资源创建或更新的时候动态地修改其行为,从而为Kubernetes集群提供更灵活、更个性化的管理。
准入控制器的工作流程如下图所示:
从上图中可知,Admission Webhooks有两种类型:
- Mutating Admission Webhooks:变更型准入控制器,用于在资源存储到etcd之前通过Mutating Webhooks对资源进行修改。例如注入一个sidecar、挂载一个volumeMounts等。
- Validating Admission Webhooks:验证型准入控制器,用于在资源存储到etcd之前通过Validating Webhooks对资源进行自定义策略验证。并返回
放行
或拒绝
。
本文将讲解这两种准入控制器的实现方式。
Webhook工作方式
Admission Webhook和其它我们所熟知的webhook一样,说白了就是一个HTTP回调接口,Apiserver在收到一个资源的增删改查请求以后,会先去调用一下webhook,并附带上一个AdmissionReview
类型的请求参数,然后我们自己实现这个webhook逻辑,再返回一个AdmissionResponse
类型的返回值给Apiserver,Apiserver据此返回值来决定该资源是否要进行以及如何进行下一步操作。
既然是一个HTTP回调接口,我们就可以用任意语言(但仍然首选client-go框架)、任意Web框架来实现。
另外,Apiserver规定了webhook接口必须是https的,Apiserver会以https的方式发送给webhook,因此http不能被接受。
根据上述要求,我们最终选择go-gin框架来实现一个https接口。
生成自签CA证书和服务端证书
有两种方式可以方便的生成自签CA证书和服务端证书:cfssl工具和openssl工具。
cfssl
- 创建CA证书机构配置文件ca-config.json
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"server": {
"usages": ["signing", "key encipherment", "server auth", "client auth"],
"expiry": "876000h"
}
}
}
}
- 创建CA证书请求配置文件ca-csr.json
{
"CA":{"expiry":"876000h"},
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BJ",
"ST": "BJ",
"O": "k8s",
"OU": "System"
}
]
}
- 生成CA证书和私钥
## cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2023/12/13 15:18:40 [INFO] generating a new CA key and certificate from CSR
2023/12/13 15:18:40 [INFO] generate received request
2023/12/13 15:18:40 [INFO] received CSR
2023/12/13 15:18:40 [INFO] generating key: rsa-2048
2023/12/13 15:18:40 [INFO] encoded CSR
2023/12/13 15:18:40 [INFO] signed certificate with serial number 237767326485485052000983960416676961846303085620
执行完后生成ca.pem
和ca-key.pem
- 创建服务端证书请求配置文件tls-csr.json
{
"CN": "admission",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BJ",
"ST": "BJ",
"O": "k8s",
"OU": "System"
}
],
"hosts": [
"podaffinity-webhook-server",
"podaffinity-webhook-server.kube-system",
"podaffinity-webhook-server.kube-system.svc",
"192.168.126.100"
]
}
最重要的就是hosts字段,webhook要部署到哪个域名/IP下,就得把对应的域名/IP写进csr。
- 用前一步生成的CA证书签发服务端证书和私钥
## cfssl gencert -ca ca.pem -ca-key ca-key.pem -config ca-config.json -profile server tls-csr.json | cfssljson -bare tls
2023/12/13 15:24:01 [INFO] generate received request
2023/12/13 15:24:01 [INFO] received CSR
2023/12/13 15:24:01 [INFO] generating key: rsa-2048
2023/12/13 15:24:02 [INFO] encoded CSR
2023/12/13 15:24:02 [INFO] signed certificate with serial number 724880176019206124126086747821903513868891190856
执行完以后生成tls.pem
和tls-key.pem
,这两个就是我们需要的服务端证书和私钥文件。
如果webhook是单独部署,只要启动的时候能找到这两个文件就行。如果webhook以deployment形式部署到K8S,可以再用secret对象来存放这两个文件,然后再挂载进deployment:
kubectl create secret tls admission-certs --key=tls-key.pem --cert=tls.pem
openssl
- 生成CA私钥
openssl genrsa -out ca.key 4096
- 准备CA证书配置文件ca.conf
[ req ]
default_bits = 4096
distinguished_name = req_distinguished_name
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = BJ
localityName = Locality Name (eg, city)
localityName_default = BJ
organizationName = Organization Name (eg, company)
organizationName_default = Kubernetes
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
commonName_default = podaffinity-webhook-server.kube-system
- 生成CA证书请求文件
openssl req -new -sha256 -out ca.csr -key ca.key -config ca.conf
- 生成CA证书
openssl x509 -req -days 3650 -in ca.csr -signkey ca.key -out ca.crt
- 生成服务端私钥
openssl genrsa -out tls.key 2048
- 准备服务端证书配置文件tls.conf
[ req ]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = CN
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = BJ
localityName = Locality Name (eg, city)
localityName_default = BJ
organizationName = Organization Name (eg, company)
organizationName_default = LZBBY
commonName = Common Name (e.g. server FQDN or YOUR name)
commonName_max = 64
commonName_default = podaffinity-webhook-server.kube-system
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1 = podaffinity-webhook-server
DNS.2 = podaffinity-webhook-server.kube-system
DNS.3 = podaffinity-webhook-server.kube-system.svc
alt_names和上面cfssl的hosts是一样的,webhook要部署到哪个域名/IP下,就得把对应的域名/IP写进去
- 生成服务端证书请求文件
openssl req -new -sha256 -out tls.csr -key tls.key -config tls.conf
- 用前一步生成的CA证书签发服务端证书
openssl x509 -req -days 3650 -CA ca.crt -CAkey ca.key -CAcreateserial -in tls.csr -out tls.crt -extensions req_ext -extfile tls.conf
执行完以后生成tls.crt
和tls.key
两个文件即服务端证书和私钥。
现在服务端证书和私钥都准备好了,开始编写webhook逻辑。
Mutating Admission Webhook
本节将实现一个注入逻辑,给提交上来的Pod自动添加反亲和性配置podAntiAffinity
编写代码逻辑
- 新建一个工程目录podaffinity-webhook-server并初始化mod
mkdir podaffinity-webhook-server && cd podaffinity-webhook-server && go mod init podaffinity
- 按以下目录组织
## tree -L 1 podaffinity-webhook-server
podaffinity-webhook-server
├── bin
├── certs
├── Dockerfile
├── go.mod
├── go.sum
├── main.go
├── manifest
└── types
certs目录里面放前面生成好的服务端证书和私钥。manifest目录放置一会要用到的yaml文件。因为逻辑比较简单,所有代码都写在main.go里。
var (
certFile = kingpin.Flag("certFile", "SSL certificate file.").Short('c').Required().String()
keyFile = kingpin.Flag("keyFile", "SSL certificate key file.").Short('k').Required().String()
listeningAddress = kingpin.Flag("listen.address", "Address on which to expose metrics.").Default(":8443").Short('l').String()
gracefulStop = make(chan os.Signal)
)
func main() {
// Parse flags
kingpin.Version("0.1")
kingpin.HelpFlag.Short('h')
kingpin.Parse()
// listen to termination signals from the OS
signal.Notify(gracefulStop, syscall.SIGTERM)
signal.Notify(gracefulStop, syscall.SIGINT)
signal.Notify(gracefulStop, syscall.SIGHUP)
signal.Notify(gracefulStop, syscall.SIGQUIT)
// listener for the termination signals from the OS
go func() {
log.Infof("Listening and wait for graceful stop")
sig := <-gracefulStop
log.Infof("Caught signal: %+v. Wait 1 seconds...", sig)
time.Sleep(1 * time.Second)
log.Infof("Terminate program on port: %s", *listeningAddress)
os.Exit(0)
}()
router := gin.New()
router.GET("/", func(c *gin.Context) {
c.JSON(200, gin.H{
"data": "hello podaffinity-webhook-server!",
})
})
router.POST("/mutate", func(c *gin.Context) {
body, err := ioutil.ReadAll(c.Request.Body)
defer c.Request.Body.Close()
log.Infof("Received body: %+v", string(body))
if err != nil {
log.Error(err)
c.JSON(http.StatusInternalServerError, gin.H{
"code": http.StatusInternalServerError,
"message": err.Error(),
})
}
mutRes, err := mutate(body)
if err != nil {
log.Error(err)
c.JSON(http.StatusInternalServerError, gin.H{
"code": http.StatusInternalServerError,
"message": err.Error(),
})
}
c.Writer.Write(mutRes)
})
server := &http.Server{Addr: *listeningAddress, Handler: router}
server.ListenAndServeTLS(*certFile, *keyFile)
}
使用gin框架,注册了一个handler叫/mutate,收到的请求参数是一个json,将其作为参数传递给主要函数逻辑mutate。下来看一下这个mutate函数:
func mutate(body []byte) ([]byte, error) {
admReview := admissionv1.AdmissionReview{}
if err := json.Unmarshal(body, &admReview); err != nil {
return nil, fmt.Errorf("unmarshal request body failed: %v", err)
}
var err error
var pod *corev1.Pod
responseBody := []byte{}
ar := admReview.Request
resp := admissionv1.AdmissionResponse{}
if ar != nil {
if err := json.Unmarshal(ar.Object.Raw, &pod); err != nil {
return nil, fmt.Errorf("unable to unmarshal pod json object %v", err)
}
// set response options
resp.Allowed = true
resp.UID = ar.UID
pT := admissionv1.PatchTypeJSONPatch
resp.PatchType = &pT
// add some audit annotations, helpful to know why a object was modified.
resp.AuditAnnotations = map[string]string{
"mutateme": "webhook add it",
}
// Get Pod labels
labels := pod.ObjectMeta.GetLabels()
if appType, ok := labels["app_type"]; ok { // if has app_type, then will add PodAntiAffinity
log.Infof("Pod label app_type = %s", appType)
var patches []types.PatchOperation
op := types.PatchOperation{
Op: "add",
Path: "/spec/affinity",
Value: corev1.Affinity{
PodAntiAffinity: &corev1.PodAntiAffinity{
RequiredDuringSchedulingIgnoredDuringExecution: []corev1.PodAffinityTerm{
{
LabelSelector: &metav1.LabelSelector{
MatchExpressions: []metav1.LabelSelectorRequirement{
{
Key: "app_type",
Operator: "In",
Values: []string{appType},
},
},
},
TopologyKey: "kubernetes.io/hostname",
},
},
},
},
}
patches = append(patches, op)
// parse the []map into JSON
resp.Patch, _ = json.Marshal(patches)
}
// Success, of course ;)
resp.Result = &metav1.Status{
Status: "Success",
}
admReview.Response = &resp
// back into JSON so we can return the finished AdmissionReview w/ Response directly
// w/o needing to convert things in the http handler
responseBody, err = json.Marshal(admReview)
if err != nil {
return nil, err // untested section
}
}
return responseBody, nil
}
将请求参数body反序列化至AdmissionReview
类型的变量,从中拿出Pod,通过patch为Pod增加反亲和性配置,注入的path为/spec/affinity
。
patch的时候用到了一个struct叫types.PatchOperation
,在types目录下新建types.go
type PatchOperation struct {
Op string `json:"op"`
Path string `json:"path"`
Value interface{} `json:"value,omitempty"`
}
本例就注入了一个反亲和性配置。当然我们可以在mutate逻辑里随意注入其它属性字段,比如注入sidecar等。
接口最终需要返回一个AdmissionResponse
类型的返回值,该类型指定了几个必要的属性:UID
、Allowed
、Result
,最终将AdmissionResponse
封装到admReview.Response
里返回给Apiserver。
部署webhook
podaffinity-webhook-server写好以后可以独立于集群之外运行(也可以deployment形式部署于集群内运行),编译及启动方式:
go build -o bin/podaffinity-webhook-server
./bin/podaffinity-webhook-server -c certs/tls.pem -k certs/tls-key.pem
注册Mutating Webhook到集群
在manifest目录下创建文件mutatingwebhookconfig.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: podaffinity-webhook-server
webhooks:
- name: podaffinity-webhook-server.kube-system.svc
clientConfig:
caBundle: <CA BASE64>
#service:
## name: podaffinity-webhook-server
## namespace: kube-system
## path: "/mutate"
url: https://192.168.126.100:8443/mutate
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
failurePolicy: Fail
namespaceSelector:
matchLabels:
podaffinity-webhook-admission-injection: enabled
sideEffects: None
admissionReviewVersions: ["v1", "v1beta1"]
几个注意点:
webhooks.name
必须是FQDN格式,不能随便写。webhooks.clientConfig.caBundle
是前面生成的CA证书base64加密后的内容,注意加密后内容可能是换行的格式,要把他们连成一行。webhooks.clientConfig.url
和webhooks.clientConfig.service
二者必须取其一,如果webhook部署于集群内,可以写service。webhooks.rules
告诉K8S需要在CREATE POD的时候触发webhook。webhooks.namespaceSelector
指定了只有具有podaffinity-webhook-admission-injection: enabled
这个label的ns才需要触发webhook
把这个文件提交到K8S,然后给某ns打上label
kubectl apply -f manifest/mutatingwebhookconfig.yaml
kubectl label ns default podaffinity-webhook-admission-injection=enabled
测试
在default下面随便提交一个deployment,看Pod是否自动补充了两个volumes。
可能遇到的问题
如果Pod没建出来的话,一定要describe replicasets看看event:
## kubectl describe replicasets.apps nginx-5776d4fd9d -n default
……
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 45m (x16 over 47m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "podaffinity-webhook-server.kube-system.svc": failed to call webhook: Post "https://podaffinity-webhook-server.kube-system.svc:443/mutate?timeout=10s": x509: certificate is valid for podaffinity.webhook-server.kube-system.svc, not podaffinity-webhook-server.kube-system.svc
Warning FailedCreate 37m (x2 over 42m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "podaffinity-webhook-server.kube-system.svc": failed to call webhook: Post "https://podaffinity-webhook-server.kube-system.svc:443/mutate?timeout=10s": service "podaffinity-webhook-server" not found
Warning FailedCreate 9m28s (x2 over 26m) replicaset-controller Error creating: Internal error occurred: failed calling webhook "podaffinity-webhook-server.kube-system.svc": failed to call webhook: Post "https://podaffinity-webhook-server.kube-system.svc:443/mutate?timeout=10s": x509: certificate signed by unknown authority
原因:
- 在创建服务端证书请求文件的时候,hosts里没有加入对应的域名/IP,重新生成证书请求文件、证书即可。
- 没有找到service,创建service即可。
- caBundle的内容和服务端证书不匹配,按照前面的步骤重新生成。
Validating Admission Webhook
本节将实现一个验证型准入控制器逻辑,判断提交上来的Pod的image是否含有nginx子串,如果有则拒绝这个资源请求。
编写代码逻辑
main函数和mutating adminission webhook基本一样,只不过/mutate
换成了/validate
var (
certFile = kingpin.Flag("certFile", "SSL certificate file.").Short('c').Required().String()
keyFile = kingpin.Flag("keyFile", "SSL certificate key file.").Short('k').Required().String()
listeningAddress = kingpin.Flag("listen.address", "Address on which to expose metrics.").Default(":8443").Short('l').String()
gracefulStop = make(chan os.Signal)
)
func main() {
// Parse flags
kingpin.Version("0.1")
kingpin.HelpFlag.Short('h')
kingpin.Parse()
// listen to termination signals from the OS
signal.Notify(gracefulStop, syscall.SIGTERM)
signal.Notify(gracefulStop, syscall.SIGINT)
signal.Notify(gracefulStop, syscall.SIGHUP)
signal.Notify(gracefulStop, syscall.SIGQUIT)
// listener for the termination signals from the OS
go func() {
log.Infof("Listening and wait for graceful stop")
sig := <-gracefulStop
log.Infof("Caught signal: %+v. Wait 1 seconds...", sig)
time.Sleep(1 * time.Second)
log.Infof("Terminate program on port: %s", *listeningAddress)
os.Exit(0)
}()
router := gin.New()
router.GET("/", func(c *gin.Context) {
c.JSON(200, gin.H{
"data": "hello image-webhook-server!",
})
})
router.POST("/validate", func(c *gin.Context) {
body, err := ioutil.ReadAll(c.Request.Body)
defer c.Request.Body.Close()
log.Infof("Received body: %+v", string(body))
if err != nil {
log.Error(err)
c.JSON(http.StatusInternalServerError, gin.H{
"code": http.StatusInternalServerError,
"message": err.Error(),
})
}
validRes, err := validate(body)
if err != nil {
log.Error(err)
c.JSON(http.StatusInternalServerError, gin.H{
"code": http.StatusInternalServerError,
"message": err.Error(),
})
}
c.Writer.Write(validRes)
})
server := &http.Server{Addr: *listeningAddress, Handler: router}
server.ListenAndServeTLS(*certFile, *keyFile)
}
validate函数如下:
func validate(body []byte) ([]byte, error) {
admReview := admissionv1.AdmissionReview{}
if err := json.Unmarshal(body, &admReview); err != nil {
return nil, fmt.Errorf("unmarshal request body failed: %v", err)
}
var err error
var pod *corev1.Pod
responseBody := []byte{}
ar := admReview.Request
resp := admissionv1.AdmissionResponse{}
if ar != nil {
if err := json.Unmarshal(ar.Object.Raw, &pod); err != nil {
return nil, fmt.Errorf("unable to unmarshal pod json object %v", err)
}
// set response options
resp.UID = ar.UID
containers := pod.Spec.Containers
// Modify the Pod spec to include the volumes, then op the original pod.
for i := range containers {
if strings.Contains(containers[i].Image, "nginx") { // nginx image will be denied
msg := fmt.Sprintf("Image[%s] which has 'nginx' str is not allowed", containers[i].Image)
resp.Allowed = false
resp.Result = &metav1.Status{
Status: "Failure",
Code: int32(http.StatusForbidden),
Reason: metav1.StatusReason(msg),
Message: msg,
}
log.Errorln(msg)
break
} else {
resp.Allowed = true
resp.Result = &metav1.Status{
Status: "Success",
Code: int32(http.StatusOK),
}
}
}
admReview.Response = &resp
// back into JSON so we can return the finished AdmissionReview w/ Response directly
// w/o needing to convert things in the http handler
responseBody, err = json.Marshal(admReview)
if err != nil {
return nil, err // untested section
}
}
return responseBody, nil
}
部署webhook
略
注册Validate Webhook到集群
在manifest目录下创建文件validatingwebhookconfig.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: image-webhook-server
webhooks:
- name: image-webhook-server.kube-system.svc
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
clientConfig:
caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpRENDQW5DZ0F3SUJBZ0lVS2FYYkRMd0pNcGxIZEtVaFNqRUZkWGpqVkRRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1d6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUQWtKS01Rc3dDUVlEVlFRSEV3SkNTakVNTUFvRwpBMVVFQ2hNRGF6aHpNUTh3RFFZRFZRUUxFd1pUZVhOMFpXMHhFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13CklCY05Nak14TWpFek1EY3hOREF3V2hnUE1qRXlNekV4TVRrd056RTBNREJhTUZzeEN6QUpCZ05WQkFZVEFrTk8KTVFzd0NRWURWUVFJRXdKQ1NqRUxNQWtHQTFVRUJ4TUNRa294RERBS0JnTlZCQW9UQTJzNGN6RVBNQTBHQTFVRQpDeE1HVTNsemRHVnRNUk13RVFZRFZRUURFd3ByZFdKbGNtNWxkR1Z6TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGCkFBT0NBUThBTUlJQkNnS0NBUUVBdTJid2NtS0RweXk2NDZlRGtDRzdxbitIeEsyZFJqN2tuVk4vbGZwWldtRjkKMmg4ZENNN2xzRm1Ed3VhSHRmQnc0UUtFYnNGUklvMGVLV0szLzAvdUoyRlluLzI1aldNWGRmdjYyMVAvOE1CVgp6UG5vQkU0OENyck95b2dIRmU5NlBOVE9DaGI1K3JUeU9UV3pqbEFXb1ptNlBpVVF1M2pkTmh6Wk1ZRzJSKzRvCk9oekRranZYbjdzQWpnNmZHOWd2R2wrSStuY2tQdTg2OEJLWWNwclIwSGx1MjR4M015cjBHMUxYd1RIU0g0cDgKSUlOWitqZGxvamQvNTlkb2tuVTcyYWoxS1R0RjZaSXdzbk51cHc4TUhyVHNoZFVqTGZWZ2xDYVZ4WG1nR2F6SQp3NVExWGNnQ2hqUFVEMWxuNXJ6bjgwMktiTXpOdGRyd0hUR04vTTY2cHdJREFRQUJvMEl3UURBT0JnTlZIUThCCkFmOEVCQU1DQVFZd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUN6YkRaTGRVSlMyTVQvVW4KeHRUL3lIOVBLRXd3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUliRFlNdFg4SGN0NTlDTFNKRXRvbkF4OUxlQgpsc1RlSFN6OEUvZVJsMHRBaGFxZ1Mxd29lTXZBbnE2ZFBZVklSbmhveERDRGcybElJdTR4QWpPNWFUN3cwOUFtCkkxTlpFeGtmVGVGbW56VU92VkFxV0U3YTc2WEJLcHFkeWVhVlloYmw5ekVwWXZ1T1Y4YStvRWR4Ky83QmxJQnoKaU4xNlRCK2U2QWY5dWRYZEN5bFRYaGh1TE1panlRbUhYVnpxYUtvczdaTXNWeENSR0pjdWYyeS9lbmswZlpmSwpkbGxqUDdKWmUwa05NRkJ2QTgxUE80NlgyL3N4Snk0QUZKcEpGTnU2clJ5amZ0bDBHUDBLS29iSG9UUWdzRmQ5Cmx1a29odDExZEk0WUFQMnVySVA3dHVrSlNoZ241Q1VITzYvWDNvNGdDY2gzMkhsd2RWb3lTZ25TNzRNPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
url: https://192.168.126.100:8443/validate
admissionReviewVersions: ["v1"]
sideEffects: None
failurePolicy: Fail
namespaceSelector:
matchLabels:
podaffinity-webhook-admission-injection: enabled
提交这个yaml文件到集群,即完成注册。
测试
在default下面提交一个deployment,image设置成nginx:1.21.4,看该请求是否被拒绝。
## kubectl describe replicasets.apps nginx-5776d4fd9d -n default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 2s (x14 over 43s) replicaset-controller Error creating: admission webhook "image-webhook-server.kube-system.svc" denied the request: Image[nginx:1.21.4] which has 'nginx' str is not allowed