When monitoring Kubernetes with Zabbix, the typical approach is to deploy a Zabbix Agent2 pod (or DaemonSet) on each node to collect host metrics, while a Zabbix Proxy pod gathers cluster-level data and forwards it to the Server.
However, if each node already has the zabbix-agent2 installed directly on the OS, or if your operational policy requires running only the Proxy inside the cluster, then deploying only the proxy pod and collecting node metrics through the host-level agent2 becomes the correct architecture.
Under this assumption, the goal is to deploy the Zabbix Proxy on worker nodes where taints are applied.

Related article:

Zabbix for Kubernetes Monitoring

---

Verified Environment

• Kubernetes: Ubuntu 24.04
• Zabbix Helm Chart: 7.4
• zabbix-agent2 installed directly on the host OS (no agent pods)
• Only Zabbix Proxy + kube-state-metrics (KSM) exist inside the cluster
• Zabbix agent DaemonSet is disabled in Helm chart (zabbixAgent.enabled: false)

---

0. Overall Architecture

• Zabbix Server lives outside the cluster
• The Server must be able to reach cluster node CIDR
• kubelet / node / API server templates work without Proxy
• But the “Kubernetes cluster state by HTTP” template requires a Proxy
  (KSM → Proxy → Server)

---

1. Prepare Helm values.yaml

# helm show values . > $HOME/zabbix_values.yaml

All remaining steps are based on $HOME/zabbix_values.yaml.

---

1.1 Disable Zabbix Agent pod (important)

Since each node already has a host-level zabbix-agent2 installed,
you must ensure the Helm chart does not deploy the Agent pod (DaemonSet).

Find the zabbixAgent section in values.yaml and configure:

This ensures:

• No Zabbix Agent pods are deployed inside Kubernetes
• Node monitoring is completely handled by host-installed zabbix-agent2
• Only Proxy / KSM related settings remain inside the cluster

---

2. Disable default proxy nodeSelector/tolerations

At the bottom of values.yaml, comment out the following entries to avoid conflicts:

---

3. Add nodeSelector + tolerations under zabbixProxy section

Find the following block:

zabbixProxy:
  enabled: true

Add the nodeSelector and tolerations directly under it:

---

📌 Important — How to Identify Proxy Node Hostname & Taint Values

(You must follow this section to correctly match tolerations with node taints.)

---

✔ 1) Identify node hostname (label)

# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" = "}{.metadata.labels.kubernetes\.io/hostname}{"\n"}{end}'

Example output:

worker-01 = worker-01

→ You only need to insert the exact value obtained from the node-label command into <example-system-1>.

Tip : In most Kubernetes clusters,
the hostname label value equals the node name.

---

✔ 2) Check taints applied to the node

These values must be placed exactly into tolerations.

Method A (most intuitive)

# kubectl describe node <node> | grep -A3 Taints

Example:

Taints:  Service=system:NoSchedule

Method B (jsonpath)

# kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" -> "}{.spec.taints}{"\n"}{end}'

Example:

worker-01 -> [{"effect":"NoSchedule","key":"Service","value":"system"}]

---

✔ 3) Toleration format to insert into values.yaml

If the node taint is:

Service=system:NoSchedule

values.yaml must contain:

tolerations:
  - key: Service
    operator: Equal
    value: system
    effect: NoSchedule

⚠ Important

• key/value/effect must match exactly
• even letter case differences (Service vs service) will break scheduling

---

4. Helm install

# helm install zabbix . \
  -n monitoring \
  --dependency-update \
  -f $HOME/zabbix_values.yaml

Confirm Proxy pod is running on the expected node:

# kubectl get pods -n monitoring -o wide

---

5. Handling kube-state-metrics (KSM) Pending issue

(Using patch is the most reliable workaround when values.yaml fails to apply)

In the environment of Ubuntu 24.04 + Zabbix Helm Chart 7.4,
tolerations configured inside values.yaml are not always applied to the KSM Deployment.

Therefore:

• It’s not that “values.yaml never works”
• It’s that in certain environments, toleration injection is skipped
• In such cases, the patch method is the most stable workaround

---

5.1 Workaround: kubectl patch

# kubectl patch deploy zabbix-kube-state-metrics -n monitoring --type=merge -p '
spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/hostname: <target-node-hostname>
      tolerations:
        - key: <taint-key>
          operator: "Equal"
          value: "<taint-value>"
          effect: "NoSchedule"
'

Verify:

# kubectl get pods -n monitoring

→ If zabbix-kube-state-metrics appears as Running, it is successful.

---

6. Concept Guide: Understanding Taint / Toleration

✔ taint

A taint restricts which pods can be scheduled on a node.

Example:

# kubectl taint nodes <node> key=value:NoSchedule

Only pods with a matching toleration may run on that node.

---

✔ toleration

A toleration declares that the pod is allowed to run on a node with a given taint.

Example:

tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"

---

✔ Matching summary

taint	toleration	result
key=value:NoSchedule	key=value + NoSchedule	OK
key=value:NoSchedule	key only, no value	mismatch → fail
NoExecute	tolerationSeconds	limited allow

---

7. Summary

• Node monitoring is handled by host-installed zabbix-agent2
• Helm chart must disable agent pods using zabbixAgent.enabled: false
• Only Zabbix Proxy + KSM are deployed inside Kubernetes
• Proxy tolerations are written directly under zabbixProxy section
• KSM tolerations may not apply via values.yaml → patching is the safest workaround
• taint key/value/effect must match exactly for scheduling to succeed

🛠 마지막 수정일: 2025.11.21