Monitoring APISIX Ingress Controller with Prometheus

This article introduces the relevant steps of how to use Prometheus to monitor APISIX Ingress Controller and the display effect of some indicators.

Whether in the days of monolithic applications or today's cloud-native era, "monitoring functions" have always played a very important role. A good monitoring system can help engineers quickly understand the status of services running in production environments, and quickly locate problems or warn of anomalies when they occur.

Apache APISIX Ingress Controller has been enhanced to support Prometheus Metrics in recent releases. In this article, we will introduce how to use Prometheus to collect Metrics data from APISIX Ingress Controller and subsequently visualize it through Grafana.

Step 1: Install APISIX Ingress Controller

First we deploy Apache APISIX, ETCD and APISIX Ingress Controller to the local Kubernetes cluster via Helm.

helm repo add apisix https://charts.apiseven.comhelm repo updatekubectl create namespace ingress-apisixhelm install apisix apisix/apisix --namespace ingress-apisix \ --set ingress-controller.enabled=true

After installation, please wait until all services are up and running. Specific status confirmation can be checked with the following command.

kubectl get all -n ingress-apisix

Step 2: Enable the Prometheus Plugin

In most cases, the monitoring function must involve more than just the APISIX Ingress Controller component. If you need to monitor Apache APISIX at the same time, you can create the following ApisixClusterConfig resource.

Installing Prometheus and Grafana

Next, we will enable the Prometheus service through the Prometheus Operator, so you will need to install the Prometheus Operator first.

note

The following command will also install Grafana.

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartsheml repo updatekubectl create namespace prometheushelm install prometheus prometheus-community/kube-prometheus-stack -n prometheus

After installation, you need to prepare the RBAC configuration for the Prometheus instance. This configuration gives Prometheus the ability to obtain Pod and Service resources from the Kubernetes API Server.

apiVersion: v1kind: ServiceAccountmetadata:  name: ingress-apisix  namespace: ingress-apisix---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  name: ingress-apisixrules:- apiGroups: [""]  resources:  - nodes  - nodes/metrics  - services  - endpoints  - pods  verbs: ["get", "list", "watch"]- apiGroups: [""]  resources:  - configmaps  verbs: ["get"]- apiGroups:  - networking.k8s.io  resources:  - ingresses  verbs: ["get", "list", "watch"]- nonResourceURLs: ["/metrics"]  verbs: ["get"]---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: ingress-apisixroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: ingress-apisixsubjects:- kind: ServiceAccount  name: ingress-apisix  namespace: ingress-apisix

After completing the above instance configuration, PodMonitor needs to be defined, or you can choose to use ServiceMonior depending on the scenario requirements. The following PodMonitor resources will focus on Metrics collection for the APISIX Ingress Controller Pod.

apiVersion: monitoring.coreos.com/v1kind: PodMonitormetadata:  name: ingress-apisix  namespace: ingress-apisix  labels:    use-for: ingress-apisixspec:  selector:    matchLabels:      app.kubernetes.io/name: ingress-controller  podMetricsEndpoints:  - port: http

note

The reason for not using ServiceMonitor here is that the http port is not exposed to the Service level.

Finally, the Prometheus instance can be defined with the following command.

apiVersion: monitoring.coreos.com/v1kind: Prometheusmetadata:  name: ingress-apisix  namespace: ingress-apisixspec:  serviceAccountName: ingress-apisix  podMonitorSelector:    matchLabels:      use-for: ingress-apisix  resources:    requests:      memory: 400Mi  enableAdminAPI: false  image: prom/prometheus:v2.31.0

After applying all of the above resources to the Kubernetes cluster, wait for the relevant components to be ready.

Step 3: Configuring Grafana

Next, we configure Grafana for visual presentation.

Start by accessing the prometheus-grafana service. Note that if you do not have a means to expose the service to the outside of the cluster, you can try using port forwarding. The Grafana administrator username and password are stored in the prometheus-grafana Secret.

After opening Grafana, import the Dashboard template of APISIX Ingress Controller to see the monitoring dashboard, which looks like the following.

For more information on creating Dashboard templates, please see related PR.

Addendum: Monitoring Metrics Explained

The current monitoring metrics for the APISIX Ingress Controller focus on its interaction with the data plane Apache APISIX instances (configuration delivery), including data related to the number and latency of configuration synchronization.

• is_leader: Whether the current APISIX Ingress Controller instance is in the Leader role. The same group of APISIX Ingress Controllers will have only one Leader, the rest of the instances are Candidate.
• sync_operations: Includes some metrics when the APISIX Ingress Controller synchronizes the configuration to the data plane, including the number of pushes, failure rate, latency, etc. Through these metrics, you can monitor whether the configuration delivery is normal or not, so as to help R&D and operation and maintenance students to monitor and alert and locate faults.

Summary

This article introduced how to use Prometheus to monitor APISIX Ingress Controller and how to display some of the metrics. Currently, only some basic monitoring metrics are included, we will continue to polish and upgrade, add more metrics and integrate data surface APISIX metrics to bring you a better monitoring experience.

Of course, interested parties are welcome to contribute to the Apache APISIX Ingress Controller project, and we look forward to working together to make the APISIX Ingress Controller more comprehensive.