---
title: prometheus
keywords:
  - Apache APISIX
  - API Gateway
  - Plugin
  - Prometheus
description: The prometheus Plugin provides the capability to integrate APISIX with Prometheus for metric collection and continuous monitoring.
---

<!--
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
-->

<head>
  <link rel="canonical" href="https://docs.api7.ai/hub/prometheus" />
</head>

## Description

The `prometheus` Plugin provides the capability to integrate APISIX with [Prometheus](https://prometheus.io).

After enabling the Plugin, APISIX will start collecting relevant metrics, such as API requests and latencies, and exporting them in a [text-based exposition format](https://prometheus.io/docs/instrumenting/exposition_formats/#exposition-formats) to Prometheus. You can then create event monitoring and alerting in Prometheus to monitor the health of your API gateway and APIs.

## Static Configurations

By default, `prometheus` configurations are pre-configured in the [default configuration](https://github.com/apache/apisix/blob/master/apisix/cli/config.lua).

To customize these values, add the corresponding configurations to `config.yaml`. For example:

```yaml
plugin_attr:
  prometheus:                               # Plugin: prometheus attributes
    export_uri: /apisix/prometheus/metrics  # Set the URI for the Prometheus metrics endpoint.
    metric_prefix: apisix_                  # Set the prefix for Prometheus metrics generated by APISIX.
    enable_export_server: true              # Enable the Prometheus export server.
    export_addr:                            # Set the address for the Prometheus export server.
      ip: 127.0.0.1                         # Set the IP.
      port: 9091                            # Set the port.
    # metrics:                              # Create extra labels for metrics.
    #  http_status:                         # These metrics will be prefixed with `apisix_`.
    #    extra_labels:                      # Set the extra labels for http_status metrics.
    #      - upstream_addr: $upstream_addr
    #      - status: $upstream_status
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  http_latency:
    #    extra_labels:                      # Set the extra labels for http_latency metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    #  bandwidth:
    #    extra_labels:                      # Set the extra labels for bandwidth metrics.
    #      - upstream_addr: $upstream_addr
    #    expire: 0                          # The expiration time of metrics in seconds.
                                            # 0 means the metrics will not expire.
    # default_buckets:                      # Set the default buckets for the `http_latency` metrics histogram.
    #   - 10
    #   - 50
    #   - 100
    #   - 200
    #   - 500
    #   - 1000
    #   - 2000
    #   - 5000
    #   - 10000
    #   - 30000
    #   - 60000
    #   - 500
```

You can use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) to create `extra_labels`. See [add extra labels](#add-extra-labels-for-metrics).

Reload APISIX for changes to take effect.

## Attribute

| Name          | Type    | Required | Default | Valid values | Description                                |
| ------------- | ------- | -------- | ------- | ------------ | ------------------------------------------ |
| prefer_name | boolean |          | False   |              | If true, export Route/Service name instead of their ID in Prometheus metrics. |

## Metrics

There are different types of metrics in Prometheus. To understand their differences, see [metrics types](https://prometheus.io/docs/concepts/metric_types/).

The following metrics are exported by the `prometheus` Plugin by default. See [get APISIX metrics](#get-apisix-metrics) for an example. Note that some metrics, such as `apisix_batch_process_entries`, are not readily visible if there are no data.

| Name                    | Type      | Description                                                                                                                                                                   |
| ------------------------------ | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| apisix_bandwidth                      | counter   | Total amount of traffic flowing through APISIX in bytes.                                                       |
| apisix_etcd_modify_indexes            | gauge     | Number of changes to etcd by APISIX keys.                                                                                                                                         |
| apisix_batch_process_entries          | gauge     | Number of remaining entries in a batch when sending data in batches, such as with `http logger`, and other logging Plugins.  |
| apisix_etcd_reachable                 | gauge     | Whether APISIX can reach etcd. A value of `1` represents reachable and `0` represents unreachable.                                          |
| apisix_http_status                    | counter   | HTTP status codes returned from upstream Services.                                                            |
| apisix_http_requests_total            | gauge     | Number of HTTP requests from clients.                                                                                                                                     |
| apisix_nginx_http_current_connections | gauge     | Number of current connections with clients.                                                                                   |
| apisix_nginx_metric_errors_total      | counter   | Total number of `nginx-lua-prometheus` errors.                                                                                                                                |
| apisix_http_latency                   | histogram | HTTP request latency in milliseconds.                                                                                                              |
| apisix_node_info                      | gauge     | Information of the APISIX node, such as host name and the current APISIX version.                                                                                                                                                         |
| apisix_shared_dict_capacity_bytes     | gauge     | The total capacity of an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict).                                                                                                                     |
| apisix_shared_dict_free_space_bytes   | gauge     | The remaining space in an [NGINX shared dictionary](https://github.com/openresty/lua-nginx-module#ngxshareddict).                                                                                                                   |
| apisix_upstream_status                | gauge     | Health check status of upstream nodes, available if health checks are configured on the upstream. A value of `1` represents healthy and `0` represents unhealthy.                                                                 |
| apisix_stream_connection_total        | counter   | Total number of connections handled per Stream Route.                                                                                                               |

## Labels

[Labels](https://prometheus.io/docs/practices/naming/#labels) are attributes of metrics that are used to differentiate metrics.

For example, the `apisix_http_status` metric can be labeled with `route` information to identify which Route the HTTP status originates from.

The following are labels for a non-exhaustive list of APISIX metrics and their descriptions.

### Labels for `apisix_http_status`

The following labels are used to differentiate `apisix_http_status` metrics.

| Name   | Description                                                                                                                   |
| ------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| code         | HTTP response code returned by the upstream node.                                                                            |
| route        | ID of the Route that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route.                         |
| matched_uri  | URI of the Route that matches the request. Default to an empty string if a request does not match any Route.                              |
| matched_host | Host of the Route that matches the request. Default to an empty string if a request does not match any Route, or host is not configured on the Route.                             |
| service      | ID of the Service that the HTTP status originates from when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
| consumer     | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.                       |
| node         | IP address of the upstream node.                                                                                          |

### Labels for `apisix_bandwidth`

The following labels are used to differentiate `apisix_bandwidth` metrics.

| Name | Description                                                                                                                   |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------- |
| type       | Type of traffic, `egress` or `ingress`.                                                                                             |
| route      | ID of the Route that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route.                         |
| service    | ID of the Service that bandwidth corresponds to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
| consumer   | Name of the Consumer associated with a request. Default to an empty string if no Consumer is associated with the request.                       |
| node       | IP address of the upstream node.                                                                                          |

### Labels for `apisix_http_latency`

The following labels are used to differentiate `apisix_http_latency` metrics.

| Name | Description                                                                                                                         |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| type       | Type of latencies. See [latency types](#latency-types) for details. |
| route      | ID of the Route that latencies correspond to when `prefer_name` is `false` (default), and name of the Route when `prefer_name` to `true`. Default to an empty string if a request does not match any Route.                         |
| service    | ID of the Service that latencies correspond to when `prefer_name` is `false` (default), and name of the Service when `prefer_name` to `true`. Default to the configured value of host on the Route if the matched Route does not belong to any Service. |
| consumer   | Name of the Consumer associated with latencies. Default to an empty string if no Consumer is associated with the request.                             |
| node       | IP address of the upstream node associated with latencies.                                                                                                |

#### Latency Types

`apisix_http_latency` can be labeled with one of the three types:

* `request` represents the time elapsed between the first byte was read from the client and the log write after the last byte was sent to the client.

* `upstream` represents the time elapsed waiting on responses from the upstream Service.

* `apisix` represents the difference between the `request` latency and `upstream` latency.

In other words, the APISIX latency is not only attributed to the Lua processing. It should be understood as follows:

```text
APISIX latency
  = downstream request time - upstream response time
  = downstream traffic latency + NGINX latency
```

### Labels for `apisix_upstream_status`

The following labels are used to differentiate `apisix_upstream_status` metrics.

| Name | Description                                                                                         |
| ---------- | --------------------------------------------------------------------------------------------------- |
| name       | Resource ID corresponding to the upstream configured with health checks, such as `/apisix/routes/1` and `/apisix/upstreams/1`. |
| ip         | IP address of the upstream node.                                                                         |
| port       | Port number of the node.                                                                            |

## Examples

The examples below demonstrate how you can work with the `prometheus` Plugin for different scenarios.

### Get APISIX Metrics

The following example demonstrates how you can get metrics from APISIX.

The default Prometheus metrics endpoint and other Prometheus related configurations can be found in the [static configuration](#static-configurations). If you would like to customize these configuration, update `config.yaml` and reload APISIX.

If you deploy APISIX in a containerized environment and would like to access the Prometheus metrics endpoint externally, update the configuration file as follows and reload APISIX:

```yaml title="conf/config.yaml"
plugin_attr:
  prometheus:
    export_addr:
      ip: 0.0.0.0
```

Send a request to the APISIX Prometheus metrics endpoint:

```shell
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```text
# HELP apisix_bandwidth Total bandwidth in bytes consumed per Service in Apisix
# TYPE apisix_bandwidth counter
apisix_bandwidth{type="egress",route="",service="",consumer="",node=""} 8417
apisix_bandwidth{type="egress",route="1",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="egress",route="2",service="",consumer="",node="127.0.0.1"} 1420
apisix_bandwidth{type="ingress",route="",service="",consumer="",node=""} 189
apisix_bandwidth{type="ingress",route="1",service="",consumer="",node="127.0.0.1"} 332
apisix_bandwidth{type="ingress",route="2",service="",consumer="",node="127.0.0.1"} 332
# HELP apisix_etcd_modify_indexes Etcd modify index for APISIX keys
# TYPE apisix_etcd_modify_indexes gauge
apisix_etcd_modify_indexes{key="consumers"} 0
apisix_etcd_modify_indexes{key="global_rules"} 0
...
```

### Expose APISIX Metrics on Public API Endpoint

The following example demonstrates how you can disable the Prometheus export server that, by default, exposes an endpoint on port `9091`, and expose APISIX Prometheus metrics on a new public API endpoint on port `9080`, which APISIX uses to listen to other client requests.

Disable the Prometheus export server in the configuration file and reload APISIX for changes to take effect:

```yaml title="conf/config.yaml"
plugin_attr:
  prometheus:
    enable_export_server: false
```

Next, create a Route with [`public-api`](../../../en/latest/plugins/public-api.md) Plugin and expose a public API endpoint for APISIX metrics:

```shell
curl "http://127.0.0.1:9180/apisix/admin/routes/prometheus-metrics" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/apisix/prometheus/metrics",
    "plugins": {
      "public-api": {}
    }
  }'
```

Send a request to the new metrics endpoint to verify:

```shell
curl "http://127.0.0.1:9080/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```text
# HELP apisix_http_requests_total The total number of client requests since APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 1
# HELP apisix_nginx_http_current_connections Number of HTTP connections
# TYPE apisix_nginx_http_current_connections gauge
apisix_nginx_http_current_connections{state="accepted"} 1
apisix_nginx_http_current_connections{state="active"} 1
apisix_nginx_http_current_connections{state="handled"} 1
apisix_nginx_http_current_connections{state="reading"} 0
apisix_nginx_http_current_connections{state="waiting"} 0
apisix_nginx_http_current_connections{state="writing"} 1
...
```

### Monitor Upstream Health Statuses

The following example demonstrates how to monitor the health status of upstream nodes.

Create a Route with the `prometheus` Plugin and configure upstream active health checks:

```shell
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
    "uri": "/get",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1,
        "127.0.0.1:20001": 1
      },
      "checks": {
        "active": {
          "timeout": 5,
          "http_path": "/status",
          "healthy": {
            "interval": 2,
            "successes": 1
          },
          "unhealthy": {
            "interval": 1,
            "http_failures": 2
          }
        },
        "passive": {
          "healthy": {
            "http_statuses": [200, 201],
            "successes": 3
          },
          "unhealthy": {
            "http_statuses": [500],
            "http_failures": 3,
            "tcp_failures": 3
          }
        }
      }
    }
  }'
```

Send a request to the APISIX Prometheus metrics endpoint:

```shell
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```text
# HELP apisix_upstream_status upstream status from health check
# TYPE apisix_upstream_status gauge
apisix_upstream_status{name="/apisix/routes/1",ip="54.237.103.220",port="80"} 1
apisix_upstream_status{name="/apisix/routes/1",ip="127.0.0.1",port="20001"} 0
```

This shows that the upstream node `httpbin.org:80` is healthy and the upstream node `127.0.0.1:20001` is unhealthy.

### Add Extra Labels for Metrics

The following example demonstrates how to add additional labels to metrics and use the [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html) in label values.

Currently, only the following metrics support extra labels:

* apisix_http_status
* apisix_http_latency
* apisix_bandwidth

Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:

```yaml title="conf/config.yaml"
plugin_attr:
  prometheus:                                # Plugin: prometheus
    metrics:                                 # Create extra labels from the NGINX variables.
      http_status:
        extra_labels:                        # Set the extra labels for http_status metrics.
          - upstream_addr: $upstream_addr    # Add an extra upstream_addr label with value being the NGINX variable $upstream_addr.
          - route_name: $route_name          # Add an extra route_name label with value being the APISIX variable $route_name.
```

Note that if you define a variable in the label value but it does not correspond to any existing [APISIX variables](https://apisix.apache.org/docs/apisix/apisix-variable/) and [Nginx variable](https://nginx.org/en/docs/http/ngx_http_core_module.html), the label value will default to an empty string.

Create a Route with the `prometheus` Plugin:

```shell
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "id": "prometheus-route",
Include the following configurations in the configuration file to add labels for metrics and reload APISIX for changes to take effect:
    "name": "extra-label",
    "plugins": {
      "prometheus": {}
    },
    "upstream": {
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'
```

Send a request to the Route to verify:

```shell
curl -i "http://127.0.0.1:9080/get"
```

You should see an `HTTP/1.1 200 OK` response.

Send a request to the APISIX Prometheus metrics endpoint:

```shell
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```text
# HELP apisix_http_status HTTP status codes per Service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="1",matched_uri="/get",matched_host="",service="",consumer="",node="54.237.103.220",upstream_addr="54.237.103.220:80",route_name="extra-label"} 1
```

### Monitor TCP/UDP Traffic with Prometheus

The following example demonstrates how to collect TCP/UDP traffic metrics in APISIX.

Include the following configurations in `config.yaml` to enable stream proxy and `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect:

```yaml title="conf/config.yaml"
apisix:
  proxy_mode: http&stream   # Enable both L4 & L7 proxies
  stream_proxy:             # Configure L4 proxy
    tcp:
      - 9100                # Set TCP proxy listening port
    udp:
      - 9200                # Set UDP proxy listening port

stream_plugins:
  - prometheus              # Enable prometheus for stream proxy
```

Create a Stream Route with the `prometheus` Plugin:

```shell
curl "http://127.0.0.1:9180/apisix/admin/stream_routes" -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
Include the following configurations in `config.yaml` to enable stream proxy and enable `prometheus` Plugin for stream proxy. Reload APISIX for changes to take effect:
    "plugins": {
      "prometheus":{}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "httpbin.org:80": 1
      }
    }
  }'
```

Send a request to the Stream Route to verify:

```shell
curl -i "http://127.0.0.1:9100"
```

You should see an `HTTP/1.1 200 OK` response.

Send a request to the APISIX Prometheus metrics endpoint:

```shell
curl "http://127.0.0.1:9091/apisix/prometheus/metrics"
```

You should see an output similar to the following:

```text
# HELP apisix_stream_connection_total Total number of connections handled per Stream Route in APISIX
# TYPE apisix_stream_connection_total counter
apisix_stream_connection_total{route="1"} 1
```
