This migrates the device-sessions from Redis into PostgreSQL. This fixes
a performance issue in case the same DevAddr is reused many times
(e.g. devices rejoining very often or a NetID with small DevAddr space).
There were two issues:
The Redis key containing the DevAddr -> DevEUIs mapping could contain
DevEUIs that no longer used the DevAddr. This mapping would only expire
from the Redis database after none of the devices would use the DevAddr
for more than the configured device_session_ttl.
The other issue with the previous approach was that on for example a
Type 7 NetID, a single DevAddr could be re-used multiple times. As each
device-session could be stored on a different Redis Cluster instance,
there was no option to retrieve all device-sessions at once. Thus a high
re-usage of a single DevAddr would cause an increase in Redis queries.
Both issues are solved by moving the device-session into PostgreSQL
as the DevAddr is a column of the device record and thus filtering on
this DevAddr would always result in the devices using that DevAddr, as
well all device-sessions for a DevAddr can be retrieved by a single
query.
Note that to migrate the device-sessions, you must run:
chirpstack -c path/to/config migrate-device-sessions-to-postgres
A nice side-effect is that a PostgreSQL backup / restore will also
restore the device connectivity.
Closes#362 and #74.
This merges the deduplication_put and deduplication_locked Redis queries
into a single pipelined query. Simulating 400 uplinks / seconds through
3 gateways (thus 1200 messages / second), and an artificial TCP latency
of 5ms between ChirpStack <> Redis, this reduces the
storage_redis_conn_get_duration_seconds_sum metric by about 50%.
What we are trying to solve here is that under high (simulated) load, we
exhaust the Redis connection pool. This situation gets worse when there
is a significant latency between ChirpStack <> Redis, because the query
takes longer to complete and is reserved from the pool for a longer
time. The result can be that during the de-duplication process, the key
containing the uplink set has already expired before the
deduplicate_collect function is able to get a Redis connection from the
pool.
This query merge is a quick win, because each uplink can be received by
N gateways, thus this merge saves N Redis queries per uplink.
This data is useful when debugging connection-pool size related issues.
E.g. when more connections are requested than the connection pool allows
then this will result in a longer wait time.
Same as the previous commit, this will returns the PG connection back to
the pool after usage, avoiding the risk that we are holding the conn
longer than needed.
Before the Redis connection would be returned once variable c went out
of scope, in most cases at the return of the function. This would mean
that during the execution of the remaining code within the function,
the Redis connection would be reserved.
With this change, the Redis connection is immediately returned to the
pool after usage.
This refactors the last bit of code that depends on the OpenSSL C
library. Note that the openssl-probe does not depend on OpenSSL, it only
tries to detect the CA certificate directory on the host system.
This code still used channel.blocking_send(..) and sleep from std::time.
As async anonymous functions are not (yet) supported, this moves the
stream handling into a separate async function.
This still depends on unreleased diesel and diesel-async code. As soon
as new diesel and diesel-async code has been released, we can remove
the [patch.crates-io] from Cargo.toml.
This flag helps to migrate from v3 to v4 and adds a compatibility layer
to support ChirpStack Gateway Bridge v3.14+. The plan was to default
this to false in an earlier release, but this change never happened.
