Commit Graph

578 Commits

Author SHA1 Message Date
Orne Brocaar
411cd681a9 Add cmd to migrate ds from Redis to Pg. 2024-02-22 12:44:14 +00:00
Orne Brocaar
8a986d04ce Fix tests after device-session refactor. 2024-02-21 16:57:03 +00:00
Orne Brocaar
da6b7e1b37 Use Device.device_session field for DS (WIP).
This is work-in-progress, still need to refactor all the tests.
2024-02-21 13:30:08 +00:00
Orne Brocaar
fae182aa3d Migrate code to store device-session in PG (WIP). 2024-02-19 15:27:25 +00:00
Orne Brocaar
5c3624cfbe Work-in-progress test.
This is work-in-progress and only contains a partial implementation.
Downlink (other than OTAA) is not yet implemented. Therefore you should
disable ADR in the region_.toml config when testing.
2024-02-07 15:06:28 +00:00
Orne Brocaar
d599e7a276 Add js context to get_activation api method. 2024-02-06 13:16:26 +00:00
Orne Brocaar
8cde64c4b7 Save metrics in one pipelined Redis query.
This saves the metrics in one pipelined Redis query, rather than one
query per aggregation level. This saves 2 queries per save.
2024-01-31 12:11:54 +00:00
Orne Brocaar
b65faf7b98 Merge deduplication_put and _locked queries.
This merges the deduplication_put and deduplication_locked Redis queries
into a single pipelined query. Simulating 400 uplinks / seconds through
3 gateways (thus 1200 messages / second), and an artificial TCP latency
of 5ms between ChirpStack <> Redis, this reduces the
storage_redis_conn_get_duration_seconds_sum metric by about 50%.

What we are trying to solve here is that under high (simulated) load, we
exhaust the Redis connection pool. This situation gets worse when there
is a significant latency between ChirpStack <> Redis, because the query
takes longer to complete and is reserved from the pool for a longer
time. The result can be that during the de-duplication process, the key
containing the uplink set has already expired before the
deduplicate_collect function is able to get a Redis connection from the
pool.

This query merge is a quick win, because each uplink can be received by
N gateways, thus this merge saves N Redis queries per uplink.
2024-01-31 11:06:03 +00:00
Orne Brocaar
9de86ffdec Track pg and redis get conn metrics.
This data is useful when debugging connection-pool size related issues.
E.g. when more connections are requested than the connection pool allows
then this will result in a longer wait time.
2024-01-31 09:54:00 +00:00
Orne Brocaar
107fbfa7bb Fix setting dev_addr field on ABP activate. 2024-01-25 11:23:33 +00:00
dependabot[bot]
4ce886b21d
Bump unsafe-libyaml from 0.2.9 to 0.2.10 (#341)
Bumps [unsafe-libyaml](https://github.com/dtolnay/unsafe-libyaml) from 0.2.9 to 0.2.10.
- [Release notes](https://github.com/dtolnay/unsafe-libyaml/releases)
- [Commits](https://github.com/dtolnay/unsafe-libyaml/compare/0.2.9...0.2.10)

---
updated-dependencies:
- dependency-name: unsafe-libyaml
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-12 15:55:03 +00:00
dependabot[bot]
a1c3431719
Bump follow-redirects from 1.15.3 to 1.15.4 in /ui (#353)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.15.3...v1.15.4)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-01-12 15:54:33 +00:00
Orne Brocaar
be26e60111 Bump version to 4.7.0-test.3 2024-01-12 09:19:50 +00:00
Orne Brocaar
2e104dc6ef Fix OIDC and oAuth2 logout redirect.
This regression was introduced when implementing the oAuth2 auth
backend. As we are redirecting to an external URL, we should not use
`navigate`.
2024-01-12 09:03:05 +00:00
Orne Brocaar
e1334f647d Bump version to 4.7.0-test.2 2024-01-11 15:42:16 +00:00
Orne Brocaar
769d590fb2 Remove cargo-bitbake dev-dependency.
We no longer need this and it breaks the build as it requires OpenSSL.
2024-01-11 15:26:13 +00:00
Orne Brocaar
cc47958839 Bump version to 4.7.0-test.1 2024-01-11 12:48:18 +00:00
Liu Woon Yung
3b7ab175b4
Fix device activation not setting the device class for LoRaWAN 1.0 on ABP. (#344) 2024-01-11 12:39:15 +00:00
Liu Woon Yung
f248379259
Fix Chirpstack overwriting user updates by OIDC user provisioning callback (#343)
Co-authored-by: Orne Brocaar <info@brocaar.com>
2024-01-11 12:28:49 +00:00
Orne Brocaar
e379e8738c ui: Fix black bg in jsontree after update. 2023-12-20 13:24:33 +00:00
Orne Brocaar
9f67b9c221 Implement direct redirect for OIDC and OAuth login. 2023-12-19 14:31:21 +00:00
Orne Brocaar
9198d7d51b Implement Clerk as authentication backend (OAuth2). 2023-12-19 13:49:16 +00:00
Orne Brocaar
f45d12519c Update deadpool-redis dependency.
This deprecates the deadpool-redis-cluster crate in favor of the
deadpool-redis crate with cluster feature.
2023-12-18 10:04:05 +00:00
Orne Brocaar
fcd0ef2ca5 Update rcgen dependency. 2023-12-18 09:56:23 +00:00
Orne Brocaar
c62f2b6474 Immediately return PG conn after use.
Same as the previous commit, this will returns the PG connection back to
the pool after usage, avoiding the risk that we are holding the conn
longer than needed.
2023-12-12 11:22:15 +00:00
Orne Brocaar
fccf762c39 Immediately return Redis conn after use.
Before the Redis connection would be returned once variable c went out
of scope, in most cases at the return of the function. This would mean
that during the execution of the remaining code within the function,
the Redis connection would be reserved.

With this change, the Redis connection is immediately returned to the
pool after usage.
2023-12-12 11:02:29 +00:00
Orne Brocaar
f57aa32f23 ui: Update dependencies. 2023-12-11 16:14:13 +00:00
Orne Brocaar
fb29d7f58d Update rdkafka dependency. 2023-12-11 11:37:12 +00:00
Orne Brocaar
f0987af114 Update prometheus-client. 2023-12-11 11:33:19 +00:00
Orne Brocaar
691cee7beb Upgrade jsonwebtoken dependency.
Since v9 setting the audience is mandatory.
2023-12-11 11:23:52 +00:00
Orne Brocaar
0ff05a9285 Update dependencies (cargo update). 2023-12-11 11:10:06 +00:00
Orne Brocaar
2774c51ea2 Refactor client-cert generator to use rustls/rcgen.
This refactors the last bit of code that depends on the OpenSSL C
library. Note that the openssl-probe does not depend on OpenSSL, it only
tries to detect the CA certificate directory on the host system.
2023-12-11 10:55:36 +00:00
Orne Brocaar
fe98673bbb Fix uncommited configuration change. 2023-12-11 10:55:36 +00:00
Orne Brocaar
a0d4e80720 Fix remaining blocking code after async Redis refactor.
This code still used channel.blocking_send(..) and sleep from std::time.
As async anonymous functions are not (yet) supported, this moves the
stream handling into a separate async function.
2023-12-11 10:55:36 +00:00
Orne Brocaar
53a570b0b0 Update httpmock (dev dep) to 0.7.0-rc.1.
This version removes the isahc depencency, which causes openssl to be
added as a dependency. Version 0.7 makes this dependency optional.
2023-12-11 10:55:36 +00:00
Orne Brocaar
5108f4451c Implement MQTT shared-subscription for gw backend. 2023-12-11 10:55:36 +00:00
Orne Brocaar
5b6d037469 Refactor gateway MQTT backend to use rumqttc. 2023-12-11 10:55:36 +00:00
Orne Brocaar
ed06de231e Refactor getting PostgreSQL ca certs. 2023-12-11 10:55:36 +00:00
Orne Brocaar
17f0d8c495 Refactor MQTT integration to use rumqttc. 2023-12-11 10:55:36 +00:00
Orne Brocaar
345d0d8462 Refactor code to use async redis. 2023-12-11 10:55:36 +00:00
Orne Brocaar
8e2eda3d5b Refactor code to use diesel-async.
This still depends on unreleased diesel and diesel-async code. As soon
as new diesel and diesel-async code has been released, we can remove
the [patch.crates-io] from Cargo.toml.
2023-12-11 10:55:35 +00:00
Orne Brocaar
3f57609981 Update code examples. 2023-12-07 09:43:57 +00:00
Orne Brocaar
8e3f2aa3d2 Bump chirpstack_integration version. 2023-12-07 09:22:19 +00:00
Orne Brocaar
f647949c05 Bump version to 4.6.0 2023-12-07 09:13:36 +00:00
Orne Brocaar
20e28407d0 Bump version to 4.6.0-test.4 2023-11-28 13:01:40 +00:00
Orne Brocaar
1a153b01f4 Change v4_migrate default to false.
This flag helps to migrate from v3 to v4 and adds a compatibility layer
to support ChirpStack Gateway Bridge v3.14+. The plan was to default
this to false in an earlier release, but this change never happened.
2023-11-27 09:09:37 +00:00
Orne Brocaar
1b0deedb64 ui: Update dependencies. 2023-11-23 15:54:00 +00:00
Orne Brocaar
20ab51052d Update dependencies. 2023-11-23 15:27:20 +00:00
Orne Brocaar
8797df2e05 Return err if ThingsBoardAccessToken is missing.
Before this would default to an empty string, which would return in an
invalid URL.

Fixes #277.
2023-11-23 15:14:40 +00:00
Orne Brocaar
687d0b1c62 Add assume_email_verified option for OIDC.
Fixes #302.
2023-11-23 15:05:27 +00:00