Commit Graph

298 Commits

Author SHA1 Message Date
Felipe Lalanne
1d4614c969 Remove restrictions on docker env vars
Docker doesn't impose any restrictions on the contents of an env
var, and neither does the CLI and API so it doesn't make much sense
for the env var to be as restrictive.

Change-type: patch
2022-03-31 16:45:11 -03:00
Felipe Lalanne
8e40f1c2f5 Ignore unknown image classes on the target state
Starting with v3 state endpoint, the supervisor can receive
service configuration for services that are meant to be installed as
overlays or filesets on the host, as well as configuration for services
that are meant to be installed on the root partition. This commit just
ignores those services from the target state until support is added
2022-03-22 19:28:43 -03:00
Felipe Lalanne
8bf8792583 Only uninstall 'fleet' apps when localMode is set
Local mode is still a device level config. Eventually it will become a
property of an app, but for now, we don't want the supervisor trying to
uninstall supervisor or host app when local mode is set
2022-03-22 19:28:43 -03:00
Felipe Lalanne
f1cd3d367c Cleanup unused methods and dependencies on db ids 2022-03-22 19:28:43 -03:00
Felipe Lalanne
381abeadb9 Refactor current state report to patch v3 state
This change makes the `api-binder/report` module more agnostic
to internal device state implementation details, moving necessary
healthchecks and data filtering to getCurrentForReport in device-state.

This also adds generic functions to perform comparison between current
state reports.
2022-03-22 19:28:36 -03:00
Felipe Lalanne
25e9ab4786 Refactor api-binder as a directory
The role of the api-binder module is to be the intermediary
between the cloud API and the device-state. For this reason it makes sense to
isolate target state retrieval and current state reporting into this
module. This change just moves current state reporting to the directory.
2022-03-22 19:08:03 -03:00
Felipe Lalanne
97f3b2a51e Update types and create methods for reporting v3 state 2022-03-22 19:08:03 -03:00
Felipe Lalanne
5c5483dd3d Rename networks to <appUuid>_<networkName>
This is required as we are phasing out app ids and we need to be able to
get app uuid from the current state of the network. The app-id now
exists as a container in new networks

This commit will restart containers as it needs to recreate the network.
2022-03-22 19:08:03 -03:00
Felipe Lalanne
0835b29874 Add app uuid as metadata to new volumes
We cannot modify older volumes but newly created volumes will contain
app uuid as metadata so they can be migrated at some point in the
future.
2022-03-22 19:08:03 -03:00
Felipe Lalanne
063bd400a4 Convert target state in local endpoints
Convert target state from to v3 in `/v2/local/target-state`. Add tests
for target state conversion
2022-03-22 19:08:03 -03:00
Felipe Lalanne
7425d1110b Add support for GET v3 target state
This change updates types and database format in order to allow
receiving the new format of the target state from the cloud and allow
applications to keep working.

This change also updates metadata in the containers, meaning services
will need to be restarted on supervisor update

Change-type: major
2022-03-22 19:08:02 -03:00
Felipe Lalanne
ccae1f7cb8 Rename aplication manager getStatus as getLegacyState
With the move to v3 target state and the move forward to remove
database ids from the supervisor, we want to ensure the ids are only
used for legacy support (such as within the API). This change renames
the method and sets it as deprecated
2022-03-22 19:08:02 -03:00
20k-ultra
b068c209b0 Moved test setup into file included for all tests
Change-type: patch
Signed-off-by: 20k-ultra <3946250+20k-ultra@users.noreply.github.com>
2022-03-18 18:19:21 -04:00
20k-ultra
2fdb83839c Move report throttle out of reporting logic
Change-type: patch
Signed-off-by: 20k-ultra <3946250+20k-ultra@users.noreply.github.com>
2022-03-15 22:53:34 -04:00
Felipe Lalanne
1b54ce8bfd Ignore selinux security opts when comparing services
The moby engine v20.x.y adds some selinux [security configurations](https://docs.docker.com/engine/reference/run/#security-configuration)
depending on the [container configuration](https://github.com/moby/moby/blob/master/daemon/create.go#L214).
This would cause the supervisor to enter a service restart loop as the
current and target service configurations will never match. The
supervisor now ignores selinux specific security options since those are
not supported by balenaOS.

Closes: #1890
Change-type: patch
2022-02-23 18:12:27 -03:00
Felipe Lalanne
e7ec42fadc Use a breadcrumb to mark that a reboot is required
As changes to config.json may restart the supervisor before it can
trigger the reboot (or something can kill the supervisor before it can run that step),
the supervisor needs a persistent signal that a reboot is required
(instead of the current transient signal).

With this commit, the supervisor will now create a breadcrumb in the
host `/tmp` folder, that will be checked as the last step of the
configuration changes.
2022-02-15 12:52:48 -03:00
Felipe Lalanne
a2d6db1e1d Update signature of fsUtils.getPathOnHost
The function now returns either a string array if it receives multiple
arguments or a single string if it receives a single argument.
2022-02-15 12:52:46 -03:00
Felipe Lalanne
2917f03452 Perform config.json sequentially to other config changes
As config.json changes may restart the engine (and hence the supervisor)
in newer OS versions, this ensures that the supervisor does not get
interrupted while writing to backends.
2022-02-15 12:49:03 -03:00
Felipe Lalanne
118875e12e Fix apiUpdatePollInterval default to line up with API 2022-02-15 12:49:03 -03:00
Felipe Lalanne
a4d91d381a Create touch and getBootTime utility functions
Change-type: patch
2022-02-15 12:49:03 -03:00
Christina Wang
5f1a77da25 Add update lock check to PATCH /v1/device/host-config
This is necessary with the changes as of balenaOS 2.82.6, which watches config.json
and will restart balena-hostname and some other services automatically on file change.

Change-type: patch
Relates-to: #1876
Signed-off-by: Christina Wang <christina@balena.io>
2022-02-14 22:22:00 +00:00
Felipe Lalanne
72f6cbe4c7 Add support for local ipv6 reporting
With more and more devices in ipv6 only networks, this ensures the
local addresses are reported to the cloud as part of the state patch.

Change-type: patch
2022-02-08 19:06:13 -03:00
Felipe Lalanne
f471ad736c Throw if target states gets a 304 without an ETAG
The API uses 304 as a mechanism for load management on target state
requests. This may cause that the supervisor receives a 304 response
without having received a copy of the target state first, leading to
issues. This change checks for an etag when receiving a 304, throwing an
exception otherwise.

Change-type: patch
2022-01-26 11:27:15 -03:00
Felipe Lalanne
d06b8e053e Use dmidecode to read cpuid in non ARM devices
Cpu id is set to null so far for non ARM devices (e.g. Intel NUC). This
parses the output of dmidecode to get the cpu id and system model.

Change-type: patch
2022-01-13 22:49:42 +00:00
Felipe Lalanne
c7fc7aacf8 Use dmidecode to read cpuid in non ARM devices
Cpu id is set to null so far for non ARM devices (e.g. Intel NUC). This
parses the output of dmidecode to get the cpu id and system model.

Change-type: patch
2022-01-06 21:01:53 +00:00
Pagan Gazzard
157fd95196 Increase delta request timeout to 59s to better align with our backends
Change-type: patch
2022-01-18 10:02:13 +00:00
Felipe Lalanne
9c6e5ee11f Remove apps.json after initial preload
This avoids the supervisor trying to get back to the preloaded target
state if the database is deleted by any reason. It does this by moving the
used apps.json to a backup location.

Change-type: patch
Depends-on: #1841
2021-12-13 20:11:42 +00:00
Felipe Lalanne
f6692ab918 Convert target state types to io-ts for better validation
This simplifies target state validation and improves validation
messages.

Change-type: patch
2021-12-02 15:29:37 -03:00
Alexandru Costache
1d1b1aa1bf test: Update extra_uEnv test slugs list
Do so to include the Nano 2GB Devkit device

Signed-off-by: Alexandru Costache <alexandru@balena.io>
2021-11-17 13:48:19 +01:00
Felipe Lalanne
394377e0a1 Fix delete-then-download strategy
The strategy has been broken for a while but it was not clear how to
fix it before the changes to image management. This PR fixes application
manager to remove images before downloading the new image. This will
only have an effect on changing images.

Closes: #1233
Change-type: patch
2021-11-16 16:40:15 -03:00
Felipe Lalanne
7aedc97ee1 Wait for images to be ready before moving between releases
For download-then-kill strategy, this waits for all changing images on the target
release to be available on device before killing the old services. This
will prevent that multicontainer applications get to a state where some
services of the new release start runnning much before others have been
downloaded.

When adding new services to a multicontainer app, the supervisor will
now wait for other changing services to be downloaded before starting
the new service.

Closes: #1812
Change-type: patch
2021-11-11 14:08:36 -03:00
Felipe Lalanne
969f4225e5 Check config for networks and volumes inside Service
This removes the need for the app module to know about the naming
conventions for networks and volumes since those exist now within the
service itself. This also fixes a small bug where the volume would be
removed before the service itself had been successfully stopped.

Change-type: patch
2021-10-28 10:20:53 -03:00
Alexandru Costache
7693f490b4 test: Update extra_uEnv test slugs list
We do this since we added for TX2 NX and
derived device types.

Signed-off-by: Alexandru Costache <alexandru@balena.io>
2021-08-24 07:25:20 +00:00
Felipe Lalanne
298f7f523e Simplify splash test to verify just correct outcomes
Splash tests are a bit flaky. Simplify to reduce chance of false
positives
2021-09-28 14:53:26 -04:00
Felipe Lalanne
aab000209b Add backoff to state reporting when 503 is received
Current state reporting had a backoff when network or inconsistency
errors were found, but not on API errors. This change adds a backoff
using RetryAfter header if present to reduce load on API

Change-type: patch
2021-09-28 14:53:26 -04:00
Alex Gonzalez
1abd10a129 os-release: Use developmentMode to ascertain OS variant in new releases
Newer BalenaOS releases have replaced OS variants for a developmentMode
configuration setting. This commit uses this variable to set the OS
variant in the absence of `VARIANT_ID` from the os-release file.

Change-type: patch
Signed-off-by: Alex Gonzalez <alexg@balena.io>
2021-08-05 09:30:35 +00:00
Kyle Harding
669866b4c2
Skip restarting services if they are part of conf targets
Some recent changes to the OS allowed some services to restart
automatically when the associated config files are changed.

In these cases we want to avoid restarting the same services
manually from the supervisor.

Change-type: patch
Signed-off-by: Kyle Harding <kyle@balena.io>
2021-08-24 14:03:55 -04:00
Felipe Lalanne
104a8006fb Update apiSecret table to id services by name
It adds a migration replacing the serviceId column by serviceName and
populates serviceNames from services in the target state.
2021-07-28 09:57:38 -04:00
Felipe Lalanne
50aab3ba78 Update tests removing dependency on db ids 2021-07-28 09:57:38 -04:00
Felipe Lalanne
f1bd4b8d9b Use tags to track supervised images in docker
The image manager module now uses tags instead of docker IDs as the main
way to identify docker images on the engine. That is, if the target
state image has a name `imageName:tag@digest`, the supervisor will always use
the given `imageName` and `tag` (which may be empty) to tag the image on
the engine after fetching. This PR also adds checkups to ensure
consistency is maintained between the database and the engine.

Using tags allows to simplify query and removal operations, since now
removing the image now means removing tags matching the image name.

Before this change the supervisor relied only on information in the
supervisor database, and used that to remove images by docker ID. However, the docker
id is not a reliable identifier, since images retain the same id between
releases or between services in the same release.

List of squashed commits
- Remove custom type NormalizedImageInfo
- Remove dependency on docker-toolbelt
- Use tags to traack supervised images in docker
- Ensure tag removal occurs in sequence
- Only save database image after download confirmed

Relates-to: #1616 #1579
Change-type: patch
2021-07-26 09:52:25 -04:00
Felipe Lalanne
357d1baf61 Fix db-helper module for tests
The previous module was using `rewire` to get the knex instance from the
db module but that was leading to issues when running tests using `test:fast`.
This provides a fix for the test module that just removes the destroy
call entirely (it turns out it is not necessary).

Change-type: patch
2021-07-08 14:43:13 -04:00
Christina Wang
17e740a4ba
Allow users to override HUP lock if device is stuck in invalid state
This functionality is needed when breadcrumbs aren't deleted after a HUP
rollback for whatever reason. Also rename HUP lock function.

Change-type: patch
Connects-to: #1459
Signed-off-by: Christina Wang <christina@balena.io>
2021-07-08 12:43:32 +09:00
Felipe Lalanne
e04e64763f Improve testing for supervisor composition modules
This PR cleans up testing for supervisor compose modules. It also fixes broken
tests for application manager and removes a lot of dependencies for those tests
on DB and other unnecessary mocks. There are probably a lot of cases that tests
are missing but this should make writing new tests a lot easier.

This PR also creates a new mock dockerode (mockerode) module that should make it
easier to test operations that interact with the engine. All references
to the old mock-dockerode have not yet been removed but that should come
soon in another PR

List of squashed commits:
- Add tests for network create/remove
- Move compose service tests to test/src/compose and reorganize test descriptions
- Add support for image creation to mockerode
- Add additional tests for compose volumes
- Update mockerode so unimplemented fake methods throw. This is to ensure
  tests using mockerode fail if an unimplemented method is used
- Update tests for volume-manager with mockerode
- Update tests for compose/images
- Simplify tests using mockerode
- Clean up compose/app tests
- Create application manager tests

Change-type: minor
2021-07-05 17:50:52 -04:00
Christina Wang
a9028e58ec
Prevent updates/reboots with locks when HUP breadcrumbs present
On HUP, some healthceck services need to complete before
it's safe for the Supervisor to reboot the device when
applying state changes. rollback-{health|altboot}-breadcrumb
are the two files that Supervisor looks for and locks the device
on when present in this patch.

Not closing issue 1459 because there is a possible case where,
on altboot rollback, the breadcrumbs are not present. 1459
may be closed when this edge case is investigated.

Change-type: patch
Connects-to: #1459
See: https://www.flowdock.com/app/rulemotion/r-supervisor/threads/cL7YfNOLSfTPfw05h59GEW0kfOt
Signed-off-by: Christina Wang <christina@balena.io>
2021-06-30 13:27:03 +09:00
Pagan Gazzard
ee4d919fca Improve target state typings
Change-type: patch
2021-06-08 13:45:44 +01:00
Miguel Casqueira
ab4fb454e0 Refactor debug log when unmanaged volume is found
Change-type: patch
Signed-off-by: Miguel Casqueira <miguel@balena.io>
2021-06-02 13:07:24 -04:00
Miguel Casqueira
55a344dceb Prevent a recursive loop when reporting current state
Closes: #1673
Change-type: patch
Signed-off-by: Miguel Casqueira <miguel@balena.io>
2021-05-28 16:20:27 -04:00
Christina Wang
5004cc5fd2
Add SUPERVISOR_HARDWARE_METRICS to config documentation
Signed-off-by: Christina Wang <christina@balena.io>
2021-05-13 14:47:15 +09:00
Christina Wang
dcd863eed8
Add toggleable SUPERVISOR_HARDWARE_METRICS config
On devices with bandwidth sensitivity, this config var
disables sending system information such as memory
usage or cpu temp as current state.

Closes: #1645
Change-type: minor
Signed-off-by: Christina Wang <christina@balena.io>
2021-05-13 13:59:07 +09:00
Christina Wang
ea3e50e96e
Create & unify src/device-state/current-state tests
Signed-off-by: Christina Wang <christina@balena.io>
2021-05-12 18:33:01 +09:00