This PR changes the way the supervisor reads and writes files from /mnt/boot. Reads will
now use the [fatrw utility](https://github.com/balena-os/fatrw/) as a way to minimize corruption of
files in the boot partition, and thus preventing possible bricking of the device.
Since this basically changes the way a lot of configurations are read, this work was being blocked because of
the way tests were being done. While there still remain a couple of legacy tests to be migrated, this PR disables
test:legacy tests when running npm run test, as the work on refactoring those tests is in progress (see #2048) and
fatrw integration is of higher priority.
Change-type: minor
getImagesForCleanup used to query the Engine for the Supervisor
image, which is unnecessary given that the Supervisor has access
to constants.supervisorImage. Thus, this Engine query is removed.
The method is simplified and made more clear, and
imageManager.isCleanupNeeded doesn't need to be stubbed in tests.
Change-type: patch
Signed-off-by: Christina Ying Wang <christina@balena.io>
This eliminates chances of host-Docker address collision for apps such
as the Supervisor where all services have host networking.
Closes: #2062
Change-type: patch
Signed-off-by: Christina Ying Wang <christina@balena.io>
This means that configuration backend tests no longer use stubs and
(mostly) avoid internal dependencies in the tests. Instead of stubs and
mock-fs, the tests use [testfs](https://github.com/balena-io-modules/mocha-pod#working-with-the-filesystem)
which allows working with a real filesystem and ensuring everything is
re-set between tests.
This is the last change needed in order to be able to merge #1971. Here is the list of changes
- [x] Migrate splash image backend tests
- [x] Migrate extlinux backend tests
- [x] Migrate config.txt backend tests
- [x] Migrate extra-uenv config tests
- [x] Migrate odmdata config tests
- [x] Migrate config utils tests
- [x] Migrate device-config tests
Change-type: patch
Also remove system interface check from ensureSupervisorNetwork.
Previously `ensure` was a Bluebird promise which wasn't awaited in
its composition step. This has been here for some time and may contribute
to issues with duplicate networks. The conversion to native Promises
allows `ensure` to be awaited, hopefully reducing instances of duplicate
networks.
Removing the system interface check for /sys/class/net/supervisor0
because it's superfluous given that the Engine creates the interface
with NetworkManager. It also makes testing a lot more difficult to set up
as /sys/class/net isn't a directory that can be written to for emulating
system interface creation / removal.
Relates-to: https://github.com/balena-os/balena-supervisor/issues/1110
Change-type: minor
Signed-off-by: Christina Ying Wang <christina@balena.io>
Previously it was set at /mnt/root/sys/class/net, which is
the same as /sys/class/net because Supervisor has a network
mode of `host`.
Signed-off-by: Christina Ying Wang <christina@balena.io>
Use isNotFoundError which converts an error of the default type
`unknown` into NotFoundError if the error is an instance of NotFoundError.
Thrown errors are of type `unknown` by default so we should use methods
with type guards for better type narrowing.
Signed-off-by: Christina Ying Wang <christina@balena.io>
This allows to test that the supervisor build actually runs and opens up the
possibility of running more exhaustive API tests against a working supervisor.
Change-type: patch
This excludes route tests or refactoring. Also, created tests
for API middleware.
Change-type: patch
Signed-off-by: Christina Ying Wang <christina@balena.io>
Merged docker-utils and delta tests into a single test suite. They are
now ran as part of the integration tests using the real engine.
Change-type: patch
When code that is unit tested is part of a file that imports modules which
depend on the dbus module, this breaks the unit test environment because there
is no system socket set up, as the unit test mocha config doesn't import fixtures.ts.
For example, if we change src/compose/utils to import device-config or api-binder, both
of those modules import lib/dbus which invokes a dbus.getBus call at the root level. This
is problematic for unit testing.
We can get around the root-level dbus.getBus call by initializing dbus only when it's first
needed. The mocked-dbus test setup code can also be removed in favor of legacy mocha
hooks, which makes the dbus stubbing in the legacy test environment more clear.
We can remove these legacy hooks when all the legacy tests are migrated to unit/integration.
Signed-off-by: Christina Ying Wang <christina@balena.io>
This commit also changes the test:integration command to run integration
tests as different processes. This allows to avoid any test leaking into
each-other because of the use of singletons. This however has the side
effect of tests being slower, but that is a forcing function to refactor
the code.
Update-lock tests now use the actual filesystem for testing, instead of
relying on stubs and spies.
This commit also fixes a small bug with update-lock that would cause a
`PromiseRejectionHandledWarning` when the lock callback would throw.
Now the tests are ran against the actual docker engine instead of
against mockerode.
The new tests actually caught a bug in
`volumeManager.removeOrphanedVolumes`, where that function would try to
remove volumes for stopped containers, causing an exception.
This commit also fixes that bug.
This also needs to modify the test environment as database migrations
will look for `config.json` in the location given by the variable
`CONFIG_MOUNT_POINT`.
The volume tests now run against the actual docker engine setup via dind
Change-type: patch
The supervisor used to rely on specific event reporting for identifying
issues at runtime. As the platform has grown, it has become much more
difficult to get any signal from the event noise. Recently the API side
for these events has been disabled, meaning these events only
contribute to bandwidth consumption. This commit disables the
event reporting feature of the supervisor which will be most likely
replaced by something like Sentry in the near future.
Change-type: minor
`withDefault` is a type helper that allows to create a type that
defaults to a default value when trying to decode a nullish value.
That type was not correctly working with boolean types, causing `false`
values to be replaced by true. This would specifically cause issues when
parsing the target state, where a `running: false` in a service would
become a `running: true` due to the type decoding.
Change-type: patch
The supervisor uses the following pattern for async module
initialization
```typescript
// module.ts
export const initialised = (async () => {
// do some async initialization
})();
// somewhere else
import * as module from 'module';
async function setup() {
await module.initialise;
}
```
The above pattern means that whenever the module is imported, the
initialisation procedure will be ran, which is an anti-pattern.
This converts any instance of this pattern into a function
```typescript
export const initialised = _.once(async () => {
// do some async initialization
});
```
And anywhere else on the code it replaces the call with a
```typescript
await module.initialised();
```
Change-type: patch
This allows to run integration tests during development and on CI
with the right dependencies. There are several changes that this
involves, but the gist of it is that a test environment is setup using
`docker-compose.test.yml`. This file is loaded by `resin-ci` during the
build, and ensures that integration tests are ran after setting up all
requirements. This commit also defines a test environment command that
can be setup using `npm run test:env` in order to run tests in a local
development machine.
This sets up the new `test/unit` and `test/integration` folders
and starts classification of some of the test files.
Note that unit tests include, `fs-utils` and `system-info` tests.
While these tests interact with the filesystem, the implementation
of these modules is simple enough, and the tests are fast enough to
allow these tests to fall under the `unit` test category (according to
test/README)
Change-type: patch
We are refactoring the supervisor test suite into unit tests (for
algorithms an domain model tests) and integration
tests (for interaction with out-of-process dependencies).
This means the current test suite needs to be classified into
these two categories, and fixed whenever possible.
This commit moves the test suite under the `test/legacy` folder, this
folder should be progressively migrated and eventually removed.
Subsequent commits will begin to split these files into unit and
integration whenever possible.
Depends-on: #1996
Change-type: patch
This replaces all relative paths in the test suite (e.g
`../src/compose/service.ts`) with the aliased path configured through
tsconfig.
This is a big change but it doesn't affect any functionality
Currently, tests only can import source code modules through relative
paths `../../`. This makes it very difficult to refactor and organize
tests in folders as the paths change.
[tsconfig-paths](https://www.npmjs.com/package/tsconfig-paths) allows to
reference the source through an alias defined in the "paths" section of
tsconfig.json
The supervisor used to perform tests both for the transpiled code (after
tsc) and one for the typescript code (using
ts-node/register/transpile-only). There is not really a reason for this
and this added complexity to the test configuration. This used to make
testing harder, as the built code didn't include source maps, meaning
the tests did not point to the right code.
Since we want to split tests in unit and integration tests as the next
test improvement, it makes sense to simplify these commands before
adding more complexity.
Change-type: patch
This mitigates an edge case bug introduced in v13.1.3 where services that
are slow to exit may get stuck in a state of Downloaded if a service var is
changed then reverted rapidly. More detailed description in linked issue.
Change-type: patch
Closes: #1991
Signed-off-by: Christina Wang <christina@balena.io>
Some libraries, like [proper-lockfile](https://www.npmjs.com/package/proper-lockfile)
use directories instead of files for locking. This PR allows the supervisor to be able to
work with those types of locks when lock override is requested.
Closes: #1978
Change-type: patch
We don't need to read the host's hostname through /mnt/root/etc/hostname,
because the hostname is written to config.json on a change. When the hostname
has never changed, it won't be found in config.json, so we can default to
the Supervisor container's /etc/hostname as it will match the host's
/etc/hostname, the network mode being `host`.
Closes: #1968
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
balena-compose already supports this, and with this PR, Supervisor can
have the option of using HostConfig.Mounts for internal bind mounts such as
ones added by feature labels. This will be handled in a future PR.
The only blocker to having users use long syntax is adding this feature
to target state. This PR does not add that feature.
Relates-to: https://github.com/balena-os/balena-supervisor/pull/1780
Relates-to: https://github.com/balena-os/balena-engine/issues/220
Relates-to: #1933
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
This commit updates all backends that write to /mnt/boot to do it
through a new `lib/host-utils` module. Writes are now done using write +
sync as rename is not an atomic operation in vfat.
The change also applies for writes through the `/v1/host-config`
endpoint.
Finally this change includes some improvements on tests.
Change-type: patch
This will ensure the restart policy specified is not violated
Change-type: patch
Closes: #1668
Signed-off-by: 20k-ultra <3946250+20k-ultra@users.noreply.github.com>
When disposing of resources which include Supervisor-created lockfiles,
only dispose of lockfiles for the specified user application.
Signed-off-by: Christina Wang <christina@balena.io>
The linked issue describes the Supervisor not cleaning up locks it creates due
to crashing at just the wrong time. After internal discussion we decided to
differentiate Supervisor-created lockfiles from user-created lockfiles by using
the `nobody` UID (65534) for Supervisor-created lockfiles.
As the existing NPM lockfile lib does not allow creating lockfiles atomically
with different UIDs, we move to using the lockfile binary, which is part of the
procmail package. To allow nonroot users to write to lock directories, permissions
are changed to allow write access by nonroot users.
See: https://www.flowdock.com/app/rulemotion/r-resinos/threads/gWMgK5hmR26TzWGHux62NpgJtVl
Change-type: minor
Closes: #1758
Signed-off-by: Christina Wang <christina@balena.io>
Starting with v3 state endpoint, the supervisor can receive
service configuration for services that are meant to be installed as
overlays or filesets on the host, as well as configuration for services
that are meant to be installed on the root partition. This commit just
ignores those services from the target state until support is added
Local mode is still a device level config. Eventually it will become a
property of an app, but for now, we don't want the supervisor trying to
uninstall supervisor or host app when local mode is set
This change makes the `api-binder/report` module more agnostic
to internal device state implementation details, moving necessary
healthchecks and data filtering to getCurrentForReport in device-state.
This also adds generic functions to perform comparison between current
state reports.
The role of the api-binder module is to be the intermediary
between the cloud API and the device-state. For this reason it makes sense to
isolate target state retrieval and current state reporting into this
module. This change just moves current state reporting to the directory.
This is required as we are phasing out app ids and we need to be able to
get app uuid from the current state of the network. The app-id now
exists as a container in new networks
This commit will restart containers as it needs to recreate the network.
This change updates types and database format in order to allow
receiving the new format of the target state from the cloud and allow
applications to keep working.
This change also updates metadata in the containers, meaning services
will need to be restarted on supervisor update
Change-type: major
With the move to v3 target state and the move forward to remove
database ids from the supervisor, we want to ensure the ids are only
used for legacy support (such as within the API). This change renames
the method and sets it as deprecated
As changes to config.json may restart the supervisor before it can
trigger the reboot (or something can kill the supervisor before it can run that step),
the supervisor needs a persistent signal that a reboot is required
(instead of the current transient signal).
With this commit, the supervisor will now create a breadcrumb in the
host `/tmp` folder, that will be checked as the last step of the
configuration changes.
As config.json changes may restart the engine (and hence the supervisor)
in newer OS versions, this ensures that the supervisor does not get
interrupted while writing to backends.
This is necessary with the changes as of balenaOS 2.82.6, which watches config.json
and will restart balena-hostname and some other services automatically on file change.
Change-type: patch
Relates-to: #1876
Signed-off-by: Christina Wang <christina@balena.io>
With more and more devices in ipv6 only networks, this ensures the
local addresses are reported to the cloud as part of the state patch.
Change-type: patch
The API uses 304 as a mechanism for load management on target state
requests. This may cause that the supervisor receives a 304 response
without having received a copy of the target state first, leading to
issues. This change checks for an etag when receiving a 304, throwing an
exception otherwise.
Change-type: patch
Cpu id is set to null so far for non ARM devices (e.g. Intel NUC). This
parses the output of dmidecode to get the cpu id and system model.
Change-type: patch
Cpu id is set to null so far for non ARM devices (e.g. Intel NUC). This
parses the output of dmidecode to get the cpu id and system model.
Change-type: patch
This avoids the supervisor trying to get back to the preloaded target
state if the database is deleted by any reason. It does this by moving the
used apps.json to a backup location.
Change-type: patch
Depends-on: #1841
The strategy has been broken for a while but it was not clear how to
fix it before the changes to image management. This PR fixes application
manager to remove images before downloading the new image. This will
only have an effect on changing images.
Closes: #1233
Change-type: patch
For download-then-kill strategy, this waits for all changing images on the target
release to be available on device before killing the old services. This
will prevent that multicontainer applications get to a state where some
services of the new release start runnning much before others have been
downloaded.
When adding new services to a multicontainer app, the supervisor will
now wait for other changing services to be downloaded before starting
the new service.
Closes: #1812
Change-type: patch
This removes the need for the app module to know about the naming
conventions for networks and volumes since those exist now within the
service itself. This also fixes a small bug where the volume would be
removed before the service itself had been successfully stopped.
Change-type: patch
Current state reporting had a backoff when network or inconsistency
errors were found, but not on API errors. This change adds a backoff
using RetryAfter header if present to reduce load on API
Change-type: patch
Newer BalenaOS releases have replaced OS variants for a developmentMode
configuration setting. This commit uses this variable to set the OS
variant in the absence of `VARIANT_ID` from the os-release file.
Change-type: patch
Signed-off-by: Alex Gonzalez <alexg@balena.io>
Some recent changes to the OS allowed some services to restart
automatically when the associated config files are changed.
In these cases we want to avoid restarting the same services
manually from the supervisor.
Change-type: patch
Signed-off-by: Kyle Harding <kyle@balena.io>
The image manager module now uses tags instead of docker IDs as the main
way to identify docker images on the engine. That is, if the target
state image has a name `imageName:tag@digest`, the supervisor will always use
the given `imageName` and `tag` (which may be empty) to tag the image on
the engine after fetching. This PR also adds checkups to ensure
consistency is maintained between the database and the engine.
Using tags allows to simplify query and removal operations, since now
removing the image now means removing tags matching the image name.
Before this change the supervisor relied only on information in the
supervisor database, and used that to remove images by docker ID. However, the docker
id is not a reliable identifier, since images retain the same id between
releases or between services in the same release.
List of squashed commits
- Remove custom type NormalizedImageInfo
- Remove dependency on docker-toolbelt
- Use tags to traack supervised images in docker
- Ensure tag removal occurs in sequence
- Only save database image after download confirmed
Relates-to: #1616#1579
Change-type: patch
The previous module was using `rewire` to get the knex instance from the
db module but that was leading to issues when running tests using `test:fast`.
This provides a fix for the test module that just removes the destroy
call entirely (it turns out it is not necessary).
Change-type: patch
This functionality is needed when breadcrumbs aren't deleted after a HUP
rollback for whatever reason. Also rename HUP lock function.
Change-type: patch
Connects-to: #1459
Signed-off-by: Christina Wang <christina@balena.io>
This PR cleans up testing for supervisor compose modules. It also fixes broken
tests for application manager and removes a lot of dependencies for those tests
on DB and other unnecessary mocks. There are probably a lot of cases that tests
are missing but this should make writing new tests a lot easier.
This PR also creates a new mock dockerode (mockerode) module that should make it
easier to test operations that interact with the engine. All references
to the old mock-dockerode have not yet been removed but that should come
soon in another PR
List of squashed commits:
- Add tests for network create/remove
- Move compose service tests to test/src/compose and reorganize test descriptions
- Add support for image creation to mockerode
- Add additional tests for compose volumes
- Update mockerode so unimplemented fake methods throw. This is to ensure
tests using mockerode fail if an unimplemented method is used
- Update tests for volume-manager with mockerode
- Update tests for compose/images
- Simplify tests using mockerode
- Clean up compose/app tests
- Create application manager tests
Change-type: minor
On HUP, some healthceck services need to complete before
it's safe for the Supervisor to reboot the device when
applying state changes. rollback-{health|altboot}-breadcrumb
are the two files that Supervisor looks for and locks the device
on when present in this patch.
Not closing issue 1459 because there is a possible case where,
on altboot rollback, the breadcrumbs are not present. 1459
may be closed when this edge case is investigated.
Change-type: patch
Connects-to: #1459
See: https://www.flowdock.com/app/rulemotion/r-supervisor/threads/cL7YfNOLSfTPfw05h59GEW0kfOt
Signed-off-by: Christina Wang <christina@balena.io>
On devices with bandwidth sensitivity, this config var
disables sending system information such as memory
usage or cpu temp as current state.
Closes: #1645
Change-type: minor
Signed-off-by: Christina Wang <christina@balena.io>
A previous PR (#1656) fixed validation for network ipam config,
checking that both network and subnet are defined for each ipam config entry
(as described in the docker documentation).
After that PR, the validations throws an exception if the network target state is incorrect,
but this turns out to be the wrong approach, because that exception is also triggered
when querying target state.
This isn't a problem in normal operation, but it is in local mode, because local
mode queries the old target state before sending a new one. Since the query fails,
the CLI can never push the new target state.
This PR replaces the exception with a warning on the logs, since a
misconfigured network won't cause any engine failures, it will just
prevent containers to communicate through the provided network.
A future improvement should move this validation to an earlier point in the process,
so the target state can get rejected before it even gets to a point it
can be used.
Relates-to: #1693
Change-type: patch
This extra info will mean the API is able to immediately set default
config vars based on the os/supervisor version so that they are
available on the first target state fetch rather than having a delay
whilst waiting for the supervisor to report them as part of a state
patch
Update balena-register-device from 6.1.6 to 7.2.0
Change-type: patch
'mz' can be safely replaced with fs.promises
and util.promisify for faster native methods.
'mkdirp' after Node v8 uses native fs.mkdir, thus
is redundant. 'body-parser' is deprecated and
contained within express v4.x.
Closes: #1567
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
This commit updates dockerode types to the latest 2.x version, removing the need
for custom composer types for network.
This commit also modifies network tests to use the new types
Change-type: minor
Tests using the mock `testWithData` method were not restoring the dockerode
prototype to the default values, making some tests behave differently
when run individually that when run with the suite.
This commit fixes the `testWithData` method and some malfunctioning v1
API tests because of the change. It doesn't fix all the tests
Using safeStateClone within doPurge to applyIntermediateTarget after
successful volume purge has led to various type deficiencies being revealed
in common.js. Add several inline types in common.js to satisfy
the type checker (credit: Page <page@balena.io>). Delete common.d.ts
since it's not required and might mistakenly mask true I/O types of
functions in common.js.
Closes: #1611
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
