The strategy has been broken for a while but it was not clear how to
fix it before the changes to image management. This PR fixes application
manager to remove images before downloading the new image. This will
only have an effect on changing images.
Closes: #1233
Change-type: patch
For download-then-kill strategy, this waits for all changing images on the target
release to be available on device before killing the old services. This
will prevent that multicontainer applications get to a state where some
services of the new release start runnning much before others have been
downloaded.
When adding new services to a multicontainer app, the supervisor will
now wait for other changing services to be downloaded before starting
the new service.
Closes: #1812
Change-type: patch
This removes the need for the app module to know about the naming
conventions for networks and volumes since those exist now within the
service itself. This also fixes a small bug where the volume would be
removed before the service itself had been successfully stopped.
Change-type: patch
Current state reporting had a backoff when network or inconsistency
errors were found, but not on API errors. This change adds a backoff
using RetryAfter header if present to reduce load on API
Change-type: patch
Newer BalenaOS releases have replaced OS variants for a developmentMode
configuration setting. This commit uses this variable to set the OS
variant in the absence of `VARIANT_ID` from the os-release file.
Change-type: patch
Signed-off-by: Alex Gonzalez <alexg@balena.io>
Some recent changes to the OS allowed some services to restart
automatically when the associated config files are changed.
In these cases we want to avoid restarting the same services
manually from the supervisor.
Change-type: patch
Signed-off-by: Kyle Harding <kyle@balena.io>
The image manager module now uses tags instead of docker IDs as the main
way to identify docker images on the engine. That is, if the target
state image has a name `imageName:tag@digest`, the supervisor will always use
the given `imageName` and `tag` (which may be empty) to tag the image on
the engine after fetching. This PR also adds checkups to ensure
consistency is maintained between the database and the engine.
Using tags allows to simplify query and removal operations, since now
removing the image now means removing tags matching the image name.
Before this change the supervisor relied only on information in the
supervisor database, and used that to remove images by docker ID. However, the docker
id is not a reliable identifier, since images retain the same id between
releases or between services in the same release.
List of squashed commits
- Remove custom type NormalizedImageInfo
- Remove dependency on docker-toolbelt
- Use tags to traack supervised images in docker
- Ensure tag removal occurs in sequence
- Only save database image after download confirmed
Relates-to: #1616#1579
Change-type: patch
The previous module was using `rewire` to get the knex instance from the
db module but that was leading to issues when running tests using `test:fast`.
This provides a fix for the test module that just removes the destroy
call entirely (it turns out it is not necessary).
Change-type: patch
This functionality is needed when breadcrumbs aren't deleted after a HUP
rollback for whatever reason. Also rename HUP lock function.
Change-type: patch
Connects-to: #1459
Signed-off-by: Christina Wang <christina@balena.io>
This PR cleans up testing for supervisor compose modules. It also fixes broken
tests for application manager and removes a lot of dependencies for those tests
on DB and other unnecessary mocks. There are probably a lot of cases that tests
are missing but this should make writing new tests a lot easier.
This PR also creates a new mock dockerode (mockerode) module that should make it
easier to test operations that interact with the engine. All references
to the old mock-dockerode have not yet been removed but that should come
soon in another PR
List of squashed commits:
- Add tests for network create/remove
- Move compose service tests to test/src/compose and reorganize test descriptions
- Add support for image creation to mockerode
- Add additional tests for compose volumes
- Update mockerode so unimplemented fake methods throw. This is to ensure
tests using mockerode fail if an unimplemented method is used
- Update tests for volume-manager with mockerode
- Update tests for compose/images
- Simplify tests using mockerode
- Clean up compose/app tests
- Create application manager tests
Change-type: minor
On HUP, some healthceck services need to complete before
it's safe for the Supervisor to reboot the device when
applying state changes. rollback-{health|altboot}-breadcrumb
are the two files that Supervisor looks for and locks the device
on when present in this patch.
Not closing issue 1459 because there is a possible case where,
on altboot rollback, the breadcrumbs are not present. 1459
may be closed when this edge case is investigated.
Change-type: patch
Connects-to: #1459
See: https://www.flowdock.com/app/rulemotion/r-supervisor/threads/cL7YfNOLSfTPfw05h59GEW0kfOt
Signed-off-by: Christina Wang <christina@balena.io>
On devices with bandwidth sensitivity, this config var
disables sending system information such as memory
usage or cpu temp as current state.
Closes: #1645
Change-type: minor
Signed-off-by: Christina Wang <christina@balena.io>
A previous PR (#1656) fixed validation for network ipam config,
checking that both network and subnet are defined for each ipam config entry
(as described in the docker documentation).
After that PR, the validations throws an exception if the network target state is incorrect,
but this turns out to be the wrong approach, because that exception is also triggered
when querying target state.
This isn't a problem in normal operation, but it is in local mode, because local
mode queries the old target state before sending a new one. Since the query fails,
the CLI can never push the new target state.
This PR replaces the exception with a warning on the logs, since a
misconfigured network won't cause any engine failures, it will just
prevent containers to communicate through the provided network.
A future improvement should move this validation to an earlier point in the process,
so the target state can get rejected before it even gets to a point it
can be used.
Relates-to: #1693
Change-type: patch
This extra info will mean the API is able to immediately set default
config vars based on the os/supervisor version so that they are
available on the first target state fetch rather than having a delay
whilst waiting for the supervisor to report them as part of a state
patch
Update balena-register-device from 6.1.6 to 7.2.0
Change-type: patch
'mz' can be safely replaced with fs.promises
and util.promisify for faster native methods.
'mkdirp' after Node v8 uses native fs.mkdir, thus
is redundant. 'body-parser' is deprecated and
contained within express v4.x.
Closes: #1567
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
This commit updates dockerode types to the latest 2.x version, removing the need
for custom composer types for network.
This commit also modifies network tests to use the new types
Change-type: minor
Tests using the mock `testWithData` method were not restoring the dockerode
prototype to the default values, making some tests behave differently
when run individually that when run with the suite.
This commit fixes the `testWithData` method and some malfunctioning v1
API tests because of the change. It doesn't fix all the tests
Using safeStateClone within doPurge to applyIntermediateTarget after
successful volume purge has led to various type deficiencies being revealed
in common.js. Add several inline types in common.js to satisfy
the type checker (credit: Page <page@balena.io>). Delete common.d.ts
since it's not required and might mistakenly mask true I/O types of
functions in common.js.
Closes: #1611
Change-type: patch
Signed-off-by: Christina Wang <christina@balena.io>
The device request object was created with untouched fields left unset. When
comparing state to determine if a transition is required this would
result in a mismatch between:
{
Driver: '',
Count: 1,
DeviceIDs: null,
Capabilities: [Array],
Options: null
}
and
{
Count: 1,
Capabilities: [Array],
}
Which in turn resulted in the target service being continously restarted.
The fix is to instantiate the object in full.
Connects-to: https://github.com/balena-io/balena-supervisor/issues/1449
Connects-to: ae646a07ec
Change-type: patch
Signed-off-by: Robert Günzler <robertg@balena.io>
Setting this this variable to a base64 encoded string will replace the splash
image on the device by rewriting `/mnt/boot/splash/balena-logo.png`.
This will also make a copy of the default balena logo so the splash can
be restored if the variable is removed.
Change-type: minor
Signed-off-by: Felipe Lalanne <felipe@balena.io>
The `ensureRequiredOverlay` function is currently ran for any backend,
at this moment this causes no issue, since most configuration backends
are defined per single device type. However, with the option to modify splash
images, which is available for all device types, the function would add
unwanted configuration vars to the splash image configuration. Moving it
to the config txt backend solves this issue.
This PR adds the following
* Supervisor v1 API application actions now return HTTP status code 423 when locks
are preventing the action to be performed. Previously this resulted in a
503 error
* Supervisor API v2 service actions now returns HTTP status code 423 when locks are
preventing the action to be performed. Previously, this resulted in an
exception logged by the supervisor and the API query timing out
* Supervisor API `/v2/applications/:appId/start-service` now does not
check for a lock. Lock handling in v2 actions is now performed by each
step executor
* `/v1/apps/:appId/start` now queries the target state and uses that
information to execute the start step (as v2 does). Previously start
resulted in `cannot get appId from undefined`
* Extra tests for API methods
Change-type: patch
Connects-to: #1523
Signed-off-by: Felipe Lalanne <felipe@balena.io>
During first time run of the supervisor, the target state is queried
by `reportInitialEnv`. Since this happens early on the initialization
process, this target state report is missed by any listeners and this
can lead to the initial target state not beeing applied (see #1455).
This PR ensures that target state is re-emitted if there were no
listeners setup on call to update.
Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1455
Some endpoints filter data based on the scope of the API key
used to make the request. When in LocalMode the check was not
being made correctly and all apps were considered out of scope.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
This paves the way for running multiple applications and storing
information related to the application against the application itself. A
couple of hacks have been added to v1 and v2 endpoints to maintain
compatability but these should eventually be removed with the addition
of a v3 api.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@balena.io>
Currently, when the label `io.balena.features.balena-socket` is set,
the balena engine socket is mounted under `/run/balena-engine.sock`.
This causes a problem when using systemd inside the container, since
this service remounts `/run` and `/run/lock` as tmpfs, causing the
socket to become unavailable.
Making a mount of the socket into `/host/run` solves this issue. This is
the same approach taken with DBUS.
Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1494
The source of truth for the device-type should be
device-type.json instead of config.json
Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1472
The memory information reported by the supervisor currently
estimates the value of used memory as `MemTotal - MemFree`.
However, linux systems will try to cache and buffer as much
memory as possible, which will affect the output of `MemFree`
(from /proc/meminfo) and in consequence the memory usage seen
by the user on the dashboard, which will appear much greater than
it is.
The correct calculation should be `MemTotal - MemFree - Buffers - Cached`,
which the calculation performed by `htop` and the `free` commands.
Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1471
In order to make supervisor upgrades more transparent, lets move away
from this env var since it requires a container restart any time the supervisor
is upgraded. We should ultimately move towards providing the supervisors
set of capabilities, but that can come later
Connects-to: #1447
Change-type: major
Signed-off-by: Matthew McGinn <matthew@balena.io>
When trying to apply SSDT overlays in Up Board, the supervisor currently
gets stuck in a loop trying to apply target state. See #1465
This was due to a bug in parsing the configuration, which lead to
the method bootConfigChangeRequired returning true when no change was
needed.
Change-type: patch
Signed-off-by: Felipe Lalanne <felipe@balena.io>
Connects-to: #1465
Each service, when requesting access to the Supervisor API, will
now get an individual key which can be scoped to specific resources.
In this iteration the default scope will be to the application that
the service belongs to.
We also have a `global` scope which is used by the cloud API when in
managed mode.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
We provide a local DNS server for containers to use and this
was not allowed through the firewall when enabled.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
When invoking iptables-restore it can fail. This wasn't handled
and this makes sure that it fails gracefully.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
The host config variable HOST_DISCOVERABILITY can be set to
true or false, controlling the state of the avahi service. This
determines if the device advertises it's presence over mDNS.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
Signed-off-by: Rich Bayliss <rich@balena.io>
Controlled by BALENA_HOST_FIREWALL_MODE, the firewall can
either be 'on' or 'off'.
- In the 'off' state, all traffic is allowed.
- In the 'on' state, only traffic for the core services provided
by Balena is allowed.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
When reporting device information, send the MAC address of any
interfaces on the system. Also expose in the Supervisor API at
the route GET /v1/device.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
This is part of the work to make the application-manager module much
less monolithic, in preperation for system apps and more generally
multi-app.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
We were treating the database class as a singleton, but still having to pass
around the db instance. Now we can simply require the db module and have
access to the database handle.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
This allows a response to an input with dport=`supevisor api port` and
is required when the host OS is doing stateful firewalling.
This should not affect things when stateful firewalling is not in
effect, as the standard OUTPUT chain policy is ACCEPT, so we're just
being explicit about it.
Change-type: patch
Backport-to: next, current, sunset
Signed-off-by: Cameron Diver <cameron@balena.io>
We add an implicit .0 to the end of l4t versions which do not fulfill
semver, which allows us to always match using comparison operators, such
as < and <=.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@balena.io>
ConfigJson users now don't need to worry about initialization.
It's done lazily on the first public method call.
Implementation is also rewritten to follow async/await pattern.
Change-type: patch
Signed-off-by: Roman Mazur <roman@balena.io>
A compose file can now contain a volume which uses a different driver
from the default one; local.
Change-type: patch
Signed-off-by: Rich Bayliss <rich@balena.io>
If we had it eearlier, it would have prevented #1148.
When init is called, withImagesFromDockerAndDB in images.ts is called as well.
Change-type: patch
Signed-off-by: Roman Mazur <roman@balena.io>
Custom type definitions are removed for mixpanel module since they are embedded
into their npm package.
Change-type: patch
Signed-off-by: Roman Mazur <roman@balena.io>
These contracts can specify an OS version and supervisor version that
they require. If any of the containers in a release have requirements
that are not met, the release is rejected, and the previous release
continues to run.
Change-type: minor
Closes: #1086
Signed-off-by: Cameron Diver <cameron@balena.io>
Various fields returned from the docker daemon don't retain order (for
example the volumes field). We now only select certain array values to
compare taking order into account.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
With recent changes to local mode behavior, supervisor removes all engine objects
that were not present before entering local mode.
With this change, supervisor also detects its own resources and does not delete them
if it was updated during local mode operations.
Change-type: patch
Signed-off-by: Roman Mazur <roman@balena.io>
That way that this is performed is by first adding a depends_on entry
for the target service if it appears in a network mode. Then when we
generate the docker container for this service, we use the containerId
of the target container and replace the network_mode with
`container:<id>`.
When comparing state, we check that the containerId still points to the
contianerId of the target container, and in this way we ensure that
when a network mode target container changes, we change the dependent
container too.
Change-type: minor
Closes: #851
Signed-off-by: Cameron Diver <cameron@balena.io>
This change makes DeviceState to wait until local mode switch is definitely
completed before actually applying the state, which avoids races in state cleanup.
Change-type: patch
Signed-off-by: Roman Mazur <roman@balena.io>
In local mode, we now update device status on the backend,
but omit applications info in our updates.
Closes: #959
Change-type: minor
Signed-off-by: Roman Mazur <roman@balena.io>
Changes in the node engine related to streams would cause the gzip
streams flush function to be called at the wrong times. The sinon fake
timers were also interacting with this.
We use setImmediate to call the flush function, and remove sinon timers
for the logging tests.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
When assigning multiple host ports to a single container port before
this change, the supervisor would incorrectly take only the first host
port into consideration. This change makes it so that every host port
per container port is considered.
Closes: #986
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
We add a database table, which holds information about the last
timestamp of a log successfully reported to a backend (local or remote).
We then use this value to calculate from which point in time to start
reporting logs from the container. If this is the first time we've seen
a container, we get all logs, and for every log reported we save the
timestamp. If it is not the first time we've seen a container, we
request all logs since the last reported time, ensuring no interruption
of service.
Change-type: minor
Closes: #937
Signed-off-by: Cameron Diver <cameron@balena.io>
This commit does two related things:
* We make the poll interval a random time between 0.5 and 1.5 times the configured interval.
* We introduce the BALENA_SUPERVISOR_INSTANT_UPDATE_TRIGGER configuration variable, that defaults to true. If this variable is set
to false, then calls to /v1/update are ignored, and on startup the supervisor waits for a poll interval before getting the target state.
This will help especially on cases where there's a large number of devices on a single network. By disabling instant updates and setting a large
poll interval, we can now achieve a sitation where not all devices apply an update at the same time, which can help avoid
overwhelming the network.
Change-type: minor
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
Before this change, service name resolution would only occur in the
default network. This was because we were not explicitly adding aliases
of the service names to the aliases fields.
We also fix the comparison, which would do funny things based on
container IDs, which was correct but unnecessary.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
We run the risk of the state engine exiting early when a dependency is
not ready, especially in local mode. This changes forces a noop to be
returned when we are waiting on another service, which is the process
used elsewhere in the state engine.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
We were not allowing newlines previously by virtue of the regex not
allowing them. The docker daemon and supervisor handling code both
support them, so we allow them in the parsing code too.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
Adjacent ports are always grouped together by docker when reporting the
container state (from an inspect), so adjacent ports defined in the
compose file would not match as they would not have been normalized.
We make sure to always normalize the input port configuration, so that
it will match the docker output (if it should).
We also don't sort in the fromComposePorts function anymore as that is
handled by the normalize function.
Closes: #897
Change-type: patch
Signed-off-by: Cameron Diver <cameron@balena.io>
We define the type for each config value, and validate the data when
retrieving and setting it.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@balena.io>
We've seen at least one case where the 10ms delay
wasn't enough, causing the tests to fail where they shouldn't.
(Ideally we should find a better way to detect when the streams
have finished propagating, but this will avoid failures in the meantime).
Change-type: patch
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
This avoids issues on provisioning where the current state
(esp. config.txt) that we want to save is retrieved without
a RESIN_ or BALENA_ prefix, causing those values to be lost.
Change-type: patch
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
They will take precedence over any existing RESIN_ variables. We strip both namespaces now
whenever we get the target values.
This also fixes preloading with a legacy config (the interface to get the config keys from
the legacy apps.json was broken).
Change-type: minor
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
But we keep backwards compatibility by normalizing existing io.resin labels
into io.balena ones, and adding both RESIN_ and BALENA_ env vars for these features.
Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
We change the lockfile to /tmp/balena/updates.lock, and the resin-kill-me file to /tmp/balena/handover-complete.
In the host, we change to use /tmp/balena-supervisor instead of /tmp/resin-supervisor.
We add BALENA_ env vars in addition to the RESIN_ env vars.
We keep backwards compatibility by using both paths for the lockfile and handover, and keeping the RESIN_ env vars.
Changelog-entry: Move the handover and lock files to /tmp/balena, rename them, and add BALENA_ env vars
Change-type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
This enables the switch to be added to the compose, and the handling of
docker messages has been changed to ensure that the multiplexed logs
which result are handled properly.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
Stability improvements;
* Printing of unsupported compose fields
* Added a lot of tests
* All compose configuration has a default value, enabling better
comparison
Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
