Commit Graph

1619 Commits

Author SHA1 Message Date
Pablo Carranza Velez
dc62418db4 Some fixes in current state reporting, error handling for "container not found", plus more style improvements
Also, ensure the properties argument to eventTracker.track is an object

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
bb57bcc37c Fix default volume name in migrations, and do not use the legacy-container flag on containers from older supervisors (they'll be restarted anyways)
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
58fede2976 Only report initial config variables when they're different from the default
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
58b167b43d Various bugfixes and sytlistic improvements
* Use the correct defaults for the delta config variables that have them

* Only mount /lib/firmware and /lib/modules if they exist on the host

* hardcode-migrations.js: Nicer line separation

* APIBinder: switch to using a header for authentication, and keep credentials saved in the API clients

* Fix hrtime measurements in milliseconds

* Do not uses classes for routers

* compose: properly initialize networkMode to the first entry in networks if there is one

* Fix some details regarding defaults in validation and service

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
ec22bfcb29 Support for several compose features
Plus a few bugfixes.

* Add support for cgroup_parent

* Add support for specifying a single value in tmpfs

* Fix support for extra_hosts

* Add support for group_add

* Add support for pid mode (only host and empty value are supported for now)

* Add support for pids_limit

* Add support for security_opt

* Add support for storage_opt

* Add support for userns_mode

* Add support for ipc (except for another container's)

* Add support for mac_address

* Add support for oom_kill_disable

* Add support for 'user' compose option

* Add support for working_dir and fix support for user when image specifies it

* Add support for bind-mounting the balena socket using a label

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
38c3a8bdf3 Avoid starting services that exit repeatedly
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
ba829412e1 Various bugfixes
* Get imageId when normalising a dep. app from the DB

* Fix the appId in migrations when updating the supervisor

* Use the update lock to update a service's metadata

* Restart clears volatile target state

* Fix function definition for updateMetadata

* Improve backwards compatibility of /v1/apps/:appId endpoint

* Fix multicontainer deltas to work with resumable-request 2.0

* Fix dependent target normalisation logic

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Lucian Buzzo
da2359e224 Fix API payload when provisioning a dependent device 2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
7ed27ea203 Some fixes on migrations, dependent devices and deltas
* Switch default dependent device type to generic

* Reduce noise in logs

* Limit to 3 simultaneous delta downloads

* Better check for deltaSource

* When checking volume dependencies, do not compare regular (non-named) volumes

* Store imageId for dependent apps, and don't report dependent images with invalid imageIds

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
e43c9052dd Improve backwards-compatible response of GET /v1/device
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
3fd52bb0c7 Simplify the update logic by making fetch and kill (the only long-running actions) happen in the background, and always waiting for all actions before continuing
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
c0ac2c21a4 Do not override stop_grace_period
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
ef849f8d82 When exiting, try to clean up any taken update locks
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2ed3c832ac Do a regular pull instead of delta from scratch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
484a688dbd Pause updates while purging or restarting apps, and ensure an applyTarget is triggered after the actions run
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
82d5a16c8c Add change events when services exit and restart, and add stop-service and start-service endpoints
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
8548222a00 Several bugfixes:
* Ensure commit is only reported when update has finished

* Change default delay between actions to 100ms

* Fix envArrayToObject for cases where the env var has an equal sign

* Use shell-quote to properly parse string command and entrypoint

* Fix preloading with a legacy apps.json

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
0a6d948bd2 Some fixes on image management and tagging
* Fix deleting unneeded image tags

* Fix inspectByName to work with tags besides digests when the image isn't really tagged

* Tag deltas that should have tags, and fix cleanup of dangling images

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2809d3c2ca Avoid failed updates causing several instances of applyTarget
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
554d4789ff Add support for all possible port definitions in services
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
cba3a8e5fe Make v1 restart and purge work for multicontainer apps too
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2b223f55fa Fix image comparison when running intermediate targets with depends_on
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
8956fc1e5a Actually send the hostname as part of container config
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
534f7d13cb Fix local mode and the host-config endpoint
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
cee8aae850 When in local mode, stop all services and ignore images
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
5ec8e57aa0 Implement v2 API endpoints to restart and purge apps, and restart a service
This also changes the deviceState object to use promises instead of timeouts to schedule
applying the target state.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
f653fa4961 Add support for service hostname
Plus several small bug fixes:

* Allow target states with apps with no release

* Fix lock override and a TypeError in compareServicesForUpdate

* Lowercase service names when doing migrations and legacy preload

* Fix deltas from scratch

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
839ebf8688 Fix preloaded apps and support legacy preloading, and fix some details in the default service when migrating
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
ac3e31edfb Support the legacy RESIN_OVERRIDE_LOCK
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
056a5bc6c9 dind: Use Wants instead of Requires so that the supervisor doesn't exit when VPN is turned off
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
60f0cd2fcb Change all labels to use hyphens instead of underscores, and fix some instances of using split instead of a regex
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
3a710506a6 Switch to a new image management system keeping the docker image ID in the database, allowing deltas and proper comparison for images that have a digest.
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
d84bcf0fb4 When applying host config values like dtoverlay and dtparam, take values not starting with double quotes as single entries instead of arrays to parse
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
918372b569 Some bugfixes and style improvements
* Fix validation of 0, fix ulimits, don't compare mem_limit or mem_reservation until OS supports them

* Remove all instances of _.forEach

* ApplicationManager: have separate compareNetworksForUpdate and compareVolumesForUpdate

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
1706afa7a2 Remove deprecated and broken OOM protection from gosuper, and clean up its dependencies and unused files
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
25695aade5 Add support for init, mem_reservation, shm_size, read_only and sysctls.
Also several bugfixes:

* Fix VPN control, logging in deviceConfig, and action executors in proxyvisor

* Fix bug in calculation of dependencies due to fields still using snake_case

* Fix snake_case in a migration, and remove unused lib/migration.coffee

* In healthcheck, count deviceState as healthy when a fetch is in progress (as in the non-multicontainer supervisor)

* Set always as default restart policy

* Fix healthcheck, stop_grace_period and mem_limit

* Lint and reduce some cyclomatic complexities

* Namespace volumes and networks by appId, switch default network name to 'default', fix dependencies in networks and volumes, fix duplicated kill steps, fix fat arrow on provisioning

* Check that supervisor network is okay every time we're applying target state

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
739fe13cad Use a supervisor0 network interface for the supervisor network API. Remove RESIN_APP_COMMIT and RESIN_APP_RELEASE env vars.
Also add support for several networks per container (but with no configuration yet).
Also some bugfixes and implement healthcheck and not disabling VPN on startup.

Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
b003f48d7b Switch to using knex migrations to set up the database, and change the database format to use integers for ids instead of strings.
Also includes various improvements and bugfixes to services and the migration from legacy /data to volumes.

The switch ti migrations involves a dirty hack for webpack to properly resolve the paths to the migrations js files - it uses an expression
that webpack can't resolve, so we hardcode it to a value and use the ContextReplacementPlugin to make that value resolve to the migrations folder.

The downsides to this approach are:
- a change in knex code would break this
- the migration code is added twice to the supervisor image: once in the migrations folder (because knex needs to loop through the directory to find the files),
and once inside app.js (because I can't make webpack treat them as external)

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
7c98a9d058 Supervisor API: remove the tcp-ping endpoints
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
7d8a208a06 ApplicationManager: Avoid deadlocks by killing services once its dependencies have been downloaded, and killing services with handover when it is absolutely necessary
Two cases could've caused deadlocks:
1) Two services use a volume, and one service depends on the other. The volume config changes, but we can't update the volume because we need to kill
both services, and yet we can't kill the dependent service because its dependency isn't ready either.
2) A service with handover strategy uses a volume. The volume config changes. We can't update the volume because the running service is using it, and we can't
start the handover because it depends on the volume being ready. So we need to kill the service to update the volume config.

(Same for networks as with volumes)

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
0d27658a87 Various improvements and fixes to how compositions are handled
Change the way we get the network gateway to set up the supervisor API address.

Added support for cap_add, cap_drop and devices.

Some fixes like missing fat arrows and removing leftover code.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
5f651c71f7 app.coffee: Switch to the multicontainer supervisor, add missing dependencies, and remove all files that are not used anymore
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
14d2bc3f39 APIBinder: implement a module to handle all interactions with the Resin API
This module provisions the device and takes care of getting the target state from the API, calling deviceState to apply it.
It also reports the current state of the device back to the API.

An important change is that the initial values of the device configuration (e.g. config.txt) are reported to the API, creating new config
variables if no values exist for a particular key. This will allow better management of config.txt by giving visibility to the initial configuration.

Changelog-Entry: Remove support for keeping the provisioning apiKey on Resin OS 1.X. Report initial values from config.txt and other device configuration variables to the Resin API.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
bc191ee86c Proxyvisor: implement the Proxyvisor for the multicontainer supervisor
This will be quickly replaced by a newer version with a different API, but for now we needed to maintain backwards compatibility (see #508).

This proxyvisor handles dependent apps and devices with a multicontainer parent app.
It also switches to the new update mechanism by inferring and applying updates step by step.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
195697a7e1 compose: implement the models that make up multicontainer applications
This commit adds models to manage services, images, volumes and networks.

The main model for this is ServiceManager, which manages the collection of services on the device. It has functions to query what services are running, and to perform actions like starting, killing or performing handovers.

The Service model allows defining the transformations between a container and its service representation, and includes the functions to compare a running service with a target to determine if an update needs to happen.
This model includes the relevant compose file entries for a service that are supported. Bind mounts are disallowed except for the ones that relate to supervisor features, and persistent data is now stored in named volumes.

The Images model allows fetching and removing images, and includes functionality to determine images that have to be cleaned up - now only dangling and old supervisor images are cleaned up automatically, and ApplicationManager
will remove images that correspond to old services that are no longer needed.

The Networks and Volumes models allow managing named networks and volumes that are part of composed applications.

Changelog-Entry: Remove all bind mounts that were specific to 1.X devices. Move the resin-kill-me file for the handover strategy to /tmp/resin. Add environment variables for the location of resin-kill-me and the lockfile. Use running containers to determine what services are running instead of storing them in the internal database. Use named volumes for persistent data.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
be5623cbf1 DockerUtils: implement the docker utilities library as a class
This commit implements what we used to have in docker-utils.coffee now making use of coffeescript classes.

We remove the cleanup function as this is now handled directly by the ApplicationManager.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
dac153eb8c updateLock: implement a module for a file-based update lock
This update lock library allows an application to take a lockfile in several locations (subdirectories inside a base folder). The user of this library must be able
to exclusively create a lockfile in each of the corresponding locations, and if any of the files exist, the locking fails.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
cb37f7ebcc ApplicationManager: implement a module to run multicontainer applications
This module takes care of inferring and applying the steps to run multicontainer applications. It will have a Proxyvisor to handle dependent apps and
devices. It understands the relationship between services, networks and volumes to infer the steps in the correct order, also taking update strategies into account.

Changelog-Entry: Allow running docker-compose-like multicontainer applications
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
7ae7ceab73 gosuper: add internal endpoints to get VPN and log-to-display status, and remove purge and IP address endpoints
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
d3e98eab11 DeviceConfig: implement a module to manage device configuration, including config.txt
This model allows modifying config.txt on raspberry pi devices, as well as logging to display, bandwidth control variables and other supervisor
configuration settings. Configuration values are read from the underlying OS and the supervisor configuration where appropriate (i.e. the Config object), instead of storing the current state
in the database. This means that the supervisor will always use the real values to determine if changes have to be made.

This fixes several issues with config.txt, as the current values are now read from the file, and can be reported on the supervisor's first run (which will be implemented in APIBinder).

It also now treats dtoverlay and dtparam values as a JSON array without the enclosing brackets, for instance:

```
RESIN_HOST_CONFIG_dtparam="audio=on","spi=on"
```

Will produce the following lines in config.txt:

```
dtparam=audio=on
dtparam=spi=on
```

Changelog-Entry: Implement inference of device configuration. Allow array values for dtoverlay and dtparam.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00