Commit Graph

833 Commits

Author SHA1 Message Date
Cameron Diver
73b19a320f
Change intialConfigReported value to be an api endpoint
This api endpoint is the endpoint which the intial config was reported.
Also changed the code to detect if the api endpoint has changed, and
therefore whether we need to re-report the initial config, to avoid
losing hardware specific information.

Change-type: minor
Closes: #649
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-16 14:18:03 +01:00
Cameron Diver
96d7bbefd0
Fix handling of systemd errors when polling for log to display
Change-type: patch
Closes: #610
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-08 15:12:13 +01:00
Cameron Diver
9150c3fdbc
Convert conversions module to typescript
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-02 09:36:21 +01:00
Cameron Diver
367e90d6e4
Move shared types to separate module
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-02 09:36:21 +01:00
Cameron Diver
273224547c
Convert blink module to typescript
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-02 09:36:21 +01:00
Cameron Diver
c1fbc2dc21
Convert lib/constants module to typescript
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-02 09:36:21 +01:00
Cameron Diver
67b9ec6ca7
Type parameters for validation functions better
They can now be string | null | undefined.

Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-05-02 09:36:21 +01:00
Pablo Carranza Velez
652b596c80 Add some more unit tests to the multicontainer supervisor
We add a bunch of additional unit tests, and also a coverage report using istanbul.

The tests are not meant to cover everything, but they're a first attempt at having *some* unit testing
on the supervisor. There's much to improve but hopefully it helps catch obvious errors.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-04-30 19:39:01 -07:00
Cameron Diver
ddbf3418de
Remove trailing slashes from working directories of services
This is to combat when a working directory is in the compose file for a
service with a trailing slash. Docker will strip this slash and that
means service comparisons will fail going forward - even if they are the
same.

Change-type: patch
Closes: #635
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-04-27 18:03:36 +01:00
Cameron Diver
cfddbf65e4
Start initial typescript conversion, and add validation debugging
Add webpack config and dependencies to have typescript built, and also
convert src/lib/validation.coffee to typescript.

In this conversion I also added a lot of debugging which should help the
upcoming local mode development.

Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-04-27 16:09:50 +01:00
Pablo Carranza Velez
6c4eaff02f On startup, only attach to logs for running containers, and remove any containers marked as dead
Closes #611
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-04-19 15:04:04 -07:00
Akis Kesoglou
1cde9dd16d Properly propagate delta failure
Change-Type: patch
Closes: #627
2018-04-18 18:34:42 +03:00
Pablo Carranza Velez
015d33f8f5 Allow services (and their images) to expose udp ports besides tcp
Closes #621
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-04-11 15:13:48 -07:00
Cameron Diver
1c27ebc354
Fix parsing of non-unit memory numbers and add tests
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-04-04 14:03:24 +01:00
Cameron Diver
7d233ea111
Fix regex for parsing memory numbers
It now allows a trailing `b`, as the docker-compose docs specify.

In addition the regex now specifies a case-insensitive flag, to catch
both upper and lower case memory numbers (the rest of the function
supported these already).

Change-type: patch
Closes: #603
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-04-04 11:25:37 +01:00
Cameron Diver
393671505c
Respond to reboot and shutdown endpoints with a success object
Change-type: patch
Closes: #607
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-03-29 13:11:02 +01:00
Pablo Carranza Velez
72a5f03b0c
Back off fetching the target state exponentially, for faster retries when there's no connectivity
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-29 10:20:52 +01:00
Cameron Diver
e458c4103c
Handle incorrectly parsed env vars from docker inspect
Change-type: patch
Closes: #604
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-03-26 16:56:06 +01:00
Pablo Carranza Velez
348ff66cee
Replace the gosuper component with a node module that handles communication with systemd, and stop using an init system in the supervisor container
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-22 15:55:15 +00:00
Pablo Carranza Velez
c8d79c3b7d Remove any leftover knex migrations locks before running migrations
Closes #598
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-21 18:20:15 -03:00
Pablo Carranza Velez
d46d6513b5 Trim whitespace from the hostname file on the host before passing it as the hostname for containers with host network
Closes #594
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-19 17:19:59 -03:00
Pablo Carranza Velez
e1e33b376e Force reboots and shutdowns if lock override is enabled
Closes #440
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-19 16:30:59 -03:00
Pablo Carranza Velez
91ac11d0e8 In /v1/apps/:appId/stop, wait for the service to exit before responding
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-16 15:13:15 -03:00
Pablo Carranza Velez
5d7e8afcee Avoid trying to clean up the supervisor image if it has several tags (even though it would never succeed)
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 22:38:52 -03:00
Pablo Carranza Velez
75ce55e932 Fix typo passing apiKey to the resin API client when exchanging apikeys
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 21:50:31 -03:00
Pablo Carranza Velez
7e342e9d80 Fix the check for whether the device is provisioned by correctly getting registered_at and deviceId from config.json into the ApiBinder
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 21:50:31 -03:00
Pablo Carranza Velez
08a98f47e4 compose: Use the hostname on the host as default hostname for services with host network mode
Otherwise if the hostname on the supervisor container differs from the hostname on the host, the current and target
services will never match.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 21:18:11 -03:00
Pablo Carranza Velez
557b6a8d07 Pass the correct working_dir to the container config, and compare containers for working dir changes
We were getting the correct working dir from the compose or image config, but not really using it.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 15:41:19 -03:00
Pablo Carranza Velez
1febdc9eb0 DeviceConfig: avoid trying to enable or disable the VPN when in offline mode
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-14 10:15:07 +02:00
Pablo Carranza Velez
0101d62ace ApplicationManager: try to match available images by imageId to avoid keeping around unused image entries in the db
This should fix an issue where, on an update that only changes container metadata, the image install for the old image
is kept around on the API.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-13 18:45:26 -03:00
Pablo Carranza Velez
4ba4736aba compose: Emit a change event when a container is renamed, so that the updated imageId and releaseId are reported
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-13 18:45:26 -03:00
Pablo Carranza Velez
de654ec60e Skip taking the lock when updating the releaseId or imageId for a service that comes from a legacy supervisor
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-13 18:00:07 -03:00
Pablo Carranza Velez
9dfb76d2c8 When migrating from legacy supervisors, mark the temporary composition as legacy
We migrate to a default composition because we need to avoid deleting existing docker images, but
we need to use the legacy-container label to avoid potentially creating a duplicated container when a target state comes in.

(Just like we do for preloaded apps)

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-13 08:17:25 -07:00
Pablo Carranza Velez
f76aacc7fb Avoid trying to send a response to API calls after we've already sent an error
In some cases we were using early `return res.status(...).send(...)` to send 400 errors
but this happened inside a promise chain that later sent another status and response.

We fix this with the correct indentation of the success response so that an early return doesn't fall there.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-09 14:48:34 -08:00
Pablo Carranza Velez
79b4d39acd Fix passing a target service to start in /v1/apps/:appId/start, and getting the containerId
We weren't passing a "target" to serviceAction, which made the start action fail.
Plus we need to get the container again after starting to get the latest containerId.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-09 14:41:19 -08:00
Pablo Carranza Velez
f009e34293 migrations: Use the correct service name for legacy apps
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-09 06:54:16 -08:00
Pablo Carranza Velez
b6631b7367 ApplicationManager: only use dockerImageId to identify images for current apps, to avoid trying to delete an image that is in use
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-09 06:54:16 -08:00
Cameron Diver
d27c529ebe
Fix bug in require for migrations for legacy preload
Change-type: patch
Connects-to: #573
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-03-09 14:10:11 +00:00
Petros Angelatos
4ba020a784
compose: fix type error, createVolume returns a promise
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
Change-type: patch
2018-03-09 00:00:44 -08:00
Petros Angelatos
7d03de51f6
lib: add missing flag to openAsync()
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
Change-type: patch
2018-03-08 14:59:07 -08:00
Petros Angelatos
4b83a08512
compose: fix volume migration code
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
Change-type: patch
2018-03-08 14:15:50 -08:00
Cameron Diver
a150dbf329
Convert object to array when normalising legacy target apps
Change-type: patch
Connects-to: #567
Signed-off-by: Cameron Diver <cameron@resin.io>
2018-03-08 15:48:03 +00:00
Petros Angelatos
cda561f997 Introduce workaround for IPv6 DNS resolution until nodejs becomes RFC 3484 compliant
Change-Type: patch
2018-03-07 07:01:43 -08:00
Pablo Carranza Velez
1b043230a5 Send logs to the resin API by default
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 21:10:04 -08:00
Pablo Carranza Velez
15da221382 Implement a new logger that sends logs to the resin API, that can be used optionally instead of PubNub
Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 21:09:06 -08:00
Pablo Carranza Velez
16f12920c1 Fix the way commands and entrypoints in string form are parsed
Turns out shell-quote's parse function also replaces environment variables, which we don't want in this case. So we escape dollar signs
before calling shell-quote's parse function.

Also shell-quote takes some characters like `>` and globs and returns an object - so we return those objects to string form.

This should still be simpler/better than writing our own shlex.split, I hope...

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
dc62418db4 Some fixes in current state reporting, error handling for "container not found", plus more style improvements
Also, ensure the properties argument to eventTracker.track is an object

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
bb57bcc37c Fix default volume name in migrations, and do not use the legacy-container flag on containers from older supervisors (they'll be restarted anyways)
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
58fede2976 Only report initial config variables when they're different from the default
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
58b167b43d Various bugfixes and sytlistic improvements
* Use the correct defaults for the delta config variables that have them

* Only mount /lib/firmware and /lib/modules if they exist on the host

* hardcode-migrations.js: Nicer line separation

* APIBinder: switch to using a header for authentication, and keep credentials saved in the API clients

* Fix hrtime measurements in milliseconds

* Do not uses classes for routers

* compose: properly initialize networkMode to the first entry in networks if there is one

* Fix some details regarding defaults in validation and service

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
ec22bfcb29 Support for several compose features
Plus a few bugfixes.

* Add support for cgroup_parent

* Add support for specifying a single value in tmpfs

* Fix support for extra_hosts

* Add support for group_add

* Add support for pid mode (only host and empty value are supported for now)

* Add support for pids_limit

* Add support for security_opt

* Add support for storage_opt

* Add support for userns_mode

* Add support for ipc (except for another container's)

* Add support for mac_address

* Add support for oom_kill_disable

* Add support for 'user' compose option

* Add support for working_dir and fix support for user when image specifies it

* Add support for bind-mounting the balena socket using a label

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
38c3a8bdf3 Avoid starting services that exit repeatedly
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
ba829412e1 Various bugfixes
* Get imageId when normalising a dep. app from the DB

* Fix the appId in migrations when updating the supervisor

* Use the update lock to update a service's metadata

* Restart clears volatile target state

* Fix function definition for updateMetadata

* Improve backwards compatibility of /v1/apps/:appId endpoint

* Fix multicontainer deltas to work with resumable-request 2.0

* Fix dependent target normalisation logic

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Lucian Buzzo
da2359e224 Fix API payload when provisioning a dependent device 2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
7ed27ea203 Some fixes on migrations, dependent devices and deltas
* Switch default dependent device type to generic

* Reduce noise in logs

* Limit to 3 simultaneous delta downloads

* Better check for deltaSource

* When checking volume dependencies, do not compare regular (non-named) volumes

* Store imageId for dependent apps, and don't report dependent images with invalid imageIds

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
e43c9052dd Improve backwards-compatible response of GET /v1/device
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
3fd52bb0c7 Simplify the update logic by making fetch and kill (the only long-running actions) happen in the background, and always waiting for all actions before continuing
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:29 -08:00
Pablo Carranza Velez
c0ac2c21a4 Do not override stop_grace_period
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
ef849f8d82 When exiting, try to clean up any taken update locks
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2ed3c832ac Do a regular pull instead of delta from scratch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
484a688dbd Pause updates while purging or restarting apps, and ensure an applyTarget is triggered after the actions run
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
82d5a16c8c Add change events when services exit and restart, and add stop-service and start-service endpoints
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
8548222a00 Several bugfixes:
* Ensure commit is only reported when update has finished

* Change default delay between actions to 100ms

* Fix envArrayToObject for cases where the env var has an equal sign

* Use shell-quote to properly parse string command and entrypoint

* Fix preloading with a legacy apps.json

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
0a6d948bd2 Some fixes on image management and tagging
* Fix deleting unneeded image tags

* Fix inspectByName to work with tags besides digests when the image isn't really tagged

* Tag deltas that should have tags, and fix cleanup of dangling images

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2809d3c2ca Avoid failed updates causing several instances of applyTarget
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
554d4789ff Add support for all possible port definitions in services
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
cba3a8e5fe Make v1 restart and purge work for multicontainer apps too
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
2b223f55fa Fix image comparison when running intermediate targets with depends_on
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
8956fc1e5a Actually send the hostname as part of container config
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
534f7d13cb Fix local mode and the host-config endpoint
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
cee8aae850 When in local mode, stop all services and ignore images
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
5ec8e57aa0 Implement v2 API endpoints to restart and purge apps, and restart a service
This also changes the deviceState object to use promises instead of timeouts to schedule
applying the target state.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
f653fa4961 Add support for service hostname
Plus several small bug fixes:

* Allow target states with apps with no release

* Fix lock override and a TypeError in compareServicesForUpdate

* Lowercase service names when doing migrations and legacy preload

* Fix deltas from scratch

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
839ebf8688 Fix preloaded apps and support legacy preloading, and fix some details in the default service when migrating
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
ac3e31edfb Support the legacy RESIN_OVERRIDE_LOCK
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
60f0cd2fcb Change all labels to use hyphens instead of underscores, and fix some instances of using split instead of a regex
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
3a710506a6 Switch to a new image management system keeping the docker image ID in the database, allowing deltas and proper comparison for images that have a digest.
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
d84bcf0fb4 When applying host config values like dtoverlay and dtparam, take values not starting with double quotes as single entries instead of arrays to parse
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
918372b569 Some bugfixes and style improvements
* Fix validation of 0, fix ulimits, don't compare mem_limit or mem_reservation until OS supports them

* Remove all instances of _.forEach

* ApplicationManager: have separate compareNetworksForUpdate and compareVolumesForUpdate

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
25695aade5 Add support for init, mem_reservation, shm_size, read_only and sysctls.
Also several bugfixes:

* Fix VPN control, logging in deviceConfig, and action executors in proxyvisor

* Fix bug in calculation of dependencies due to fields still using snake_case

* Fix snake_case in a migration, and remove unused lib/migration.coffee

* In healthcheck, count deviceState as healthy when a fetch is in progress (as in the non-multicontainer supervisor)

* Set always as default restart policy

* Fix healthcheck, stop_grace_period and mem_limit

* Lint and reduce some cyclomatic complexities

* Namespace volumes and networks by appId, switch default network name to 'default', fix dependencies in networks and volumes, fix duplicated kill steps, fix fat arrow on provisioning

* Check that supervisor network is okay every time we're applying target state

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
739fe13cad Use a supervisor0 network interface for the supervisor network API. Remove RESIN_APP_COMMIT and RESIN_APP_RELEASE env vars.
Also add support for several networks per container (but with no configuration yet).
Also some bugfixes and implement healthcheck and not disabling VPN on startup.

Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
b003f48d7b Switch to using knex migrations to set up the database, and change the database format to use integers for ids instead of strings.
Also includes various improvements and bugfixes to services and the migration from legacy /data to volumes.

The switch ti migrations involves a dirty hack for webpack to properly resolve the paths to the migrations js files - it uses an expression
that webpack can't resolve, so we hardcode it to a value and use the ContextReplacementPlugin to make that value resolve to the migrations folder.

The downsides to this approach are:
- a change in knex code would break this
- the migration code is added twice to the supervisor image: once in the migrations folder (because knex needs to loop through the directory to find the files),
and once inside app.js (because I can't make webpack treat them as external)

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
7c98a9d058 Supervisor API: remove the tcp-ping endpoints
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
7d8a208a06 ApplicationManager: Avoid deadlocks by killing services once its dependencies have been downloaded, and killing services with handover when it is absolutely necessary
Two cases could've caused deadlocks:
1) Two services use a volume, and one service depends on the other. The volume config changes, but we can't update the volume because we need to kill
both services, and yet we can't kill the dependent service because its dependency isn't ready either.
2) A service with handover strategy uses a volume. The volume config changes. We can't update the volume because the running service is using it, and we can't
start the handover because it depends on the volume being ready. So we need to kill the service to update the volume config.

(Same for networks as with volumes)

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
0d27658a87 Various improvements and fixes to how compositions are handled
Change the way we get the network gateway to set up the supervisor API address.

Added support for cap_add, cap_drop and devices.

Some fixes like missing fat arrows and removing leftover code.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
5f651c71f7 app.coffee: Switch to the multicontainer supervisor, add missing dependencies, and remove all files that are not used anymore
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
14d2bc3f39 APIBinder: implement a module to handle all interactions with the Resin API
This module provisions the device and takes care of getting the target state from the API, calling deviceState to apply it.
It also reports the current state of the device back to the API.

An important change is that the initial values of the device configuration (e.g. config.txt) are reported to the API, creating new config
variables if no values exist for a particular key. This will allow better management of config.txt by giving visibility to the initial configuration.

Changelog-Entry: Remove support for keeping the provisioning apiKey on Resin OS 1.X. Report initial values from config.txt and other device configuration variables to the Resin API.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
bc191ee86c Proxyvisor: implement the Proxyvisor for the multicontainer supervisor
This will be quickly replaced by a newer version with a different API, but for now we needed to maintain backwards compatibility (see #508).

This proxyvisor handles dependent apps and devices with a multicontainer parent app.
It also switches to the new update mechanism by inferring and applying updates step by step.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:28 -08:00
Pablo Carranza Velez
195697a7e1 compose: implement the models that make up multicontainer applications
This commit adds models to manage services, images, volumes and networks.

The main model for this is ServiceManager, which manages the collection of services on the device. It has functions to query what services are running, and to perform actions like starting, killing or performing handovers.

The Service model allows defining the transformations between a container and its service representation, and includes the functions to compare a running service with a target to determine if an update needs to happen.
This model includes the relevant compose file entries for a service that are supported. Bind mounts are disallowed except for the ones that relate to supervisor features, and persistent data is now stored in named volumes.

The Images model allows fetching and removing images, and includes functionality to determine images that have to be cleaned up - now only dangling and old supervisor images are cleaned up automatically, and ApplicationManager
will remove images that correspond to old services that are no longer needed.

The Networks and Volumes models allow managing named networks and volumes that are part of composed applications.

Changelog-Entry: Remove all bind mounts that were specific to 1.X devices. Move the resin-kill-me file for the handover strategy to /tmp/resin. Add environment variables for the location of resin-kill-me and the lockfile. Use running containers to determine what services are running instead of storing them in the internal database. Use named volumes for persistent data.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
be5623cbf1 DockerUtils: implement the docker utilities library as a class
This commit implements what we used to have in docker-utils.coffee now making use of coffeescript classes.

We remove the cleanup function as this is now handled directly by the ApplicationManager.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
dac153eb8c updateLock: implement a module for a file-based update lock
This update lock library allows an application to take a lockfile in several locations (subdirectories inside a base folder). The user of this library must be able
to exclusively create a lockfile in each of the corresponding locations, and if any of the files exist, the locking fails.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
cb37f7ebcc ApplicationManager: implement a module to run multicontainer applications
This module takes care of inferring and applying the steps to run multicontainer applications. It will have a Proxyvisor to handle dependent apps and
devices. It understands the relationship between services, networks and volumes to infer the steps in the correct order, also taking update strategies into account.

Changelog-Entry: Allow running docker-compose-like multicontainer applications
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
d3e98eab11 DeviceConfig: implement a module to manage device configuration, including config.txt
This model allows modifying config.txt on raspberry pi devices, as well as logging to display, bandwidth control variables and other supervisor
configuration settings. Configuration values are read from the underlying OS and the supervisor configuration where appropriate (i.e. the Config object), instead of storing the current state
in the database. This means that the supervisor will always use the real values to determine if changes have to be made.

This fixes several issues with config.txt, as the current values are now read from the file, and can be reported on the supervisor's first run (which will be implemented in APIBinder).

It also now treats dtoverlay and dtparam values as a JSON array without the enclosing brackets, for instance:

```
RESIN_HOST_CONFIG_dtparam="audio=on","spi=on"
```

Will produce the following lines in config.txt:

```
dtparam=audio=on
dtparam=spi=on
```

Changelog-Entry: Implement inference of device configuration. Allow array values for dtoverlay and dtparam.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
2953b745ce Logger: implement a module that handles all logging to pubnub
This module can also send logs for dependent devices (by passing a specific channel to the "log" function).

The log types are also moved to a separate module to be used by modules that perform logging.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
93832d6540 network: implement a module to get IP addresses and check network connectivity
This module now uses the native node `os.networkInterfaces()` to retrieve the addresses,
instead of the gosuper endpoint.

We also add the very simple "blink" library that is also used by the Supervisor API.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
f77d3e1563 DeviceState: implement a module to manage the device's target and current state
This module will take care of applying the target state for the device and reporting its current state.
The state itself is handled by two other modules, ApplicationManager and DeviceConfig. The former will take care of running applications (including the dependent ones
via its Proxyvisor), and the latter will take care of device configuration like config.txt and supervisor configuration variables.

The way state is applied differs radically from the previous approach: the old application.coffee had a big `update` function that took all of the steps from fetching the target state
to running the containers. DeviceState, instead, does an iterative process through `triggerApplyTarget` of inferring the next steps to perform towards the target state, by looking at the current state and asking the ApplicationManager and DeviceConfig for
the next steps. It then applies the next steps and every time a step is completed, it schedules another round of inferring and applying the next steps.

Special care is taken to ensure `applyTarget` is not called simultaneously more than once.

This commit also adds a "device" module to handle reboot and shutdown, and moves gosuper calls to a separate module.

The module also uses a "network" module to manage network-related parts of the device's current state: IP addresses and the connectivity check.

The module implements a "normaliseLegacy" function that allows a migration from the models from older versions of the supervisor to the multicontainer models,
so that in case of a supervisor update we can have minimal downtime and bandwidth consumption when updating to the multicontainer supervisor - this migration allows
us to avoid cleaning up images, and also allows migrating the contents of the old /data for the app.

Changelog-Entry: Infer the current state of the device when applying the target state
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
0dc9fea4d3 DB, Config: Implement modules to interact with the sqlite DB and to manage configuration
These modules allow managing the models in the sqlite database and the configuration for the supervisor.

The database will now have a schema version, and the supervisor will normalize any legacy data when migrating
from an older schema (i.e. when doing a supervisor update). This will make model changes cleaner.
If a migration is needed, the DB initialization will return "true" and store the legacy data in a legacyData table. Once the supervisor finishes migrating the data,
it calls `db.finishMigration` to mark the migration complete and clear the legacyData table.

Changes in the models:
* The database implements the tables for multicontainer applications that now have services, networks and volumes as in a docker compose file.
* Dependent apps and devices now have separate tables to store their target states.
* The deviceConfig table now only stores target values, as the current ones will be inferred from the state of the device.
* We keep a table for images as we have no way to label them in docker storage, so we need to keep our own track of what images are relevant for the supervisor.

The Config object allows transparent management of configuration values, mainly through `get`, `getMany` and `set` functions. The values can be stored in config.json or
the database, and this is managed with a schema definition that also defines whether values are mutable and whether they have default values.

Some configuration values are of the "func" type, which means that instead of corresponding to a config.json or database key, they result from a helper function
that aggregates other configuration values or gets the value from other sources, like OS version and supervisor version.

Writes to config.json are atomic if a path to the file via /mnt/root can be found. We keep a write-through cache of the file to avoid unnecessary IO.

Changelog-Entry: Implement the multicontainer app models, and change the supervisor configuration management to avoid duplication between fields in config.json and fields in the internal database
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
23f81c28f5 EventTracker: add a module to track mixpanel events
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
60a4cccfd2 Supervisor: Implement a Supervisor class with a SupervisorAPI
This will be the top level object in the multicontainer supervisor, using the following objects
to perform its duties:

* A DB object to manage the sqlite database models
* A Config object to manage configuration in sqlite and config.json
* An EventTracker to track events and send them to mixpanel
* A DeviceState object to manage the device state, including containers, device configuration and dependent devices
* An APIBinder object to manage all interactions with the Resin API
* The SupervisorAPI, implemented here, which exposes functionality from the other objects over an HTTP API with apikey authentication.

We also include an iptables module that the SupervisorAPI will use to only allow traffic from certain interfaces.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
f2d5a59727 Allow truthy values for deltas and lock override (i.e. the string 'true' besides '1')
We had previously done this for all the other configuration variables, but for some reason we had missed these two.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-02-27 09:27:45 -08:00
Akis Kesoglou
dc69917b5a Update resumable-request to v2.0
Turned out that disk I/O can be the bottleneck when applying deltas on some devices. When the disk can’t keep up and consume the downloaded delta, there’s memory bloat due to buffering.

The updated version provides far better reliability when the device is under load and pretty much constant memory consumption with any number of concurrent deltas.

Change-Type: patch
2018-02-05 10:59:56 +02:00
Pablo Carranza Velez
cff789ebfa Implement an API for proxy and hostname configuration, and centralize management of config.json
Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-17 14:05:40 -03:00
Pablo Carranza Velez
968d3ce1e3 Fix saving deviceApiKey to the DB (to fix the RESIN_API_KEY env var) when updating from some older supervisors
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-12 11:53:54 -03:00
Pablo Carranza Velez
8fac5c0e32 Avoid problems with null app names or invalid container names when migrating from old supervisors
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-05 11:10:41 -03:00
Pablo Carranza Velez
4b96ccbcc2 Avoid marking the supervisor as unhealthy if update is not happening because it's in offline mode
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-11 11:03:04 -08:00
Pablo Carranza Velez
54264e1b60 Measure time spent pulling images and substract it when considering the time between update cycles in the healthcheck
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-11 00:11:56 -08:00
Pablo Carranza Velez
9b4f912c7c Use the correct interface to compare times using process.hrtime to calculate time since last update cycle
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-11 00:11:56 -08:00
Pablo Carranza Velez
8484949a28 Fix config value for appUpdatePollInterval, and allow marking gosuper as healthy again after a failure
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-11 00:11:56 -08:00
Pablo Carranza Velez
95bbe6ea49 Add a /v1/healthy endpoint that fails if the supervisor is unhealthy, and a HEALTHCHECK command to the Dockerfile that uses it
We add an endpoint to the supervisor API that checks the following conditions to determine whether the supervisor is healthy:
* That the update cycle has run fully, in a time that's less than twice the poll interval. Unless we're downloading an image, in which case
we assume it's healthy (otherwise we'd get into the issue of determining a reasonable timeout for the image download, which is already done in a configurable way with delta options and the like).
* That the current state report to the Resin API hasn't failed more than 3 times. Unless the device has no connectivity, or the connectivity check is disabled, in which case we don't know
if the report failed simply because there's no network.
* That the gosuper component is working (since we periodically hit its API to get the IP addresses, we mark it as not working if this API call fails).

We need this endpoint to be unauthenticated for the docker daemon to be able to hit it (though, as the rest of the API, it is protected with iptables rules).

Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-11 00:11:56 -08:00
Pablo Carranza Velez
f65822ba94 Use for own instead of _.forEach
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-10 21:52:03 -08:00
Pablo Carranza Velez
5a0042c33d Apply config.txt changes when the new config is empty, to fix deleting config.txt values
I realized we're not deleting config.txt entries because the function checked for the values to apply
not to be empty, instead of just checking if the *changes* are empty.

So this closes #450

(Still not a complete solution to config.txt issues, which will come with the multicontainer PR, but at least it's a step forward)

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-10 19:09:10 -08:00
Pablo Carranza Velez
e0d1d89419 Filter deviceConfig values that will be stored as current values to only include relevant keys
Otherwise, devices where we update from legacy supervisors might have other keys, like RESIN_SUPERVISOR_DELTA, stored in deviceConfig.values,
causing `_.isEqual(values, targetValues)` to always return false.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-10 18:54:08 -08:00
Pablo Carranza Velez
c05474b1a9 Always execute special actions if the value stored in memory doesn't match the target. And when storing target values, only store relevant ones
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-10 18:48:30 -08:00
Pablo Carranza Velez
8fc1a0935b Avoid stopping the VPN until a remote target state has been fetched, and retry applying config variables when they fail
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-12-08 00:16:34 -08:00
Pablo Carranza Velez
21a9bb4e82 When listenPort is not specified, use 48484 as default
Should only be relevant in really old OS versions, but still this is the correct default.

Fixes #439

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-11-23 14:08:32 -08:00
Pablo Carranza Velez
51d6ab01c9 Avoid an indefinite recursion that grows the call stack when reporting the current state fails
We used to have a recursion based on Promises and Promise.delay, which caused the promise never to resolve
so eventually the stack would be exhausted.

This fixes it by using a simpler way to check if reporting the state is in progress and using a setImmediate to
call applyState outside of the Promise chain.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-11-02 15:54:09 -07:00
Pablo Carranza Velez
20d95ff024 Add whitelist-based filtering to mixpanel events
When sending events to mixpanel, we now use an explicit whitelist for the properties sent with the event, to avoid accidental leakage of any sensitive information.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-31 23:22:38 -07:00
Pablo Carranza Velez
34d37814c9 Tunnel all mixpanel events through the resin API
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-31 23:22:38 -07:00
Pablo Carranza Velez
ecf7e4206c Avoid fetching an image when it might be available or when starting an app because it might not be necessary
This change removes the behavior where we would try to fetch an app image when starting the app. This might cause an unintended
download of an app that is not really needed anymore because we're starting the app on boot and an update cycle would make this image unnecessary.
So now we try to inspect the image, and if this fails we will throw an error, causing the app to be soft-deleted and the next update cycle to properly trigger
a download of whatever image we need from the target state.

We also improve the error catching when fetching an image, to specifically catch an "image not found" error before trying to download - otherwise, any other
random error will cause us to try to download the image again, which will not be a noop if we're using deltas. If there's any other error, the correct behavior
is to throw and retry later.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-30 15:25:49 -07:00
Pablo Carranza Velez
0bc23df8c9 Refactor container cleanup to remove all spurious containers
We change the way container cleanup works so that it compares running
app containers with the container names for the known apps. This allows
the cleanup to effectively delete any spurious/duplicated app containers.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-30 15:25:49 -07:00
Pablo Carranza Velez
bd34a19a79 Use container name instead of id to identify apps, and avoid duplicated containers
By storing the container name before creating the container, we avoid problems
if the supervisor crashes or the device reboots between creating a container and storing its id.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-30 15:25:49 -07:00
Pablo Carranza Velez
c532344dce If a device is already provisioned but the key exchange fails, retry it until it succeeds
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-27 18:40:29 -07:00
Pagan Gazzard
21712ae810 Change the update retry to back off to the standard update check interval
This means that the supervisor will be less aggressive in the case of the api experiencing issues, stopping it from compounding the issue if the api is being overloaded

Change-type: patch
2017-10-24 15:36:43 -07:00
Pablo Carranza Velez
a87c6682a2 Ensure preloaded apps are properly loaded by setting their internal markedForDeletion to false, and run apps that have it set to null
Currently preloaded apps don't run because their markedForDeletion field in the database is null. In this commit we set it to false, and we
also change the startup check to also run any apps that have markedForDeletion as null (which should now never happen, but is still good as a backup
plan in case something else fails and to avoid regressions).

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-23 17:29:41 -07:00
Pablo Carranza Velez
3f198fc6aa Improve the check for when the device has been provisioned but the supervisor doesn't have knowledge of it in its local state
This change improves the check for the DuplicateUuidError that can happen if a device has been provisioned but the API's response hasn't been persisted - the error message
returned from the API has been known to have a few variations (usually an extra dot at the end), so we now use _.startsWith instead of checking for equal strings to make the
supervisor still work under these variations.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-23 17:28:36 -07:00
Pablo Carranza Velez
d98897cdcf Ensure preloaded apps get the deviceApiKey in the env vars, and apps never get the provisioning key, and improve detection of cases when the device has been pre-provisioned
It appears preloaded apps have been getting restarted because the "apiKey" configuration value was only available after provisioning succeeded. This change ensures the
deviceApiKey that the device will use is injected into the env vars of preloaded apps, ensuring the app is not restarted (unless provisioning fails and the uuid and deviceApiKey are
regenerated, but this should be rare).

We also ensure that whenever an app's RESIN_API_KEY env var is populated, it is *always* done with the deviceApiKey and never with the provisioning apiKey.

Closes #457
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-10-23 17:28:17 -07:00
Akis Kesoglou
78f74d757d Delta improvements
- Updates resumable-request to 1.0.1
- Updates docker-progress to 2.0.3
- Removes `DEFAULT_DELTA_APPLY_TIMEOUT`; it’s not needed anymore, docker-delta reliably tracks rsync.
- Properly end the update when applying the delta results in an error.

Change-Type: patch
2017-10-17 10:43:12 +03:00
Pablo Carranza Velez
31d09e70e4 Explicitly define the source for deltas, allow cross-app deltas, and iterate serially through apps when updating
This commit changes the way the source for a delta is determined. We used to do
it by comparing the available tags with the one we want and relying on the format that
includes the app in the image name. Now we explicitly choose a delta source from the previous app
version if we have one, and otherwise use the image from any available app - which will allow us
to have a valid source when moving a device between apps.

For this to work consistently if there's an unexpected reboot, we now avoid deleting an app from the db
until the full update has succeeded. Instead, we mark the app for deletion so that we still have the image stored after the reboot.

This commit also changes a .map to .mapSeries when iterating over appIds for removal/install/update - this avoids parallel treatment
of apps which can cause inconsistencies in the status reported to the API.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-09-14 14:52:06 -07:00
Pablo Carranza Velez
81a6c2f344 Fix problem catching errors when killing a container that doesn't exist
We've been using `.catch Promise.OperationalError, ...` to catch errors when stopping a container and
detecting whether the error means that the container has already been stopped of removed.

Apparently, after the recent dockerode upgrade these errors are not typed as OperationalError anymore, causing error
messages like "No such container: null" when applying an update. This commit makes us catch all errors and check for their statusCode.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-09-05 20:17:43 -07:00
Pablo Carranza Velez
dbb4fd8292 Prefer err.message when reporting errors from dockerode, then err.json and err.reason
Errors from docker-modem that are passed from dockerode can have a "json" or "reason" property,
but that is generally less descriptive than the more standard "message", and can show up in the logs
as `[object Object]`. This commit changes it so that we log err.message if it is non-empty, and otherwise
look for json and reason.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-09-01 15:08:10 -07:00
Akis Kesoglou
a5980918b4 Forward resume options
Change-Type: patch
2017-08-29 00:02:26 +03:00
Akis Kesoglou
52c55a0c1b Apply a default timeout unless one is given 2017-08-09 11:55:22 +03:00
Akis Kesoglou
1412785886 Try to resume the download of a delta if it fails due to flaky network
Applying a delta update consists of two parts:

1. The request to the delta server for the delta payload (an rsync batch file, plus some prepended Docker metadata). The response is a redirect to a URL that contains the delta (currently S3).
2. The request for the actual download of the delta. The response is streamed directly to rsync, which applies it onto the mounted root filesystem of the final image.

The first step may take a while as it may trigger the generation of the delta if the request is the first one for this combination of src/dest image and the images are large. If the request times out, either because of the delta server taking too long to respond or bad network, the Supervisor automatically schedules a retry to be performed after a while.

Currently, similar behaviour applies to the second step as well -- if the request fails, we immediately bail out and the Supervisor schedules a retry of the whole process (i.e. from step 1). But in this case it means we might have downloaded and applied some or most of the delta when a socket timeout occurs causing us to start all over again, wasting time and bandwidth.

This commit splits the process into the two discreet steps and improves the behaviour on the second step. Specifically:

- makes the Supervisor try to resume the delta download request several times before it bails out and starts the process all over again.
- removes arbitrary timeout which applied over the whole process and meant some deltas would never manage to be applied (because of large delta size and low network bandwidth).
- makes sure any launched rsync processes always exit and any opened streams consumed and closed.

Most of the improvements are in the two dependencies linked below -- `resumable-request` and `node-docker-delta` -- and this commit merely combines the updated versions of these modules.

Change-Type: minor
Connects-To: #140
Depends-On: https://github.com/resin-io/node-docker-delta/pull/19
Depends-On: https://github.com/resin-io-modules/resumable-request/pull/2
2017-08-09 11:55:22 +03:00
Pablo Carranza Velez
6f87b1db18 Avoid starting apps on startup if device has to reboot due to a configuration change
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-08-02 20:07:13 -03:00
Pablo Carranza Velez
42ac7487e7 When the device is about to reboot or shutdown, close the API server and avoid applying updates
We mark when the device is rebooting and avoid some steps in the update cycle that change the device
state, similarly to when the device is in local mode, to avoid problems with non-atomic operations.
This doesn't solve *all* the potential scenarios of a reboot happening in the middle of an update, but at least
should prevent the case where we start an app container and reboot the device before saving the containerId, potentially
causing a duplicated container issue.

We also correct the API docs to reflect the 202 response when reboot or shutdown are successful.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-27 20:07:24 -03:00
Pablo Carranza Velez
55ad977ede Avoid unhandled errors when in offline mode due to a missing apiEndpoint
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-27 12:46:14 -03:00
Pablo Carranza Velez
f0344ca4be Do not persist the uuid when in offline mode, so that the supervisor tries to provision if it goes out of offline mode
We used to store the uuid which would cause the supervisor to not attempt a provisioning even if offline mode
was turned off. This was to avoid preloaded apps being reloaded constantly leaving multiple containers.

We now avoid persisting the uuid, so that when the supervisor goes out of offline mode it can provision
without the need to wipe out the db. We avoid the problem with preloaded apps by not loading them
if there's apps already stored on the db.

(In the future, apps in the db will only represent target state and we can make preloaded apps be reloaded on every
start, but for now we can't do it as long as we store the containerId on the db - deleting an app on the db
means losing track of its containerId and therefore leaving an orphaned container)

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-27 12:46:14 -03:00
Pablo Carranza Velez
7aedd7062d Update docker-delta to 1.1.1, docker-toolbelt to 3.0.1, docker-progress to 2.6.0 to add support for deltas and overlay2
This makes the Async suffix for docker functions unnecessary. It also allows us to remove dockerode as an
explicit dependency.

Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-27 01:48:35 -03:00
Pablo Carranza Velez
279ab60233 Fix the message shown when docker gives a 500 error when starting a container
The test for an exec format error caused a `err.json.trim` is not a function
error so the message shown didn't relate to what the problem actually was.
This makes the test for the exec format error safer.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-26 10:41:25 -03:00
Pablo Carranza Velez
1790939046 Use webpack to join all modules
This saves around 13MB in the resulting uncompressed docker image.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-11 14:01:16 -07:00
Joe Roberts
087e7c3af0
Deprecate edge device type
Change-type: major
2017-07-05 10:20:26 +01:00
Pablo Carranza Velez
8b2138f744 Fix semver comparison for OS version when determining if the device has deviceApiKey support
The current setup would cause the check to always fail - the consequence is not *that* bad since
the provisioning key still gets overwritten, but it's better to delete it if we can.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-07-04 02:56:31 -07:00
Pablo Carranza Velez
928df5b140 Allow registering the deviceApiKey in a non-compatible OS by making the apiKey equal the deviceApiKey, and add an fsync to all config.json writes
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-06-30 18:00:01 -07:00
Pablo Carranza Velez
597a2c6b65 Remove the undocumented and unused sideload and compose APIs
This allows us to also remove a few npm dependencies and the docker compose binary.

Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-06-26 13:08:52 -07:00
Pablo Carranza Velez
18ca98a2ae Fix provisioning key exchange by passing apikey in the request
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-06-26 07:04:43 -07:00
Pablo Carranza Velez
00b53bd03e When apiEndpoint is not defined, work in offline mode
The supervisor uses an `API_ENDPOINT` environment variable to define what API to register to. Up to now this has been defaulted to `https://api.resin.io`.
(In Resin OS devices this environment variable ultimately comes from config.json).
This commit changes the behavior so that an empty value of that environment variable causes the supervisor to work in "offline mode", i.e. not connected to a remote server.
Basically only preloaded apps and the supervisor API work in this mode.

The config.json `supervisorOfflineMode` field still works for backwards compatibility, but we'll treat it as deprecated and it should be removed eventually.

Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-06-14 12:57:47 -07:00
Pablo Carranza Velez
1e7bdad7a9 Fix mixpanel initialization when not in offline mode
The logic to disable mixpanel initialization in offline mode was inverted :S causing mixpanel
to *only* be initialized when in offline mode.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2017-06-14 12:48:29 -07:00
Joe Roberts
d4e3e45e52
Dependent device DB migrations 2017-06-14 09:27:47 +01:00
Joe Roberts
786874dbb6
Update dependent device DB
Change-type: patch
2017-06-14 09:27:47 +01:00
Petros Angelatos
171460041f
enable SSL when connecting to pubnub
Fixes #451

Connected-to: pubnub/javascript#89
Change-Type: patch
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
2017-06-13 19:13:09 +03:00