This fixes occasional timeouts in rsync after applying v2 deltas, which
cause unnecessary fallbacks to a regular pull.
This change also removes the need for the mkfifo native dependency (since
docker-delta 2.2.2 stops using it).
Change-type: patch
Signed-off-by: Pablo Carranza Velez <pablo@balena.io>
Stability improvements;
* Printing of unsupported compose fields
* Added a lot of tests
* All compose configuration has a default value, enabling better
comparison
Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
Running mocha on the coffee and ts sources resulted in unreliable
sourcemaps, and breakpoints not working, among other things. To solve
this, this commit adds package.json tasks which will build both the
coffeescript and typescript files with their sourcemaps and places it
into a build directory. It also copies over other dependent files, such
as test data and migrations.
Another task adds a mocha debugging command, which can be connected to
using a node debugger.
Another tasks runs mocha without a debugging interface, but also without
running linters etc first. This enables the tests to be run much faster.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
Also had to change config module to bind `.this` value, due to
differences in setup.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
This commit abstracts all of the boot config code out of the
device-config module, ready to extend with different config backends.
Change-type: patch
Signed-off-by: Cameron Diver <cameron@resin.io>
Resin’s delta server supports Balena deltas as version 3 deltas. This commit adds support for triggering delta generation for Balena deltas, and applying them locally to the device via a simple pull.
The delta version to use when updating has been abstracted away as an env var that is user-defined. The default value is still instructing use of rsync deltas (v2).
Change-Type: minor
We add a bunch of additional unit tests, and also a coverage report using istanbul.
The tests are not meant to cover everything, but they're a first attempt at having *some* unit testing
on the supervisor. There's much to improve but hopefully it helps catch obvious errors.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Add webpack config and dependencies to have typescript built, and also
convert src/lib/validation.coffee to typescript.
In this conversion I also added a lot of debugging which should help the
upcoming local mode development.
Change-type: minor
Signed-off-by: Cameron Diver <cameron@resin.io>
* Ensure commit is only reported when update has finished
* Change default delay between actions to 100ms
* Fix envArrayToObject for cases where the env var has an equal sign
* Use shell-quote to properly parse string command and entrypoint
* Fix preloading with a legacy apps.json
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Also several bugfixes:
* Fix VPN control, logging in deviceConfig, and action executors in proxyvisor
* Fix bug in calculation of dependencies due to fields still using snake_case
* Fix snake_case in a migration, and remove unused lib/migration.coffee
* In healthcheck, count deviceState as healthy when a fetch is in progress (as in the non-multicontainer supervisor)
* Set always as default restart policy
* Fix healthcheck, stop_grace_period and mem_limit
* Lint and reduce some cyclomatic complexities
* Namespace volumes and networks by appId, switch default network name to 'default', fix dependencies in networks and volumes, fix duplicated kill steps, fix fat arrow on provisioning
* Check that supervisor network is okay every time we're applying target state
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Also includes various improvements and bugfixes to services and the migration from legacy /data to volumes.
The switch ti migrations involves a dirty hack for webpack to properly resolve the paths to the migrations js files - it uses an expression
that webpack can't resolve, so we hardcode it to a value and use the ContextReplacementPlugin to make that value resolve to the migrations folder.
The downsides to this approach are:
- a change in knex code would break this
- the migration code is added twice to the supervisor image: once in the migrations folder (because knex needs to loop through the directory to find the files),
and once inside app.js (because I can't make webpack treat them as external)
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Turned out that disk I/O can be the bottleneck when applying deltas on some devices. When the disk can’t keep up and consume the downloaded delta, there’s memory bloat due to buffering.
The updated version provides far better reliability when the device is under load and pretty much constant memory consumption with any number of concurrent deltas.
Change-Type: patch
The problem was caused by the build picking up 3.2.0 which had a bug, causing an `TypeError: l is not a function`.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
We've been using UglifyJS 0.4.6 (the webpack default) so far, but this doesn't support ES6 and some dependency
updates are starting to cause builds to break (e.g. https://github.com/request/request/issues/2772, which also happens to break
my builds in the multicontainer branch).
Here we switch to the latest uglifyjs-webpack-plugin which is designed for ES2015 support.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
When sending events to mixpanel, we now use an explicit whitelist for the properties sent with the event, to avoid accidental leakage of any sensitive information.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
- Updates resumable-request to 1.0.1
- Updates docker-progress to 2.0.3
- Removes `DEFAULT_DELTA_APPLY_TIMEOUT`; it’s not needed anymore, docker-delta reliably tracks rsync.
- Properly end the update when applying the delta results in an error.
Change-Type: patch
This handy tool uses the resin-sync module to rsync javascript changes into the running container
on a device in the local network. It allows rapid iterations when testing the supervisor on a real device.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2.6.2 has a serious bug which causes the js file to not be included in the published package.
Webpack gave us an unexpected workaround because it will add the .coffee file anyways, but we should
still update to the fixed version.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Applying a delta update consists of two parts:
1. The request to the delta server for the delta payload (an rsync batch file, plus some prepended Docker metadata). The response is a redirect to a URL that contains the delta (currently S3).
2. The request for the actual download of the delta. The response is streamed directly to rsync, which applies it onto the mounted root filesystem of the final image.
The first step may take a while as it may trigger the generation of the delta if the request is the first one for this combination of src/dest image and the images are large. If the request times out, either because of the delta server taking too long to respond or bad network, the Supervisor automatically schedules a retry to be performed after a while.
Currently, similar behaviour applies to the second step as well -- if the request fails, we immediately bail out and the Supervisor schedules a retry of the whole process (i.e. from step 1). But in this case it means we might have downloaded and applied some or most of the delta when a socket timeout occurs causing us to start all over again, wasting time and bandwidth.
This commit splits the process into the two discreet steps and improves the behaviour on the second step. Specifically:
- makes the Supervisor try to resume the delta download request several times before it bails out and starts the process all over again.
- removes arbitrary timeout which applied over the whole process and meant some deltas would never manage to be applied (because of large delta size and low network bandwidth).
- makes sure any launched rsync processes always exit and any opened streams consumed and closed.
Most of the improvements are in the two dependencies linked below -- `resumable-request` and `node-docker-delta` -- and this commit merely combines the updated versions of these modules.
Change-Type: minor
Connects-To: #140
Depends-On: https://github.com/resin-io/node-docker-delta/pull/19
Depends-On: https://github.com/resin-io-modules/resumable-request/pull/2
This makes the Async suffix for docker functions unnecessary. It also allows us to remove dockerode as an
explicit dependency.
Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
This allows us to also remove a few npm dependencies and the docker compose binary.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
This avoids problems when updating the supervisor on an older OS, where the VPN and other
host services still require config.json to have an apiKey field to authenticate.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Hand over authentication credentials to the docker engine
Fetch an access token from the API if possible and hand it over to the delta server
Change-Type: minor
Signed-off-by: Andreas Fitzek <andreas@resin.io>
Also remove versionist.conf.js as it's no longer necessary (versionist now
defaults to its internal versionist.conf).
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
Resin OS 2.X removes the use of compressed modules, which was the initial
motivation for us to bind mount kmod into user containers (as Debian distros
don't include support for compressed modules).
This is a breaking change, but we still keep bind mounting on devices that are
on 1.X to ensure we don't break apps currently relying on the feature.
Implementation note: some functions in device.coffee have been refactored to
extract (DRY) a memoization procedure for Promise-returning functions.
`device.getOSVersion()` now also memoizes its result.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
We use the default versionist configuration.
Changelog has been edited to match the versionist format as much as possible.
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
* Use appId in dependent app assets tar path, and only create the tar if it doesn't exist
* Support AUFS by upgrading node-docker-delta to 1.0.0 and docker-toolbelt to 1.3.0
busybox's init uses /etc/inittab for configuration just like sysvinit,
however it doesn't use any runlevels.
the tty part of inittab is appended to "/dev/", and it becomes connected
to the stdout of the spawned process
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
This build strategy lends itself to how Rockerfiles work. In the build
Dockerfile all the build utilities (e.g gcc, python) are installed and
run the build process to produce some build artifacts. There are two
build Dockerfiles, one for the nodejs part and one for the golang part.
The build artifacts of these are combined into the runtime Dockerfile.
For all this to work there is some minimal glue implemented in the
Makefile.
Part of this commit is a switch of the base image the runtime is based
on to the minimal OpenEmbedded one produced by #198
Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
- Validate the options in the YAML file
- Define bind mounts for each service as in Resin apps
- Keep the modified compose file inside the supervisor's /data folder
- Fix error reporting in the first stage of "up"
