Commit Graph

1520 Commits

Author SHA1 Message Date
Pablo Carranza Velez
d3e98eab11 DeviceConfig: implement a module to manage device configuration, including config.txt
This model allows modifying config.txt on raspberry pi devices, as well as logging to display, bandwidth control variables and other supervisor
configuration settings. Configuration values are read from the underlying OS and the supervisor configuration where appropriate (i.e. the Config object), instead of storing the current state
in the database. This means that the supervisor will always use the real values to determine if changes have to be made.

This fixes several issues with config.txt, as the current values are now read from the file, and can be reported on the supervisor's first run (which will be implemented in APIBinder).

It also now treats dtoverlay and dtparam values as a JSON array without the enclosing brackets, for instance:

```
RESIN_HOST_CONFIG_dtparam="audio=on","spi=on"
```

Will produce the following lines in config.txt:

```
dtparam=audio=on
dtparam=spi=on
```

Changelog-Entry: Implement inference of device configuration. Allow array values for dtoverlay and dtparam.
Change-Type: major
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
2953b745ce Logger: implement a module that handles all logging to pubnub
This module can also send logs for dependent devices (by passing a specific channel to the "log" function).

The log types are also moved to a separate module to be used by modules that perform logging.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
93832d6540 network: implement a module to get IP addresses and check network connectivity
This module now uses the native node `os.networkInterfaces()` to retrieve the addresses,
instead of the gosuper endpoint.

We also add the very simple "blink" library that is also used by the Supervisor API.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
f77d3e1563 DeviceState: implement a module to manage the device's target and current state
This module will take care of applying the target state for the device and reporting its current state.
The state itself is handled by two other modules, ApplicationManager and DeviceConfig. The former will take care of running applications (including the dependent ones
via its Proxyvisor), and the latter will take care of device configuration like config.txt and supervisor configuration variables.

The way state is applied differs radically from the previous approach: the old application.coffee had a big `update` function that took all of the steps from fetching the target state
to running the containers. DeviceState, instead, does an iterative process through `triggerApplyTarget` of inferring the next steps to perform towards the target state, by looking at the current state and asking the ApplicationManager and DeviceConfig for
the next steps. It then applies the next steps and every time a step is completed, it schedules another round of inferring and applying the next steps.

Special care is taken to ensure `applyTarget` is not called simultaneously more than once.

This commit also adds a "device" module to handle reboot and shutdown, and moves gosuper calls to a separate module.

The module also uses a "network" module to manage network-related parts of the device's current state: IP addresses and the connectivity check.

The module implements a "normaliseLegacy" function that allows a migration from the models from older versions of the supervisor to the multicontainer models,
so that in case of a supervisor update we can have minimal downtime and bandwidth consumption when updating to the multicontainer supervisor - this migration allows
us to avoid cleaning up images, and also allows migrating the contents of the old /data for the app.

Changelog-Entry: Infer the current state of the device when applying the target state
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
0dc9fea4d3 DB, Config: Implement modules to interact with the sqlite DB and to manage configuration
These modules allow managing the models in the sqlite database and the configuration for the supervisor.

The database will now have a schema version, and the supervisor will normalize any legacy data when migrating
from an older schema (i.e. when doing a supervisor update). This will make model changes cleaner.
If a migration is needed, the DB initialization will return "true" and store the legacy data in a legacyData table. Once the supervisor finishes migrating the data,
it calls `db.finishMigration` to mark the migration complete and clear the legacyData table.

Changes in the models:
* The database implements the tables for multicontainer applications that now have services, networks and volumes as in a docker compose file.
* Dependent apps and devices now have separate tables to store their target states.
* The deviceConfig table now only stores target values, as the current ones will be inferred from the state of the device.
* We keep a table for images as we have no way to label them in docker storage, so we need to keep our own track of what images are relevant for the supervisor.

The Config object allows transparent management of configuration values, mainly through `get`, `getMany` and `set` functions. The values can be stored in config.json or
the database, and this is managed with a schema definition that also defines whether values are mutable and whether they have default values.

Some configuration values are of the "func" type, which means that instead of corresponding to a config.json or database key, they result from a helper function
that aggregates other configuration values or gets the value from other sources, like OS version and supervisor version.

Writes to config.json are atomic if a path to the file via /mnt/root can be found. We keep a write-through cache of the file to avoid unnecessary IO.

Changelog-Entry: Implement the multicontainer app models, and change the supervisor configuration management to avoid duplication between fields in config.json and fields in the internal database
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
23f81c28f5 EventTracker: add a module to track mixpanel events
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
Pablo Carranza Velez
60a4cccfd2 Supervisor: Implement a Supervisor class with a SupervisorAPI
This will be the top level object in the multicontainer supervisor, using the following objects
to perform its duties:

* A DB object to manage the sqlite database models
* A Config object to manage configuration in sqlite and config.json
* An EventTracker to track events and send them to mixpanel
* A DeviceState object to manage the device state, including containers, device configuration and dependent devices
* An APIBinder object to manage all interactions with the Resin API
* The SupervisorAPI, implemented here, which exposes functionality from the other objects over an HTTP API with apikey authentication.

We also include an iptables module that the SupervisorAPI will use to only allow traffic from certain interfaces.

Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-03-06 10:32:27 -08:00
resin-io-versionbot[bot]
b397c998dd
Auto-merge for PR #565 via VersionBot
Update locking docs update, again
2018-03-05 19:31:13 +00:00
resin-io-versionbot[bot]
b0f2335b41 v6.6.9 2018-03-05 19:16:34 +00:00
zwalchuk
d931bf217a Update update-locking.md
Test to see if this updates commit message

Change-type: patch
Signed-off-by: Zach Walchuk zach@resin.io
2018-03-05 07:09:42 -08:00
zwalchuk
2ce831420f Small updates for standardization
A few small updates to standardize terminology and header with the rest of the docs.
2018-03-05 07:09:42 -08:00
resin-io-versionbot[bot]
cea1a4b781
Auto-merge for PR #561 via VersionBot
Allow truthy values for deltas and lock override (i.e. the string 'tr…
2018-02-27 18:42:22 +00:00
resin-io-versionbot[bot]
b6fc45b671 v6.6.8 2018-02-27 18:27:22 +00:00
Pablo Carranza Velez
f2d5a59727 Allow truthy values for deltas and lock override (i.e. the string 'true' besides '1')
We had previously done this for all the other configuration variables, but for some reason we had missed these two.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-02-27 09:27:45 -08:00
resin-io-versionbot[bot]
cb450a7f59
Auto-merge for PR #560 via VersionBot
Since armel builds are disabled, do not pull an armel node base image…
2018-02-27 17:25:26 +00:00
resin-io-versionbot[bot]
590c67333e v6.6.7 2018-02-27 17:11:11 +00:00
Pablo Carranza Velez
c463f7fa5b Since armel builds are disabled, do not pull an armel node base image, and ensure we never deploy an armel supervisor
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-02-27 08:40:43 -08:00
resin-io-versionbot[bot]
580873d069
Auto-merge for PR #558 via VersionBot
Update docker-delta to 2.0.4
2018-02-27 01:26:52 +00:00
resin-io-versionbot[bot]
7b9a87c347 v6.6.6 2018-02-27 01:11:47 +00:00
Akis Kesoglou
76ac7da1d6 Update docker-delta to 2.0.4
This brings in a fix for an edge case where rsync would exit before we had a chance to register event listeners.

Change-Type: patch
2018-02-23 11:26:32 +02:00
resin-io-versionbot[bot]
28258e3bbf
Auto-merge for PR #556 via VersionBot
circle.yml: escape branch name to ensure we always have a valid tag
2018-02-21 22:35:33 +00:00
resin-io-versionbot[bot]
9bbeea8a72 v6.6.5 2018-02-21 20:54:46 +00:00
Pablo Carranza Velez
7b404d0999 circle.yml: escape branch name to ensure we always have a valid tag
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-02-20 07:35:40 -08:00
resin-io-versionbot[bot]
650d2086f0
Auto-merge for PR #555 via VersionBot
circle.yml: Do not push images to dockerhub if building without a docker password
2018-02-20 02:42:06 +00:00
resin-io-versionbot[bot]
2361137a98 v6.6.4 2018-02-20 02:20:57 +00:00
Pablo Carranza Velez
7eacd93321 circle.yml: Do not push images to dockerhub if building without a docker password
This will allow us to build contributor PRs without passing secrets.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-02-16 14:16:22 -08:00
resin-io-versionbot[bot]
5023030c69
Auto-merge for PR #551 via VersionBot
Update resumable-request to v2.0
2018-02-06 00:28:01 +00:00
resin-io-versionbot[bot]
50e2110ba8 v6.6.3 2018-02-06 00:12:34 +00:00
Akis Kesoglou
dc69917b5a Update resumable-request to v2.0
Turned out that disk I/O can be the bottleneck when applying deltas on some devices. When the disk can’t keep up and consume the downloaded delta, there’s memory bloat due to buffering.

The updated version provides far better reliability when the device is under load and pretty much constant memory consumption with any number of concurrent deltas.

Change-Type: patch
2018-02-05 10:59:56 +02:00
resin-io-versionbot[bot]
79277fea68
Auto-merge for PR #549 via VersionBot
Use i386-nlp for supervisor releases for quark boards
2018-01-24 19:52:48 +00:00
resin-io-versionbot[bot]
bb422b1464 v6.6.2 2018-01-24 19:30:24 +00:00
Pablo Carranza Velez
03c58a2bfa Use i386-nlp for supervisor releases for quark boards
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-24 09:08:37 -08:00
resin-io-versionbot[bot]
9d88be5dcf
Auto-merge for PR #526 via VersionBot
Improve caching in dindctl build and update README
2018-01-19 15:01:56 +00:00
resin-io-versionbot[bot]
191bc733f9 v6.6.1 2018-01-19 14:47:16 +00:00
Pablo Carranza Velez
556114659e README.md: Capitalization and other fixes
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-19 11:18:41 -03:00
Pablo Carranza Velez
45b2fcef1b Improve caching in dindctl build and update README
The change in dindctl build uses automation/build.sh to pull intermediate images to improve caching. Sometimes it works to avoid
a full base image build...

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-19 11:18:41 -03:00
resin-io-versionbot[bot]
7e04b4aec0
Auto-merge for PR #547 via VersionBot
Implement an API for proxy and hostname configuration, and centralize…
2018-01-18 22:28:24 +00:00
resin-io-versionbot[bot]
4b28c11a67 v6.6.0 2018-01-18 22:13:23 +00:00
Pablo Carranza Velez
0e7bb77c41 docs: clarify that open connections will not be closed when configuring a proxy, and change all proxy API examples to use uuid
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-18 19:07:00 -03:00
Pablo Carranza Velez
0960d947ed Add documentation for the /v1/device/host-config endpoint
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-18 18:28:05 -03:00
Pablo Carranza Velez
cff789ebfa Implement an API for proxy and hostname configuration, and centralize management of config.json
Change-Type: minor
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-17 14:05:40 -03:00
resin-io-versionbot[bot]
122b55fbe5
Auto-merge for PR #544 via VersionBot
Fix saving deviceApiKey to the DB (to fix the RESIN_API_KEY env var)
2018-01-12 15:59:32 +00:00
resin-io-versionbot[bot]
c11cae4777 v6.5.9 2018-01-12 15:43:08 +00:00
Pablo Carranza Velez
968d3ce1e3 Fix saving deviceApiKey to the DB (to fix the RESIN_API_KEY env var) when updating from some older supervisors
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-12 11:53:54 -03:00
resin-io-versionbot[bot]
17017936ce
Auto-merge for PR #543 via VersionBot
Update docker-toolbelt to v3.2.1 to fix a bug in deltas
2018-01-10 18:06:53 +00:00
resin-io-versionbot[bot]
2a02938fa4 v6.5.8 2018-01-10 17:51:59 +00:00
Pablo Carranza Velez
a81cf94660 Update docker-toolbelt to v3.2.1 to fix a bug in deltas
The problem was caused by the build picking up 3.2.0 which had a bug, causing an `TypeError: l is not a function`.

Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-10 13:59:50 -03:00
resin-io-versionbot[bot]
e62da15859
Auto-merge for PR #541 via VersionBot
Avoid problems with null app names or invalid container names when migrating from old supervisors
2018-01-05 15:54:07 +00:00
resin-io-versionbot[bot]
2fb86421ab v6.5.7 2018-01-05 15:39:11 +00:00
Pablo Carranza Velez
8fac5c0e32 Avoid problems with null app names or invalid container names when migrating from old supervisors
Change-Type: patch
Signed-off-by: Pablo Carranza Velez <pablo@resin.io>
2018-01-05 11:10:41 -03:00