v3.4.0

minor: Update API, registry and VPN services

.github/CODEOWNERS

@@ -1,2 +1,2 @@
 # Main repo owners:
-*       @dfunckt @richbayliss
+*       @dfunckt @xginn8 @pdcastro

README.md

@@ -25,42 +25,25 @@ To learn more about openBalena, visit [balena.io/open][open-balena-website].
 - **Built-in VPN**: Access your devices regardless of their network environment
 
 
-## Roadmap
-
-OpenBalena is currently in beta. While fully functional, it lacks features we
-consider important before we can comfortably call it production-ready. During
-this phase, don’t be alarmed if things don’t work as expected just yet (and
-please let us know about any bugs or errors you encounter!). The following
-improvements and new functionality is planned:
-
-- Full documentation
-- Full test suite
-- Simplified deployment
-- Remote host OS updates
-- Support for custom device types
-
-
-## Contributing
-
-Everyone is welcome to contribute to openBalena. There are many different ways
-to get involved apart from submitting pull requests, including helping other
-users on the [forums][forums], reporting or triaging [issues][issue-tracker],
-reviewing and discussing [pull requests][pulls], or just spreading the word.
-
-All of openBalena is hosted on GitHub. Apart from its constituent components,
-which are the [API][open-balena-api], [VPN][open-balena-vpn], [Registry][open-balena-registry],
-[S3 storage service][open-balena-s3], and [Database][open-balena-db], contributions
-are also welcome to its client-side software such as the [balena CLI][balena-cli],
-the [balena SDK][balena-sdk], [balenaOS][balena-os] and [balenaEngine][balena-engine].
-
-
 ## Getting Started
 
-Our [Getting Started][getting-started] guide is the most direct path to getting
+Our [Getting Started guide][getting-started] is the most direct path to getting
 an openBalena installation up and running and successfully deploying your
 application to your device(s).
 
 
+## Compatibility
+
+The current release of openBalena has the following minimum version requirements:
+
+- balenaOS v2.58.3
+- balena CLI v12.38.5
+
+If you are updating from previous openBalena versions, ensure you update the balena
+CLI and reprovision any devices to at least the minimum required versions in order
+for them to be fully compatible with this release, as some features may not work.
+
+
 ## Documentation
 
 While we're still working on the project documentation, please refer to the
@@ -89,6 +72,54 @@ for help, or contribute by answering questions posted by fellow openBalena users
 Please do not use the issue tracker for support-related questions.
 
 
+## Contributing
+
+Everyone is welcome to contribute to openBalena. There are many different ways
+to get involved apart from submitting pull requests, including helping other
+users on the [forums][forums], reporting or triaging [issues][issue-tracker],
+reviewing and discussing [pull requests][pulls], or just spreading the word.
+
+All of openBalena is hosted on GitHub. Apart from its constituent components,
+which are the [API][open-balena-api], [VPN][open-balena-vpn], [Registry][open-balena-registry],
+[S3 storage service][open-balena-s3], and [Database][open-balena-db], contributions
+are also welcome to its client-side software such as the [balena CLI][balena-cli],
+the [balena SDK][balena-sdk], [balenaOS][balena-os] and [balenaEngine][balena-engine].
+
+
+## Roadmap
+
+OpenBalena is currently in beta. While fully functional, it lacks features we
+consider important before we can comfortably call it production-ready. During
+this phase, don’t be alarmed if things don’t work as expected just yet (and
+please let us know about any bugs or errors you encounter!). The following
+improvements and new functionality is planned:
+
+- Full documentation
+- Full test suite
+- Simplified deployment
+- Remote host OS updates
+- Support for custom device types
+
+
+## Differences between openBalena and balenaCloud
+
+| openBalena                                          | balenaCloud                                                                                                                                                                                                                                                             |
+| -----                                               | ----                                                                                                                                                                                                                                                                    |
+| Device updates using full images                    | Device updates using [delta images](https://www.balena.io/docs/learn/deploy/delta/)                                                                                                                                                                                     |
+| Support for a single user                           | Support for [multiple users](https://www.balena.io/docs/learn/manage/account/#application-members)                                                                                                                                                                      |
+| Self-hosted deployment and scaling                  | balena-managed scaling and deployment |
+| Community support via [forums][forums]              | Private support on [paid plans](https://www.balena.io/pricing/)                                                                                                                                           |
+| Deploy via `balena deploy` only                     | Build remotely with native builders using [`balena push`](https://www.balena.io/docs/learn/deploy/deployment/#balena-push) or  [`git push`](https://www.balena.io/docs/learn/deploy/deployment/#git-push) |
+| No support for building via `git push`              | Use the same CI workflow with [`git push`](https://www.balena.io/docs/learn/deploy/deployment/#git-push)                                                                                                  |
+| No public URL support                               | Serve websites directly from device with [public device URLs](https://www.balena.io/docs/learn/manage/actions/#enable-public-device-url)                                                                  |
+| Management via `balena-cli` only                    | Cloud-based device management dashboard                                                                                                                                                                   |
+| Download images from [balena.io][balena-os-website] | Download preconfigured images directly from the dashboard                                                                                                                                                 |
+| No supported remote diagnostics                     | Remote device diagnostics                                                                                                                                                                                 |
+| Supported devices: Raspberry Pi family, the Intel NUC, the NVIDIA Jetson TX2, and the balenaFin | All the devices listed in balena's [reference documentation](https://www.balena.io/docs/reference/hardware/devices/)   |
+
+Additionally, refer back to the [roadmap](#roadmap) above for planned but not yet implemented features.
+
+
 ## License
 
 OpenBalena is licensed under the terms of AGPL v3. See [LICENSE](LICENSE) for details.

VERSION

@@ -1 +1 @@
-2.0.0
+3.4.0

Vagrantfile

@@ -1,12 +1,11 @@
-Vagrant.require_version '>= 2.0.0'
-
-[ 'vagrant-vbguest', 'vagrant-docker-compose' ].each do |p|
-  unless Vagrant.has_plugin?(p)
-    raise "Please install missing plugin: vagrant plugin install #{p}"
-  end
-end
+Vagrant.require_version '>= 2.2.0'
 
 Vagrant.configure('2') do |config|
+  config.vagrant.plugins = [
+    'vagrant-vbguest',
+    'vagrant-docker-compose'
+  ]
+
   config.vm.define 'openbalena'
   config.vm.hostname = 'openbalena-vagrant'
   config.vm.box = 'bento/ubuntu-18.04'

compose/mdns.yml

@@ -22,7 +22,7 @@ services:
         # the resin backend (eg. that for BALENA_ROOT_CA if present).
         MDNS_TLD: ${OPENBALENA_HOST_NAME}
         # List of subdomains to advertise. This must include all required hosts.
-        MDNS_SUBDOMAINS: '["api", "db", "registry", "s3", "vpn"]'
+        MDNS_SUBDOMAINS: '["api", "db", "registry", "s3", "tunnel", "vpn"]'
         # The expectation is the DBus socket to use is always at the following location.
         DBUS_SESSION_BUS_ADDRESS: "unix:path=/host/run/dbus/system_bus_socket"
         # Selects the interface used for incoming connections from the wider subnet.

compose/services.yml

@@ -32,7 +32,7 @@ services:
       HOST: api.${OPENBALENA_HOST_NAME}
       IMAGE_MAKER_URL: img.${OPENBALENA_HOST_NAME}
       IMAGE_STORAGE_BUCKET: resin-production-img-cloudformation
-      IMAGE_STORAGE_PREFIX: resinos
+      IMAGE_STORAGE_PREFIX: images
       IMAGE_STORAGE_ENDPOINT: s3.amazonaws.com
       JSON_WEB_TOKEN_EXPIRY_MINUTES: 10080
       JSON_WEB_TOKEN_SECRET: ${OPENBALENA_JWT_SECRET}
@@ -85,6 +85,7 @@ services:
       REGISTRY2_S3_SECRET: ${OPENBALENA_S3_SECRET_KEY}
       REGISTRY2_SECRETKEY: ${OPENBALENA_REGISTRY_SECRET_KEY}
       REGISTRY2_STORAGEPATH: /data
+      REGISTRY2_DISABLE_REDIRECT: "false"
 
   vpn:
     extends:
@@ -154,9 +155,9 @@ services:
     ports:
       - "80:80"
       - "443:443"
-      - "3128:3128"
     expose:
       - "222"
+      - "3128"
       - "5432"
       - "6379"
     networks:
@@ -168,6 +169,7 @@ services:
           - db.${OPENBALENA_HOST_NAME}
           - s3.${OPENBALENA_HOST_NAME}
           - redis.${OPENBALENA_HOST_NAME}
+          - tunnel.${OPENBALENA_HOST_NAME}
     environment:
       BALENA_HAPROXY_CRT: ${OPENBALENA_ROOT_CRT}
       BALENA_HAPROXY_KEY: ${OPENBALENA_ROOT_KEY}
@@ -183,5 +185,5 @@ services:
       - cert-provider:/usr/src/app/certs
     environment:
       ACTIVE: ${OPENBALENA_ACME_CERT_ENABLED}
-      DOMAINS: "api.${OPENBALENA_HOST_NAME},registry.${OPENBALENA_HOST_NAME},s3.${OPENBALENA_HOST_NAME},vpn.${OPENBALENA_HOST_NAME}"
+      DOMAINS: "api.${OPENBALENA_HOST_NAME},registry.${OPENBALENA_HOST_NAME},s3.${OPENBALENA_HOST_NAME},vpn.${OPENBALENA_HOST_NAME},tunnel.${OPENBALENA_HOST_NAME}"
       OUTPUT_PEM: /certs/open-balena.pem

compose/versions

@@ -1,6 +1,6 @@
-export OPENBALENA_API_VERSION_TAG=v0.19.5
-export OPENBALENA_DB_VERSION_TAG=v2.0.3
-export OPENBALENA_MDNS_PUBLISHER_VERSION_TAG=v1.6.2
-export OPENBALENA_REGISTRY_VERSION_TAG=v2.11.1
-export OPENBALENA_S3_VERSION_TAG=v2.8.5
-export OPENBALENA_VPN_VERSION_TAG=v8.10.0
+export OPENBALENA_API_VERSION_TAG=v0.139.0
+export OPENBALENA_DB_VERSION_TAG=v4.1.0
+export OPENBALENA_MDNS_PUBLISHER_VERSION_TAG=v1.9.2
+export OPENBALENA_REGISTRY_VERSION_TAG=v2.16.1
+export OPENBALENA_S3_VERSION_TAG=v2.9.9
+export OPENBALENA_VPN_VERSION_TAG=v9.17.11

src/cert-provider/Dockerfile

@@ -6,9 +6,11 @@ VOLUME [ "/usr/src/app/certs" ]
 
 RUN apk add --update bash curl git openssl ncurses socat
 
+# from https://github.com/Neilpang/acme.sh/releases/tag/2.8.5
 RUN git clone https://github.com/Neilpang/acme.sh.git && \
     cd acme.sh && \
-    git checkout 08357e3cb0d80c84bdaf3e42ce0e439665387f57 . && \
+    git fetch && git fetch --tags && \
+    git checkout 2.8.5 . && \
     ./acme.sh --install  \
     --cert-home /usr/src/app/certs
 

src/cert-provider/cert-provider.sh

@@ -1,4 +1,4 @@
-#!/bin/bash
+#!/usr/bin/env bash
 
 # the acme.sh client script, installed via Git in the Dockerfile...
 ACME_BIN="$(realpath ~/.acme.sh/acme.sh)"
@@ -45,14 +45,20 @@ retryWithDelay() {
     DELAY=${3:-5}
 
     local ATTEMPT=0
-    while [ $RETRIES -gt $ATTEMPT ]; do
-        let "ATTEMPT++"
+    while [ "$RETRIES" -gt "$ATTEMPT" ]; do
+        (( ATTEMPT++ ))
+        logInfo "($ATTEMPT/$RETRIES) Connecting..."
         if $1; then
+            logInfo "($ATTEMPT/$RETRIES) Success!"
             return $?
         fi
 
-        echo "($ATTEMPT/$RETRIES) Retrying in ${DELAY} seconds..."
-        sleep $DELAY
+        if [ "$RETRIES" -gt "$ATTEMPT" ]; then
+            logInfo "($ATTEMPT/$RETRIES) Failed. Retrying in ${DELAY} seconds..."
+            sleep "$DELAY"
+        else
+            logInfo "($ATTEMPT/$RETRIES) Failed!"
+        fi
     done
 
     return 1
@@ -62,7 +68,7 @@ waitForOnline() {
     ADDRESS="${1,,}"
 
     logInfo "Waiting for ${ADDRESS} to be available via HTTP..."
-    retryWithDelay "curl --output /dev/null --silent --head --fail http://${ADDRESS}" 6 5
+    retryWithDelay "curl --output /dev/null --silent --head --fail --max-time 5 http://${ADDRESS}"
 }
 
 isUsingStagingCert() {
@@ -167,7 +173,10 @@ acquireCertificate() {
 
 pre-flight || logErrorAndStop "Unable to continue due to misconfiguration. See errors above."
 
-waitForOnline "${ACME_DOMAINS[0]}" || logErrorAndStop "Unable to access ${ACME_DOMAINS[0]} on port 80. This is needed for certificate validation."
+while ! waitForOnline "${ACME_DOMAINS[0]}"; do
+    logInfo "Unable to access ${ACME_DOMAINS[0]} on port 80. This is needed for certificate validation. Retrying in 30 seconds..."
+    sleep 30
+done
 
 if ! lastAcquiredCertFor "production"; then
     acquireCertificate "staging" || logErrorAndStop "Unable to acquire a staging certificate."

src/haproxy/haproxy.cfg

@@ -2,9 +2,9 @@ global
   tune.ssl.default-dh-param 1024
 
 defaults
-  timeout connect 5000
-  timeout client 50000
-  timeout server 50000
+  timeout connect 5s
+  timeout client 50s
+  timeout server 50s
 
 frontend http-in
   mode http
@@ -34,6 +34,10 @@ frontend ssl-in
   tcp-request content accept if { req.ssl_hello_type 1 }
 
   acl is_ssl req.ssl_ver 2:3.4
+
+  acl host_tunnel req_ssl_sni -i "tunnel.${HAPROXY_HOSTNAME}"
+  use_backend redirect-to-tunnel-in if host_tunnel
+
   use_backend redirect-to-https-in if is_ssl
   use_backend vpn-devices if !is_ssl
 
@@ -42,6 +46,11 @@ backend redirect-to-https-in
   balance roundrobin
   server localhost 127.0.0.1:444 send-proxy-v2
 
+backend redirect-to-tunnel-in
+  mode tcp
+  balance roundrobin
+  server localhost 127.0.0.1:3129
+
 frontend https-in
   mode http
   option forwardfor
@@ -118,3 +127,8 @@ listen vpn-tunnel
   mode tcp
   bind *:3128
   server balena_vpn vpn:3128 check port 3128
+
+listen vpn-tunnel-tls
+  mode tcp
+  bind *:3129 ssl crt /etc/ssl/private/open-balena.pem
+  server balena_vpn vpn:3128 check port 3128