From 17886c51aa5dce165d6b202cce71a85fb22ab2f6 Mon Sep 17 00:00:00 2001 From: Brian Warner Date: Tue, 22 Mar 2016 18:18:49 -0700 Subject: [PATCH] add docs/desert-island.rst refs ticket:2752 --- docs/desert-island.rst | 168 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 docs/desert-island.rst diff --git a/docs/desert-island.rst b/docs/desert-island.rst new file mode 100644 index 000000000..d0ccab29f --- /dev/null +++ b/docs/desert-island.rst @@ -0,0 +1,168 @@ +How To Build Tahoe-LAFS On A Desert Island +========================================== + +(or an airplane, or anywhere else without internet connectivity) + +Here's the story: you leave for the airport in 10 minutes, you know you want +to do some Tahoe hacking on the flight. What can you grab right now that will +let you build the necessary dependencies later, when you are offline? + +Pip can help, with a technique described in the pip documentation +https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages . + +Before you get shipwrecked (or leave the internet for a while), do this from +your tahoe source tree: + +* `pip download --dest tahoe-deps .` + +That will create a directory named "tahoe-deps", and download everything that +the current project (".", i.e. tahoe) needs. It will fetch wheels if +available, otherwise it will fetch tarballs. It will not compile anything. + +Later, on the plane, do this (in an active virtualenv): + +* `pip install --no-index --find-links=tahoe-deps --editable .` + +That tells pip to not try to contact PyPI (--no-index) and to use the +tarballs and wheels in `tahoe-deps/` instead. That will compile anything +necessary, create (and cache) wheels, and install them. + +If you need to rebuild the virtualenv for whatever reason, run the "pip +install" command again: it will re-use the cached wheels and skip the compile +step. + +Compiling Ahead Of Time +----------------------- + +If you want to save some battery on the flight, you can compile the wheels +ahead of time. Just do the install step before you go offline. The wheels +will be cached as a side-effect. Later, on the plane, you can populate a new +virtualenv with the same `pip install` command above, and it will use the +cached wheels instead of recompiling them. + +The pip wheel cache +------------------- + +Modern versions of pip and setuptools will, by default, cache both their HTTP +downloads and their generated wheels. When pip is asked to install a package, +it will first check with PyPI. If the PyPI index says it needs to download a +newer version, but it can find a copy of the tarball/zipball/wheel in the +HTTP cache, it will not actually download anything. Then it tries to build a +wheel: if it already has one in the wheel cache (downloaded or built +earlier), it will not actually build anything. + +If it cannot contact PyPI, it will fail. The `--no-index` above is to tell it +to skip the PyPI step, but that leaves it with no source of packages. The +`--find-links=` argument is what provides an alternate source of packages. + +The HTTP and wheel caches are not single flat directories: they use a +hierarchy of subdirectories, named after a hash of the URL or name of the +object being stored (this is to avoid filesystem limitations on the size of a +directory). As a result, the wheel cache is not suitable for use as a +`--find-links=` target (but see below). + +There is a command named `pip wheel` which only creates wheels (and stores +them in `--wheel-dir=`, which defaults to the current directory). This +command does not populate the wheel cache: it reads from (and writes to) the +HTTP cache, and reads from the wheel cache, but will only save the generated +wheels into the directory you specify with `--wheel-dir=`. It does not also +write them to the cache. + +Where Does The Cache Live? +-------------------------- + +Pip's cache location depends upon the platform. On linux, it defaults to +~/.cache/pip/ (both http/ and wheels/). On OS-X (homebrew), it uses +~/Library/Caches/pip/ . On Windows, try ~\AppData\Local\pip\cache . + +The location can be overridden by `pip.conf`. Look for the "wheel-dir", +"cache-dir", and "find-links" options. + +How Can I Tell If It's Using The Cache? +--------------------------------------- + +When "pip install" has to download a source tarball (and build a wheel), it +will say things like: + +* Collecting zfec +* Downloading zfec-1.4.24.tar.gz (175kB) +* Building wheels for collected packages: zfec +* Running setup.py bdist_wheel for zfec ... done +* Stored in directory: $CACHEDIR +* Successfully built zfec +* Installing collected packages: zfec +* Successfully installed zfec-1.4.24 + +When "pip install" can use a cached downloaded tarball, but does not have a +cached wheel, it will say: + +* Collecting zfec +* Using cached zfec-1.4.24.tar.gz +* Building wheels for collected packages: zfec +* Running setup.py bdist_wheel for zfec ... done +* Stored in directory: $CACHEDIR +* Successfully built zfec +* Installing collected packages: zfec +* Successfully installed zfec-1.4.24 + +When "pip install" can use a cached wheel, it will just say: + +* Collecting zfec +* Installed collected packages: zfec +* Successfully installed zfec-1.4.24 + +Many packages publish pre-built wheels next to their source tarballs. This is +common for non-platform-specific (pure-python) packages. It is also common +for them to provide pre-compiled windows and OS-X wheel, so users do not have +to have a compiler installed (pre-compiled Linux wheels are not common, +because there are too many platform variations). When "pip install" can use a +downloaded wheel like this, it will say: + +* Collecting six +* Downloading six-1.10.0-py2.py3-none-any.whl +* Installing collected packages: six +* Successfully installed six-1.10.0 + +Note that older versions of pip do not always use wheels, or the cache. Pip +8.0.0 or newer should be ok. The version of setuptools may also be +significant. + +Another Approach +---------------- + +An alternate approach is to set your `pip.conf` to install wheels into the +same directory that it will search for links, and use `pip wheel` to add +wheels to the cache. The `pip.conf` will look like: + + [global] + wheel-dir = ~/.pip/wheels + find-links = ~/.pip/wheels + +(see https://pip.pypa.io/en/stable/user_guide/#configuration to find out +where your `pip.conf` lives, but `~/.pip/pip.conf` probably works) + +While online, you populate the wheel-dir (from a tahoe source tree) with: + +* `pip wheel .` + +That compiles everything, so it may take a little while. Note that you can +also add specific packages (and their dependencies) any time you like, with +something like `pip wheel zfec`. + +Later, you do the offline install (in a virtualenv) with just: + +* `pip install --no-index --editable .` + +If/when you have network access, omit the `--no-index` and it will check with +PyPI for the most recent versions (and still use the stashed wheels if +appropriate). + +The upside is that the only extra `pip install` argument is `--no-index`, and +you don't need to remember the `--find-links` or `--dest` arguments. + +The downside of this approach is that `pip install` does not populate the +wheel-dir (it populates the normal wheel cache, but not ~/.pip/wheels). Only +an explicit `pip wheel` will populate ~/.pip/wheels. So if you do a `pip +install` (but not a `pip wheel`), then go offline, a second `pip install +--no-index` may fail: the wheels it needs may be somewhere in the +wheel-cache, but not in the `--find-links=` directory.