mm/jupyterhub-deploy-docker
1
0
Fork 0
Code Pull Requests Releases Activity

Merge pull request #23 from willingc/user-docs

Browse Source 
[WIP] Edit README  to be more user friendly
This commit is contained in:
Min RK 2017-08-02 22:22:41 +02:00 committed by GitHub
commit a671e520fd
3 changed files with 193 additions and 97 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

264
README.md
View File

@ -1,102 +1,171 @@
**[Technical Overview](#technical-overview)** |
**[Prerequisites](#prerequisites)** |
**[Authenticator setup](#authenticator-setup)** |
**[Build the JupyterHub Docker image](#build-the-jupyterhub-docker-image)** |
**[Spawner: Prepare the Jupyter Notebook Image](#spawner-prepare-the-jupyter-notebook-image)** |
**[Run JupyterHub](#run-jupyterhub)** |
**[Behind the scenes](#behind-the-scenes)** |
**[FAQ](#faq)**
# jupyterhub-deploy-docker # jupyterhub-deploy-docker
This repository provides a reference deployment of [JupyterHub](https://github.com/jupyter/jupyterhub), a multi-user [Jupyter Notebook](http://jupyter.org/) environment, on a **single host** using [Docker](https://docs.docker.com). **jupyterhub-deploy-docker** provides a reference
deployment of [JupyterHub](https://github.com/jupyter/jupyterhub), a
multi-user [Jupyter Notebook](http://jupyter.org/) environment, on a
**single host** using [Docker](https://docs.docker.com).
This deployment: Possible **use cases** include:
* Runs the [JupyterHub components](https://jupyterhub.readthedocs.org/en/latest/getting-started.html#overview) in a Docker container on the host * Creating a JupyterHub demo environment that you can spin up relatively
* Uses [DockerSpawner](https://github.com/jupyter/dockerspawner) to spawn single-user Jupyter Notebook servers in separate Docker containers on the same host quickly.
* Persists JupyterHub data in a Docker volume on the host * Providing a multi-user Jupyter Notebook environment for small classes,
* Persists user notebook directories in Docker volumes on the host teams, or departments.
* Uses [OAuthenticator](https://github.com/jupyter/oauthenticator) and [GitHub OAuth](https://developer.github.com/v3/oauth/) to authenticate users
**Disclaimer:** This deployment is **NOT** intended for a production
environment.
## Technical Overview
Key components of this reference deployment are:
* **Host**: Runs the [JupyterHub components](https://jupyterhub.readthedocs.org/en/latest/getting-started.html#overview)
in a Docker container on the host.
* **Authenticator**: Uses [OAuthenticator](https://github.com/jupyter/oauthenticator)
and [GitHub OAuth](https://developer.github.com/v3/oauth/) to
authenticate users.
* **Spawner**:Uses [DockerSpawner](https://github.com/jupyter/dockerspawner)
to spawn single-user Jupyter Notebook servers in separate Docker
containers on the same host.
* **Persistence of Hub data**: Persists JupyterHub data in a Docker
volume on the host.
* **Persistence of user notebook directories**: Persists user notebook
directories in Docker volumes on the host.
![JupyterHub single host Docker deployment](internal/jupyterhub-docker.png) ![JupyterHub single host Docker deployment](internal/jupyterhub-docker.png)
## Use Cases
Possible use cases for this deployment may include, but are not limited to:
* A JupyterHub demo environment that you can spin up relatively quickly.
* A multi-user Jupyter Notebook environment for small classes, teams, or departments.
## Disclaimer
This deployment is **NOT** intended for a production environment.
## Prerequisites ## Prerequisites
* This deployment uses Docker for all the things, via [Docker Compose](https://docs.docker.com/compose/overview/). ### Docker
It requires [Docker Engine](https://docs.docker.com/engine) 1.12.0 or higher.
See the [installation instructions](https://docs.docker.com/engine/installation/) for your environment.
* This example configures JupyterHub for HTTPS connections (the default).
As such, you must provide TLS certificate chain and key files to the JupyterHub server.
If you do not have your own certificate chain and key, you can either
[create self-signed versions](https://jupyter-notebook.readthedocs.org/en/latest/public_server.html#using-ssl-for-encrypted-communication),
or obtain real ones from [Let's Encrypt](https://letsencrypt.org)
(see the [letsencrypt example](examples/letsencrypt/README.md) for instructions).
From here on, we'll assume you are set up with docker, This deployment uses Docker, via [Docker Compose](https://docs.docker.com/compose/overview/), for all the things.
via a local installation or [docker-machine](./docs/docker-machine.md). [Docker Engine](https://docs.docker.com/engine) 1.12.0 or higher is
At this point, required.
1. Use [Docker's installation instructions](https://docs.docker.com/engine/installation/)
to set up Docker for your environment.
2. To verify your docker installation, whether running docker as a local
installation or using [docker-machine](./docs/docker-machine.md),
enter these commands:
```bash
docker version
docker ps docker ps
should work.
## Setup GitHub Authentication
This deployment uses GitHub OAuth to authenticate users.
It requires that you create a [GitHub application](https://github.com/settings/applications/new).
You will need to specify an OAuth callback URL in the following form:
```
https://<myhost.mydomain>/hub/oauth_callback
```
You must pass the secrets that GitHub provides for your application to JupyterHub at runtime.
You can do this by setting the `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`,
and `OAUTH_CALLBACK_URL` environment variables when you run the JupyterHub container,
or you can add them to the `.env` file in the root directory of this repository. For example,
```
GITHUB_CLIENT_ID=<github_client_id>
GITHUB_CLIENT_SECRET=<github_client_secret>
OAUTH_CALLBACK_URL=https://<myhost.mydomain>/hub/oauth_callback
```
**Note:** The `.env` file is a special file that Docker Compose uses to lookup environment variables.
If you choose to place the GitHub secrets in this file,
you should ensure that this file remains private
(e.g., do not commit the secrets to source control).
## Build the JupyterHub Docker image
Configure JupyterHub and build it into a Docker image.
1. Copy the TLS certificate chain and key files for the JupyterHub server to a directory named `secrets` within this repository directory. These will be added to the JupyterHub Docker image at build time. If you do not have a certificate chain and key, you can either [create self-signed versions](https://jupyter-notebook.readthedocs.org/en/latest/public_server.html#using-ssl-for-encrypted-communication), or obtain real ones from [Let's Encrypt](https://letsencrypt.org) (see the [letsencrypt example](examples/letsencrypt/README.md) for instructions).
``` ```
### HTTPS and SSL/TLS certificate
This deployment configures JupyterHub to use HTTPS. You must provide a
certificate and key file in the JupyterHub configuration. To configure:
1. Obtain the domain name that you wish to use for JupyterHub, for
example, `myfavoritesite.com` or `jupiterplanet.org`.
1. If you do not have an existing certificate and key, you can:
- obtain one from [Let's Encrypt](https://letsencrypt.org) using
the [certbot](https://certbot.eff.org) client,
- use the helper script in this repo's [letsencrypt example](examples/letsencrypt/README.md), or
- [create a self-signed certificate](https://jupyter-notebook.readthedocs.org/en/latest/public_server.html#using-ssl-for-encrypted-communication).
1. Copy the certificate and key files to a
directory named `secrets` in this repository's root directory. These will be
added to the JupyterHub Docker image at build time. For example, create a
`secrets` directory in the root of this repo and copy the certificate and
key files (`jupyterhub.crt` and `jupyterhub.key`) to this directory:
```bash
mkdir -p secrets mkdir -p secrets
cp jupyterhub.crt jupyterhub.key secrets/ cp jupyterhub.crt jupyterhub.key secrets/
``` ```
1. Create a `userlist` file with a list of authorized users. At a minimum, this file should contain a single admin user. The username should be a GitHub username. For example:
## Authenticator setup
This deployment uses GitHub OAuth to authenticate users.
It requires that you create and register a [GitHub OAuth application](https://github.com/settings/applications/new)
by filling out a form on the GitHub site:
![GitHub OAuth application form](docs/oauth-form.png)
In this form, you will specify the OAuth application's callback URL in
this format: `https://<myhost.mydomain>/hub/oauth_callback`.
After you submit the GitHub form, GitHub registers your OAuth application and
assigns a unique Client ID and Client Secret. The Client Secret should be
kept private.
At JupyterHub's runtime, you must pass the GitHub OAuth Client ID, Client
Secret and OAuth callback url. You can do this by either:
- setting the `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, and
`OAUTH_CALLBACK_URL` environment variables when you run the
JupyterHub container, or
- add them to the `.env` file in the root directory of this repository.
For example, update the following lines in the `.env` file:
`.env` file
```
GITHUB_CLIENT_ID=<github_client_id>
GITHUB_CLIENT_SECRET=<github_client_secret>
OAUTH_CALLBACK_URL=https://<myhost.mydomain>/hub/oauth_callback
```
**Note:** The `.env` file is a special file that Docker Compose uses
to lookup environment variables. If you choose to place the GitHub
OAuth application settings in this file, you should make sure that the
file remains private (be careful to not commit the `.env` file with
these secrets to source control).
## Build the JupyterHub Docker image
Finish configuring JupyterHub and then build the hub's Docker image. (We'll
build the Jupyter Notebook image in the next section.)
1. Configure `userlist`: Create a `userlist` file of authorized JupyterHub
users. The list should contain GitHub usernames, and this file should
designate at least one `admin` user. For instance, the example file below
contains three users, `jtyberg`, `jenny`, and `guido`, and one designated
administrator, `jtyberg`:
`userlist` file
``` ```
jtyberg admin jtyberg admin
jenny
guido
``` ```
The admin user will have the ability to add more users in the JupyterHub admin console. The admin user will have the ability to add more users through JupyterHub's
admin console.
1. Use [docker-compose](https://docs.docker.com/compose/reference/) to build the 1. Use [docker-compose](https://docs.docker.com/compose/reference/) to build
JupyterHub Docker image on the active Docker machine host: the JupyterHub Docker image on the active Docker machine host by running
the `make build` command:
``` ```bash
make build make build
``` ```
## Prepare the Jupyter Notebook Image
## Spawner: Prepare the Jupyter Notebook Image
You can configure JupyterHub to spawn Notebook servers from any Docker image, as You can configure JupyterHub to spawn Notebook servers from any Docker image, as
long as the image's `ENTRYPOINT` and/or `CMD` starts a single-user instance of long as the image's `ENTRYPOINT` and/or `CMD` starts a single-user instance of
@ -126,36 +195,48 @@ Notebook server that is compatible with JupyterHub).
You can pull the image using the following command: You can pull the image using the following command:
``` ```bash
make notebook_image make notebook_image
``` ```
## Run JupyterHub ## Run JupyterHub
Run the JupyterHub container on the host. Run the JupyterHub container on the host.
To run the JupyterHub container in detached mode: To run the JupyterHub container in detached mode:
``` ```bash
docker-compose up -d docker-compose up -d
``` ```
Once the container is running, you should be able to access the JupyterHub console at Once the container is running, you should be able to access the JupyterHub console at
**file**
``` ```
https://myhost.mydomain https://myhost.mydomain
``` ```
To bring down the JupyterHub container: To bring down the JupyterHub container:
``` ```bash
docker-compose down docker-compose down
``` ```
---
## Behind the scenes ## Behind the scenes
`make build` does a few things behind the scenes, to set up the environment for JupyterHub: `make build` does a few things behind the scenes, to set up the environment for JupyterHub:
### Create a JupyterHub Data Volume
Create a Docker volume to persist JupyterHub data. This volume will reside on the host machine. Using a volume allows user lists, cookies, etc., to persist across JupyterHub container restarts.
```bash
docker volume create --name jupyterhub-data
```
### Create a Docker Network ### Create a Docker Network
Create a Docker network for inter-container communication. The benefits of using a Docker network are: Create a Docker network for inter-container communication. The benefits of using a Docker network are:
@ -165,17 +246,11 @@ Create a Docker network for inter-container communication. The benefits of usin
Here we create a Docker network named `jupyterhub-network`. Later, we will configure the JupyterHub and single-user Jupyter Notebook containers to run attached to this network. Here we create a Docker network named `jupyterhub-network`. Later, we will configure the JupyterHub and single-user Jupyter Notebook containers to run attached to this network.
``` ```bash
docker network create jupyterhub-network docker network create jupyterhub-network
``` ```
### Create a JupyterHub Data Volume ---
Create a Docker volume to persist JupyterHub data. This volume will reside on the host machine. Using a volume allows user lists, cookies, etc., to persist across JupyterHub container restarts.
```
docker volume create --name jupyterhub-data
```
## FAQ ## FAQ
@ -183,7 +258,7 @@ docker volume create --name jupyterhub-data
Use `docker logs <container>`. For example, to view the logs of the `jupyterhub` container Use `docker logs <container>`. For example, to view the logs of the `jupyterhub` container
``` ```bash
docker logs jupyterhub docker logs jupyterhub
``` ```
@ -197,7 +272,7 @@ JupyterHub reads the Notebook image name from `jupyterhub_config.py`, which
reads the Notebook image name from the `DOCKER_NOTEBOOK_IMAGE` environment reads the Notebook image name from the `DOCKER_NOTEBOOK_IMAGE` environment
variable: variable:
``` ```python
# DockerSpawner setting in jupyterhub_config.py # DockerSpawner setting in jupyterhub_config.py
c.DockerSpawner.container_image = os.environ['DOCKER_NOTEBOOK_IMAGE'] c.DockerSpawner.container_image = os.environ['DOCKER_NOTEBOOK_IMAGE']
``` ```
@ -205,6 +280,7 @@ c.DockerSpawner.container_image = os.environ['DOCKER_NOTEBOOK_IMAGE']
By default, the`DOCKER_NOTEBOOK_IMAGE` environment variable is set in the By default, the`DOCKER_NOTEBOOK_IMAGE` environment variable is set in the
`.env` file. `.env` file.
**file**
``` ```
# Setting in the .env file # Setting in the .env file
DOCKER_NOTEBOOK_IMAGE=jupyter/scipy-notebook:2d878db5cbff DOCKER_NOTEBOOK_IMAGE=jupyter/scipy-notebook:2d878db5cbff
@ -216,7 +292,7 @@ by setting the `DOCKER_NOTEBOOK_IMAGE` variable to a different Notebook
image in the environment where you launch JupyterHub. For example, the image in the environment where you launch JupyterHub. For example, the
following setting would be used to spawn single-user `pyspark` notebook servers: following setting would be used to spawn single-user `pyspark` notebook servers:
``` ```bash
export DOCKER_NOTEBOOK_IMAGE=jupyterhub/pyspark-notebook:2d878db5cbff export DOCKER_NOTEBOOK_IMAGE=jupyterhub/pyspark-notebook:2d878db5cbff
docker-compose up -d docker-compose up -d
@ -244,25 +320,25 @@ There are multiple ways to [backup and restore](https://docs.docker.com/engine/u
Suppose you have the following running containers: Suppose you have the following running containers:
``` ```bash
docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}" docker ps --format "table {{.ID}}\t{{.Image}}\t{{.Names}}"
CONTAINER ID IMAGE NAMES CONTAINER ID IMAGE NAMES
bc02dd6bb91b jupyter/minimal-notebook jupyter-jtyberg bc02dd6bb91b jupyter/minimal-notebook jupyter-jtyberg
7b48a0b33389 jupyterhub jupyterhub 7b48a0b33389 jupyterhub jupyterhub
``` ```
In this deployment, the user's notebook directories (`/home/jovyan/work`) are backed by Docker volumes. In this deployment, the user's notebook directories (`/home/jovyan/work`) are backed by Docker volumes.
``` ```bash
docker inspect -f '{{ .Mounts }}' jupyter-jtyberg docker inspect -f '{{ .Mounts }}' jupyter-jtyberg
[{jtyberg /var/lib/docker/volumes/jtyberg/_data /home/jovyan/work local rw true rprivate}] [{jtyberg /var/lib/docker/volumes/jtyberg/_data /home/jovyan/work local rw true rprivate}]
``` ```
We can backup the user's notebook directory by running a separate container that mounts the user's volume and creates a tarball of the directory. We can backup the user's notebook directory by running a separate container that mounts the user's volume and creates a tarball of the directory.
``` ```bash
docker run --rm \ docker run --rm \
-u root \ -u root \
-v /tmp:/backups \ -v /tmp:/backups \

BIN
docs/oauth-form.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 244 KiB

36
examples/letsencrypt/README.md
View File

@ -1,14 +1,23 @@
# Let's Encrypt # Let's Encrypt
This example includes a Docker Compose configuration file that you can use to deploy [JupyterHub](https://github.com/jupyter/jupyterhub) with TLS certificate and key files generated by [Let's Encrypt](https://letsencrypt.org). This example includes a Docker Compose configuration file that you can
use to deploy [JupyterHub](https://github.com/jupyter/jupyterhub) with
TLS certificate and key files generated by [Let's Encrypt](https://letsencrypt.org).
The `docker-compose.yml` configuration file in this example extends the JupyterHub service defined in the `docker-compose.yml` file in the root directory of this repository. The `docker-compose.yml` configuration file in this example extends the
JupyterHub service defined in the `docker-compose.yml` file in the root
directory of this repository.
When you run the JupyterHub Docker container using the configuration file in this directory, Docker mounts an additional volume containing the Let's Encrypt TLS certificate and key files, and overrides the `SSL_CERT` and `SSL_KEY` environment variables to point to these files. When you run the JupyterHub Docker container using the configuration
file in this directory, Docker mounts an additional volume containing
the Let's Encrypt TLS certificate and key files, and overrides the
`SSL_CERT` and `SSL_KEY` environment variables to point to these files.
## Create a secrets volume ## Create a secrets volume
This example stores the Let's Encrypt TLS certificate and key files in a Docker volume, and mounts the volume to the JupyterHub container at runtime. This example stores the Let's Encrypt TLS certificate and key files in
a Docker volume, and mounts the volume to the JupyterHub container at
runtime.
Create a volume to store the certificate and key files. Create a volume to store the certificate and key files.
@ -21,11 +30,18 @@ docker volume create --name jupyterhub-secrets
## Generate Let's Encrypt certificate and key ## Generate Let's Encrypt certificate and key
Run the `letsencrypt.sh` script to create a TLS full-chain certificate and key. Run the `letsencrypt.sh` script to create a TLS full-chain certificate
and key.
The script downloads and runs the `letsencrypt` Docker image to create a full-chain certificate and private key, and stores the files in a Docker volume. You must provide a valid, routable, fully-qualified domain name (you must own it), and you must activate the Docker machine host that the domain points to before you run this script. You must also provide a valid email address and the name of the volume you created above. The script downloads and runs the `letsencrypt` Docker image to create a
full-chain certificate and private key, and stores the files in a Docker
volume. You must provide a valid, routable, fully-qualified domain name (you
must own it), and you must activate the Docker machine host that the domain
points to before you run this script. You must also provide a valid email
address and the name of the volume you created above.
_Notes:_ The script hard codes several `letsencrypt` options, one of which automatically agrees to the Let's Encrypt Terms of Service. _Notes:_ The script hard codes several `letsencrypt` options, one of which
automatically agrees to the Let's Encrypt Terms of Service.
``` ```
# Activate Docker machine where JupyterHub will run # Activate Docker machine where JupyterHub will run
@ -39,7 +55,11 @@ eval "$(docker-machine env jupyterhub)"
## Run JupyterHub container ## Run JupyterHub container
To run the JupyterHub container using the Let's Encrypt certificate and key, set the `SECRETS_VOLUME` environment variable to the name of the Docker volume containing the certificate and key files, and run `docker-compose` **from the root directory** of this repository while specifying the `docker-compose.yml` configuration in this directory: To run the JupyterHub container using the Let's Encrypt certificate and key,
set the `SECRETS_VOLUME` environment variable to the name of the Docker volume
containing the certificate and key files, and run `docker-compose` **from the
root directory** of this repository while specifying the `docker-compose.yml`
configuration in this directory:
``` ```
export SECRETS_VOLUME=jupyterhub-secrets export SECRETS_VOLUME=jupyterhub-secrets