mm/jupyterhub-deploy-docker
1
0
Fork 0
Code Pull Requests Releases Activity

Merge pull request #12 from minrk/makefile

Browse Source 
Add Makefile, simplify README
This commit is contained in:
Carol Willing 2016-05-12 08:54:43 -07:00
commit e12184aaf3
4 changed files with 193 additions and 125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Dockerfile.jupyterhub
View File

@ -9,9 +9,9 @@ RUN /opt/conda/bin/pip install \
dockerspawner==0.3.*
# Copy TLS certificate and key
ENV SSL_CERT /srv/jupyterhub/secrets/jupyterhub.cer
ENV SSL_CERT /srv/jupyterhub/secrets/jupyterhub.crt
ENV SSL_KEY /srv/jupyterhub/secrets/jupyterhub.key
COPY ./secrets/*.cer $SSL_CERT
COPY ./secrets/*.crt $SSL_CERT
COPY ./secrets/*.key $SSL_KEY
RUN chmod 700 /srv/jupyterhub/secrets && \
chmod 600 /srv/jupyterhub/secrets/*

40
Makefile Normal file
View File

@ -0,0 +1,40 @@
include .env
.DEFAULT_GOAL=build
network:
@docker network inspect $(DOCKER_NETWORK_NAME) >/dev/null 2>&1 || docker network create $(DOCKER_NETWORK_NAME)
volumes:
@docker volume inspect $(DATA_VOLUME_HOST) >/dev/null 2>&1 || docker volume create --name $(DATA_VOLUME_HOST)
self-signed-cert:
# make a self-signed cert
secrets/jupyterhub.crt:
@echo "Need an SSL certificate in secrets/jupyterhub.crt"
@exit 1
secrets/jupyterhub.key:
@echo "Need an SSL key in secrets/jupyterhub.key"
@exit 1
userlist:
@echo "Add usernames, one per line, to ./userlist, such as:"
@echo " zoe admin"
@echo " wash"
@exit 1
check-files: secrets/jupyterhub.crt secrets/jupyterhub.key userlist
# fail in an informative way if files don't exist
pull:
docker pull $(DOCKER_NOTEBOOK_IMAGE)
build: check-files pull network volumes
docker-compose build
up:
docker-compose up -d
.PHONY: network volumes check-files build up

192
README.md
View File

@ -25,102 +25,39 @@ This deployment is **NOT** intended for a production environment.
## Prerequisites
* This deployment uses Docker for all the things. It assumes that you wiil be using [Docker Machine](https://docs.docker.com/machine/overview/) and [Docker Compose](https://docs.docker.com/compose/overview/) on a local workstation or laptop to create, build, and run Docker images on a remote host. It requires [Docker Toolbox](https://www.docker.com/products/docker-toolbox) 1.11.0 or higher. See the [installation instructions](https://docs.docker.com/engine/installation/) for your environment.
* This example configures JupyterHub for HTTPS connections (the default). As such, you must provide TLS certificate chain and key files to the JupyterHub server. If you do not have your own certificate chain and key, you can either [create self-signed versions](https://jupyter-notebook.readthedocs.org/en/latest/public_server.html#using-ssl-for-encrypted-communication), or obtain real ones from [Let's Encrypt](https://letsencrypt.org) (see the [letsencrypt example](examples/letsencrypt/README.md) for instructions).
* This deployment uses Docker for all the things, via [Docker Compose](https://docs.docker.com/compose/overview/).
It requires [Docker Toolbox](https://www.docker.com/products/docker-toolbox) 1.11.0 or higher.
See the [installation instructions](https://docs.docker.com/engine/installation/) for your environment.
* This example configures JupyterHub for HTTPS connections (the default).
As such, you must provide TLS certificate chain and key files to the JupyterHub server.
If you do not have your own certificate chain and key, you can either
[create self-signed versions](https://jupyter-notebook.readthedocs.org/en/latest/public_server.html#using-ssl-for-encrypted-communication),
or obtain real ones from [Let's Encrypt](https://letsencrypt.org)
(see the [letsencrypt example](examples/letsencrypt/README.md) for instructions).
## Create a Docker Machine
From here on, we'll assume you are set up with docker,
via a local installation or [docker-machine](./docs/docker-machine.md).
At this point,
Use [Docker Machine](https://docs.docker.com/machine/) to provision a new host, or to connect to an existing host. In either case, the result will be a "machine" configured on your local workstation or laptop that represents the remote host. After you activate the machine on your local workstation, when you run `docker` commands locally, Docker Machine will execute them on the remote host for you.
docker ps
### Provision a new host
should work.
Docker Machine can provision new hosts on various platforms using one of its [supported drivers](https://docs.docker.com/machine/drivers/). When provisioning a host, Docker Machine will automatically install the latest version of [Docker Engine](https://www.docker.com/products/docker-engine) on the host. It will also generate local TLS certificate and key files to connect to the host and authenticate with the `docker` daemon on the host.
In the following example, we provision a new virtual server on [IBM SoftLayer](https://www.softlayer.com/promo/freeCloud/freecloud). (You can provision similarly on RackSpace, AWS, and other hosting providers). We set `DRIVER_OPTS` to the SoftLayer-specific options.
```
# Set Docker Machine SoftLayer driver options
export DRIVER_OPTS="--driver softlayer \
--softlayer-api-key <my_softlayer_api_key> \
--softlayer-user <my_softlayer_username> \
--softlayer-domain mydomain \
--softlayer-cpu 16 \
--softlayer-memory 65536 \
--softlayer-disk-size 100 \
--softlayer-region wdc01"
# Create a machine named jupyterhub
docker-machine create $DRIVER_OPTS jupyterhub
```
### Connect to an existing host
This example configures a "machine" on your local workstation that connects to an existing remote host at IP address `10.0.0.10`. To do this, you must use Docker Machine's `generic` driver, and your local workstation must have a private SSH key that allows you to perform password-less login to the host.
Substitute your own IP address and path to SSH key in the `DRIVER_OPTS` below. Note that when you run the `docker-machine create` command, Docker Machine will install and configure the latest Docker Engine on the remote host (or upgrade the Docker Engine on the host if it is already installed).
```
# Use the generic driver to create a Docker machine that
# controls the Docker daemon on an existing host
export DRIVER_OPTS="--driver generic \
--generic-ip-address 10.0.0.10 \
--generic-ssh-key /Users/jtyberg/.ssh/myhost_rsa.pem"
# Create a machine named jupyterhub
docker-machine create $DRIVER_OPTS jupyterhub
```
## Activate Docker Machine
You must set specific `DOCKER_*` environment variables to tell `docker` that it should run commands against a particular machine. The remainder of this document assumes that the machine called `jupyterhub` is active, and therefore, all `docker` commands will run on the `jupyterhub` remote host.
To set the `jupyterhub` machine as active:
```
eval "$(docker-machine env jupyterhub)"
```
All this does is set the right environment variables:
```
env|grep DOCKER
DOCKER_HOST=tcp://10.0.0.10:2376
DOCKER_MACHINE_NAME=jupyterhub
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/jtyberg/.docker/machine/machines/jupyterhub
```
To see which machine is active:
```
docker-machine active
```
## Create a Docker Network
Create a Docker network for inter-container communication. The benefits of using a Docker network are:
* container isolation - only the containers on the network can access one another
* name resolution - Docker daemon runs an embedded DNS server to provide automatic service discovery for containers connected to user-defined networks. This allows us to access containers on the same network by name.
Here we create a Docker network named `jupyterhub-network`. Later, we will configure the JupyterHub and single-user Jupyter Notebook containers to run attached to this network.
```
docker network create jupyterhub-network
```
## Setup GitHub Authentication
This deployment uses GitHub OAuth to authenticate users. It requires that you create a [GitHub application](https://github.com/settings/applications/new). You will need to specify an OAuth callback URL in the following form:
This deployment uses GitHub OAuth to authenticate users.
It requires that you create a [GitHub application](https://github.com/settings/applications/new).
You will need to specify an OAuth callback URL in the following form:
```
https://<myhost.mydomain>/hub/oauth_callback
```
You must pass the secrets that GitHub provides for your application to JupyterHub at runtime. You can do this by setting the `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, and `OAUTH_CALLBACK_URL` environment variables when you run the JupyterHub container, or you can add them to the `.env` file in the root directory of this repository. For example,
You must pass the secrets that GitHub provides for your application to JupyterHub at runtime.
You can do this by setting the `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`,
and `OAUTH_CALLBACK_URL` environment variables when you run the JupyterHub container,
or you can add them to the `.env` file in the root directory of this repository. For example,
```
GITHUB_CLIENT_ID=<github_client_id>
@ -128,7 +65,10 @@ GITHUB_CLIENT_SECRET=<github_client_secret>
OAUTH_CALLBACK_URL=https://<myhost.mydomain>/hub/oauth_callback
```
**Note:** The `.env` file is a special file that Docker Compose uses to lookup environment variables. If you choose to place the GitHub secrets in this file, you should ensure that this file remains private (e.g., do not commit the secrets to source control).
**Note:** The `.env` file is a special file that Docker Compose uses to lookup environment variables.
If you choose to place the GitHub secrets in this file,
you should ensure that this file remains private
(e.g., do not commit the secrets to source control).
## Build the JupyterHub Docker image
@ -138,7 +78,7 @@ Configure JupyterHub and build it into a Docker image.
```
mkdir -p secrets
cp jupyterhub.cer jupyterhub.key secrets/
cp jupyterhub.crt jupyterhub.key secrets/
```
1. Create a `userlist` file with a list of authorized users. At a minimum, this file should contain a single admin user. The username should be a GitHub username. For example:
@ -150,41 +90,13 @@ Configure JupyterHub and build it into a Docker image.
The admin user will have the ability to add more users in the JupyterHub admin console.
1. Use [docker-compose](https://docs.docker.com/compose/reference/) to build the
JupyterHub Docker image on the active Docker machine host:
JupyterHub Docker image on the active Docker machine host:
```
docker-compose build
make build
```
## Create a JupyterHub Data Volume
Create a Docker volume to persist JupyterHub data. This volume will reside on the host machine. Using a volume allows user lists, cookies, etc., to persist across JupyterHub container restarts.
```
docker volume create --name jupyterhub-data
```
## Pull the Jupyter Notebook Image
Pull the Jupyter Notebook Docker image that you would like JupyterHub to spawn for each user.
Note: Even though Docker will pull the image to the host the first time a user container is spawned, JupyterHub may timeout if the image is large, so it's better to do it beforehand.
This deployment uses the [jupyter/scipy-notebook](https://hub.docker.com/r/jupyter/scipy-notebook/) Docker image, which is built from the `scipy-notebook` [Docker stacks](https://github.com/jupyter/docker-stacks).
Note that the Docker stacks `*-notebook` images tagged `2d878db5cbff` include the `start-singleuser.sh` script required to start a single-user instance of the Notebook server that is compatible with JupyterHub.
```
docker pull jupyter/scipy-notebook:2d878db5cbff
```
Note: If you choose to use a container image other than
``jupyter/scipy-notebook``, you must change the `DOCKER_NOTEBOOK_IMAGE` value
of the `.env` file to the desired container image. Alternatively, you can
override the value by setting the `DOCKER_NOTEBOOK_IMAGE` variable to a
different Notebook image in the environment where you launch JupyterHub.
## Run the JupyterHub container
## Run JupyterHub
Run the JupyterHub container on the host.
@ -206,6 +118,52 @@ To bring down the JupyterHub container:
docker-compose down
```
## Behind the scenes
`make build` does a few things behind the scenes, to set up the environment for JupyterHub:
### Create a Docker Network
Create a Docker network for inter-container communication. The benefits of using a Docker network are:
* container isolation - only the containers on the network can access one another
* name resolution - Docker daemon runs an embedded DNS server to provide automatic service discovery for containers connected to user-defined networks. This allows us to access containers on the same network by name.
Here we create a Docker network named `jupyterhub-network`. Later, we will configure the JupyterHub and single-user Jupyter Notebook containers to run attached to this network.
```
docker network create jupyterhub-network
```
### Create a JupyterHub Data Volume
Create a Docker volume to persist JupyterHub data. This volume will reside on the host machine. Using a volume allows user lists, cookies, etc., to persist across JupyterHub container restarts.
```
docker volume create --name jupyterhub-data
```
### Pull the Jupyter Notebook Image
Pull the Jupyter Notebook Docker image that you would like JupyterHub to spawn for each user.
Note: Even though Docker will pull the image to the host the first time a user container is spawned, JupyterHub may timeout if the image is large, so it's better to do it beforehand.
This deployment uses the [jupyter/scipy-notebook](https://hub.docker.com/r/jupyter/scipy-notebook/) Docker image, which is built from the `scipy-notebook` [Docker stacks](https://github.com/jupyter/docker-stacks).
Note that the Docker stacks `*-notebook` images tagged `2d878db5cbff` include the `start-singleuser.sh` script required to start a single-user instance of the Notebook server that is compatible with JupyterHub.
```
make pull
```
Note: If you choose to use a container image other than
``jupyter/scipy-notebook``, you must change the `DOCKER_NOTEBOOK_IMAGE` value
of the `.env` file to the desired container image. Alternatively, you can
override the value by setting the `DOCKER_NOTEBOOK_IMAGE` variable to a
different Notebook image in the environment where you launch JupyterHub.
## FAQ
### How can I view the logs for JupyterHub or users' Notebook servers?

70
docs/docker-machine.md Normal file
View File

@ -0,0 +1,70 @@
## Create a Docker Machine
You can use [Docker Machine](https://docs.docker.com/machine/) to provision a new host, or to connect to an existing host. In either case, the result will be a "machine" configured on your local workstation or laptop that represents the remote host. After you activate the machine on your local workstation, when you run `docker` commands locally, Docker Machine will execute them on the remote host for you.
### Provision a new host
Docker Machine can provision new hosts on various platforms using one of its [supported drivers](https://docs.docker.com/machine/drivers/). When provisioning a host, Docker Machine will automatically install the latest version of [Docker Engine](https://www.docker.com/products/docker-engine) on the host. It will also generate local TLS certificate and key files to connect to the host and authenticate with the `docker` daemon on the host.
In the following example, we provision a new virtual server on [IBM SoftLayer](https://www.softlayer.com/promo/freeCloud/freecloud). (You can provision similarly on RackSpace, AWS, and other hosting providers). We set `DRIVER_OPTS` to the SoftLayer-specific options.
```
# Set Docker Machine SoftLayer driver options
export DRIVER_OPTS="--driver softlayer \
--softlayer-api-key <my_softlayer_api_key> \
--softlayer-user <my_softlayer_username> \
--softlayer-domain mydomain \
--softlayer-cpu 16 \
--softlayer-memory 65536 \
--softlayer-disk-size 100 \
--softlayer-region wdc01"
# Create a machine named jupyterhub
docker-machine create $DRIVER_OPTS jupyterhub
```
### Connect to an existing host
This example configures a "machine" on your local workstation that connects to an existing remote host at IP address `10.0.0.10`. To do this, you must use Docker Machine's `generic` driver, and your local workstation must have a private SSH key that allows you to perform password-less login to the host.
Substitute your own IP address and path to SSH key in the `DRIVER_OPTS` below. Note that when you run the `docker-machine create` command, Docker Machine will install and configure the latest Docker Engine on the remote host (or upgrade the Docker Engine on the host if it is already installed).
```
# Use the generic driver to create a Docker machine that
# controls the Docker daemon on an existing host
export DRIVER_OPTS="--driver generic \
--generic-ip-address 10.0.0.10 \
--generic-ssh-key /Users/jtyberg/.ssh/myhost_rsa.pem"
# Create a machine named jupyterhub
docker-machine create $DRIVER_OPTS jupyterhub
```
## Activate Docker Machine
You must set specific `DOCKER_*` environment variables to tell `docker` that it should run commands against a particular machine. The remainder of this document assumes that the machine called `jupyterhub` is active, and therefore, all `docker` commands will run on the `jupyterhub` remote host.
To set the `jupyterhub` machine as active:
```
eval "$(docker-machine env jupyterhub)"
```
All this does is set the right environment variables:
```
env|grep DOCKER
DOCKER_HOST=tcp://10.0.0.10:2376
DOCKER_MACHINE_NAME=jupyterhub
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/jtyberg/.docker/machine/machines/jupyterhub
```
To see which machine is active:
```
docker-machine active
```