Dask Gateway
Dask Gateway gives you your own Dask cluster at the UChicago Analysis Facility on demand. You create a cluster from Python, choose how many workers you want, and connect a Dask client to it. The workers run as HTCondor jobs on the AF batch nodes, under your AF username.
You can use it from:
- a BinderHub notebook launched from the
AB-STABLEorAB-DEVimage, where everything is already set up, or - the AF login nodes (
login.af.uchicago.edu), with a JupyterHub API token.
It does not work from outside the AF, for example from your laptop.
Cluster options and limits
You can set these options when you create a cluster:
| Option | Default | Allowed values | Description |
|---|---|---|---|
image |
depends on client | an image name | Container image for the scheduler and the workers. It must include dask-gateway. |
worker_cores |
2 |
1 to 4 | CPU cores per worker |
worker_memory |
4 |
1 to 8 | Memory per worker, in GiB |
debug_logging |
False |
True/False |
Save each worker's output in your home directory (see Troubleshooting) |
Clusters and workers are stopped automatically:
- A cluster with no activity for 1 hour is shut down.
- Every cluster is shut down 24 hours after it starts.
- Workers run in the batch system's short queue, so each worker is stopped after 4 hours.
Please shut your cluster down when you are done, so its workers are free for other users.
Pick an image that matches your Python environment
Your client, the scheduler and the workers need the same Python, dask and
distributed versions, and the worker image must have every package your tasks
use. Always set image to the one that matches where your client runs:
| Where your client runs | image |
|---|---|
AB-STABLE notebook |
hub.opensciencegrid.org/usatlas/analysis-dask-base:main |
AB-DEV notebook |
hub.opensciencegrid.org/usatlas/analysis-dask-base:dev |
| Login node, set up as in From the AF login nodes | hub.opensciencegrid.org/usatlas/analysis-dask-base:main |
Once you have a client, client.get_versions(check=True) raises an error if the
required versions don't match.
From a BinderHub notebook
Launch AB-STABLE or AB-DEV from BinderHub. These images
include the Dask Gateway client, the Dask JupyterLab extension and the settings
for the AF gateway, and they log in with your notebook's own JupyterHub token.
Other images, such as ml_platform, don't include the Dask Gateway client.
With the Dask sidebar
- Click the Dask logo in JupyterLab's left sidebar.
- Click + NEW. The new cluster uses the worker image that matches your notebook image.
- Click SCALE to choose the number of workers.
- Click <> on the cluster to insert the code that connects a client to it into your notebook.
The panels in the sidebar, such as Task Stream and Progress, show plots from the cluster's dashboard. Click SHUTDOWN on the cluster when you are done.
From code
from dask_gateway import Gateway
gateway = Gateway() # the image's settings point at the AF gateway
cluster = gateway.new_cluster(
image="hub.opensciencegrid.org/usatlas/analysis-dask-base:main", # AB-STABLE
worker_cores=2,
worker_memory=4,
)
cluster.scale(4)
client = cluster.get_client()
print(cluster.dashboard_link)
From the AF login nodes
On the login nodes, Dask Gateway logs you in with a JupyterHub API token instead of a browser login.
1. Create a JupyterHub API token
Go to https://jupyterhub.af.uchicago.edu/hub/token and log in with your AF account. Under Request new API token:
- add a note, for example
dask-gateway login nodes, - choose when the token expires, and
- under Permissions, enter
read:users:name!user.
Dask Gateway only uses the token to find out who you are, so it doesn't need any other permissions. A token with no permissions entered gets full access to your JupyterHub account, including your notebook servers.
Copy the token: JupyterHub shows it only once.
2. Save the token on the login node
Save it in a file that only you can read, then export it in the shell where you will run Python:
(umask 077 && cat > ~/.jupyterhub-token) # paste the token, press Enter, then Ctrl-D
export JUPYTERHUB_API_TOKEN=$(cat ~/.jupyterhub-token)
Run the export line again in every new shell. Treat the file like a password,
and delete the token on the token page when you no longer need it.
3. Install the Dask Gateway client
The login nodes' system Python doesn't include the Dask Gateway client. Create a
virtual environment with versions that match the analysis-dask-base:main image
(Python 3.9 and dask 2024.3.1):
python3 -m venv ~/venvs/dask-gateway
source ~/venvs/dask-gateway/bin/activate
pip install "dask-gateway==2024.1.0" "dask==2024.3.1" "distributed==2024.3.1"
Install the other packages your analysis needs in the same environment. Next
time, just run source ~/venvs/dask-gateway/bin/activate.
4. Start a cluster
from dask_gateway import Gateway
gateway = Gateway(
address="https://daskgateway.af.uchicago.edu",
proxy_address="gateway://10.101.14.69:80",
public_address="https://jupyterhub.af.uchicago.edu/services/dask-gateway/",
auth="jupyterhub", # uses JUPYTERHUB_API_TOKEN
)
cluster = gateway.new_cluster(
image="hub.opensciencegrid.org/usatlas/analysis-dask-base:main",
)
cluster.scale(2)
client = cluster.get_client()
print(cluster.dashboard_link) # open this link in your browser
addressis the gateway's API.proxy_addressis how your client reaches your cluster's scheduler. It can only be reached from inside the AF, which is why this works on the login nodes but not from your laptop.public_addressis used for the dashboard links.
Keep these settings out of ~/.config/dask
Your home directory is shared with your BinderHub notebooks, and they read
~/.config/dask too. Settings saved there would replace the ones the notebook
images come with, so pass them in your code as shown above.
Managing your clusters
# List your running clusters and reconnect to one
clusters = gateway.list_clusters()
cluster = gateway.connect(clusters[0].name)
# Shut it down when you are done
cluster.shutdown()
If your Python process ends without shutting its cluster down, for example
because the kernel died or your connection dropped, the cluster keeps running
until one of the limits stops it. Use
gateway.list_clusters() to find it and shut it down.
In scripts, context managers shut the cluster down even if the script fails:
image = "hub.opensciencegrid.org/usatlas/analysis-dask-base:main"
with gateway.new_cluster(image=image) as cluster:
cluster.scale(4)
with cluster.get_client() as client:
client.wait_for_workers(4)
... # your analysis
Troubleshooting
ValueError: No JupyterHub API token provided:JUPYTERHUB_API_TOKENisn't set in this shell. Run theexportline from step 2.401, message='Unauthorized': the token has expired or was deleted. Create a new one.- Version mismatch warnings, or errors when tasks run on the workers: your client environment doesn't match the cluster image. See Pick an image that matches your Python environment.
- Timeouts connecting to the scheduler:
proxy_addresscan only be reached from inside the AF. Run your client on a login node or in a BinderHub notebook. - Workers don't start, or stop unexpectedly: workers are batch jobs, so they
wait for free slots when the facility is busy, and they are stopped after 4
hours. To see their output, create the cluster with
debug_logging=True. Each worker's output is then saved in~/.dask-gateway/<id>/, where<id>is the part ofcluster.nameafterdask-gateway..
For anything else, see Getting Help.