Distributed Computing

HOWTO Set up and run Tensorflow on multiple nodes. This is to this particular configuration.

Software

Main software in use:

Debian
Proxmox
Ceph
Python 3
Tensorflow
Jupyter
clusterssh

Installation

Major steps.

Install Proxmox on bare metal.
Clone Debian KVM Nodes.
Set up nodes.
Install Tensorflow.
Set up Ceph.

Proxmox

Setting up Proxmox is outside the scope of this document. All you really need is some virtual machines, however they are created.

https://www.proxmox.com/en/proxmox-ve

Set up nodes

# On main workstation or node where you built tensorflow:
NODES="ml1 ml2 ml3 ml4 ml5"
for i in $NODES
	do scp -p tensorflow-2.1.0-cp37-cp37m-linux_x86_64.whl $i:
done

# On worker nodes:
sudo apt update
sudo apt install python3-pip sshfs
# XXX deps...
pip3 install --upgrade setuptools
pip3 install --user tensorflow-2.1.0-cp37-cp37m-linux_x86_64.whl
pip3 install --user simplejson
pip3 install --user pillow

Another way, using upstream tensorflow packages. You also have to install the latest pip from pip, or you'll get tensorflow 1.x.

pip3 install pip
pip3 install --upgrade pip
# make sure new `pip3` at `~/.local/bin/pip3` is in front in `$PATH`.
# install tensorflow
pip3 install --user tensorflow
# If that fails due to the PATH, run like:
~/.local/bin/pip3 install --user tensorflow
pip3 list | grep tensorflow
# There's a bunch of tests that can be run, such as:
python3 ~/devel/tensorflow/tensorflow/tensorflow/python/distribute/multi_worker_continuous_run_test.py

Usage

top

Meh

# for running some tensorflow tests:
pip3 install --user portpicker
# For other examples/tests:
#pip3 install --user opencv-python
apt install python3-opencv
pip3 install --user pandas

1.8 KiB Raw Blame History