Python setup noted
parent
d547174905
commit
652f59b7d9
137
README.md
137
README.md
|
@ -1,9 +1,142 @@
|
||||||
# Pytorch
|
# Pytorch
|
||||||
A forklet of Pytorch for my own quirky needs.
|
A forklet of Pytorch for my own quirky needs.
|
||||||
|
|
||||||
|
* Build notes and scripts for different machines I admin.
|
||||||
|
* CPU builds.
|
||||||
|
* Meh ROCm AMD GPU builds.
|
||||||
|
* Proprietary Nvidia builds.
|
||||||
|
* Flailing at getting larger GPUs + Pytorch on ppc64le going.
|
||||||
|
|
||||||
|
|
||||||
# Install
|
# Install
|
||||||
thusly.
|
thusly, on Debian stable (bookworm/12).
|
||||||
|
|
||||||
|
|
||||||
|
# Dependencies
|
||||||
|
Perhaps this and more:
|
||||||
|
|
||||||
|
## OS
|
||||||
|
```
|
||||||
|
sudo apt install git build-essential libssl-dev zlib1g-dev \
|
||||||
|
libbz2-dev libreadline-dev libsqlite3-dev curl \
|
||||||
|
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
|
||||||
|
libffi-dev liblzma-dev \
|
||||||
|
python3-virtualenv python3-pip
|
||||||
|
```
|
||||||
|
|
||||||
|
## Python
|
||||||
|
At present, seems latest Python that works happily with most Pytorch
|
||||||
|
applications is Python 3.10.
|
||||||
|
|
||||||
|
Use pyenv to manage versions, install something like:
|
||||||
|
|
||||||
|
```
|
||||||
|
# :)
|
||||||
|
curl https://pyenv.run | bash
|
||||||
|
```
|
||||||
|
|
||||||
|
Add to `~/.bashrc`, then re-source it (logout/in or whatever):
|
||||||
|
|
||||||
|
```
|
||||||
|
export PYENV_ROOT="$HOME/.pyenv"
|
||||||
|
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
|
||||||
|
eval "$(pyenv init -)"
|
||||||
|
eval "$(pyenv virtualenv-init -)"
|
||||||
|
```
|
||||||
|
|
||||||
|
Get Python version 3.10:
|
||||||
|
|
||||||
|
```
|
||||||
|
pyenv install 3.10
|
||||||
|
```
|
||||||
|
|
||||||
|
Perhaps other versions, ala:
|
||||||
|
|
||||||
|
```
|
||||||
|
pyenv install 3.9
|
||||||
|
pyenv install 3.11.6
|
||||||
|
pyenv install 3.12
|
||||||
|
pyenv install 3.13-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setup Pip Download Cache
|
||||||
|
So as to not download files from the Internet multiple times from
|
||||||
|
each machine in the cluster, a Pip download cacche can be set up
|
||||||
|
thusly, assuming server IP 192.168.100.101, username debian:
|
||||||
|
|
||||||
|
```
|
||||||
|
mkdir -p ~/devel/devpi
|
||||||
|
cd ~/devel/devpi
|
||||||
|
virtualenv env
|
||||||
|
source env/bin/activate
|
||||||
|
pip install -U setuptools wheel pip
|
||||||
|
pip install devpi-server devpi-web
|
||||||
|
|
||||||
|
sudo mkdir /srv/devpi
|
||||||
|
sudo chown debian:debian /srv/devpi
|
||||||
|
|
||||||
|
devpi-init \
|
||||||
|
--serverdir /srv/devpi
|
||||||
|
|
||||||
|
devpi-gen-config \
|
||||||
|
--host=0.0.0.0 \
|
||||||
|
--port 4040 \
|
||||||
|
--serverdir /srv/devpi \
|
||||||
|
--absolute-urls
|
||||||
|
|
||||||
|
sudo apt install nginx
|
||||||
|
|
||||||
|
sudo cp ~debian/devel/devpi/gen-config/nginx-devpi.conf /etc/nginx/sites-available/
|
||||||
|
|
||||||
|
cd /etc/nginx/sites-enabled
|
||||||
|
sudo ln -s ../sites-available/nginx-devpi.conf .
|
||||||
|
|
||||||
|
sudo apt install supervisor
|
||||||
|
sudo cp ~debian/devel/devpi/gen-config/supervisor-devpi.conf /etc/supervisor/conf.d/
|
||||||
|
|
||||||
|
crontab -e
|
||||||
|
@reboot /usr/local/sbin/supervisord -c /home/debian/etc/supervisor-devpi.conf
|
||||||
|
|
||||||
|
supervisord -c gen-config/supervisord.conf
|
||||||
|
|
||||||
|
sudo reboot
|
||||||
|
|
||||||
|
devpi use http://192.168.100.101:4040
|
||||||
|
|
||||||
|
devpi login root --password ''
|
||||||
|
devpi user -m root password=FOO
|
||||||
|
devpi user -l
|
||||||
|
devpi logoff
|
||||||
|
devpi user -c debian password=BAR email=devpi@localhost
|
||||||
|
devpi login debian --password=BAR
|
||||||
|
|
||||||
|
devpi index -c dev bases=root/pypi
|
||||||
|
devpi use debian/dev
|
||||||
|
devpi install pytest
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## Add Pip Download Cache
|
||||||
|
Add this to clients to use the cache:
|
||||||
|
|
||||||
|
```
|
||||||
|
mkdir -p ~/.config/pip
|
||||||
|
cat > ~/.config/pip/pip.conf <<EOF
|
||||||
|
[global]
|
||||||
|
trusted-host = 192.168.100.101
|
||||||
|
index-url = http://192.168.100.101:4040/root/pypi/+simple/
|
||||||
|
|
||||||
|
[search]
|
||||||
|
index = http://192.168.100.101:4040/root/pypi/
|
||||||
|
EOF
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
# Get Deepcrayon Pytorch Repo
|
||||||
|
```
|
||||||
|
git clone https://spacecruft.org/deepcrayon/pytorch
|
||||||
|
cd pytorch/
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
# Upstream
|
# Upstream
|
||||||
|
@ -11,6 +144,8 @@ Main upstream:
|
||||||
|
|
||||||
* https://github.com/pytorch/pytorch
|
* https://github.com/pytorch/pytorch
|
||||||
|
|
||||||
|
See also: `README-upstream.md`.
|
||||||
|
|
||||||
|
|
||||||
# Disclaimer
|
# Disclaimer
|
||||||
I am not a programmer.
|
I am not a programmer.
|
||||||
|
|
Loading…
Reference in New Issue