Python setup noted

deepcrayon-v2.1
Jeff Moe 2023-11-08 09:36:42 -07:00
parent d547174905
commit 652f59b7d9
1 changed files with 136 additions and 1 deletions

137
README.md
View File

@ -1,9 +1,142 @@
# Pytorch
A forklet of Pytorch for my own quirky needs.
* Build notes and scripts for different machines I admin.
* CPU builds.
* Meh ROCm AMD GPU builds.
* Proprietary Nvidia builds.
* Flailing at getting larger GPUs + Pytorch on ppc64le going.
# Install
thusly.
thusly, on Debian stable (bookworm/12).
# Dependencies
Perhaps this and more:
## OS
```
sudo apt install git build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev \
libffi-dev liblzma-dev \
python3-virtualenv python3-pip
```
## Python
At present, seems latest Python that works happily with most Pytorch
applications is Python 3.10.
Use pyenv to manage versions, install something like:
```
# :)
curl https://pyenv.run | bash
```
Add to `~/.bashrc`, then re-source it (logout/in or whatever):
```
export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
```
Get Python version 3.10:
```
pyenv install 3.10
```
Perhaps other versions, ala:
```
pyenv install 3.9
pyenv install 3.11.6
pyenv install 3.12
pyenv install 3.13-dev
```
## Setup Pip Download Cache
So as to not download files from the Internet multiple times from
each machine in the cluster, a Pip download cacche can be set up
thusly, assuming server IP 192.168.100.101, username debian:
```
mkdir -p ~/devel/devpi
cd ~/devel/devpi
virtualenv env
source env/bin/activate
pip install -U setuptools wheel pip
pip install devpi-server devpi-web
sudo mkdir /srv/devpi
sudo chown debian:debian /srv/devpi
devpi-init \
--serverdir /srv/devpi
devpi-gen-config \
--host=0.0.0.0 \
--port 4040 \
--serverdir /srv/devpi \
--absolute-urls
sudo apt install nginx
sudo cp ~debian/devel/devpi/gen-config/nginx-devpi.conf /etc/nginx/sites-available/
cd /etc/nginx/sites-enabled
sudo ln -s ../sites-available/nginx-devpi.conf .
sudo apt install supervisor
sudo cp ~debian/devel/devpi/gen-config/supervisor-devpi.conf /etc/supervisor/conf.d/
crontab -e
@reboot /usr/local/sbin/supervisord -c /home/debian/etc/supervisor-devpi.conf
supervisord -c gen-config/supervisord.conf
sudo reboot
devpi use http://192.168.100.101:4040
devpi login root --password ''
devpi user -m root password=FOO
devpi user -l
devpi logoff
devpi user -c debian password=BAR email=devpi@localhost
devpi login debian --password=BAR
devpi index -c dev bases=root/pypi
devpi use debian/dev
devpi install pytest
```
## Add Pip Download Cache
Add this to clients to use the cache:
```
mkdir -p ~/.config/pip
cat > ~/.config/pip/pip.conf <<EOF
[global]
trusted-host = 192.168.100.101
index-url = http://192.168.100.101:4040/root/pypi/+simple/
[search]
index = http://192.168.100.101:4040/root/pypi/
EOF
```
# Get Deepcrayon Pytorch Repo
```
git clone https://spacecruft.org/deepcrayon/pytorch
cd pytorch/
```
# Upstream
@ -11,6 +144,8 @@ Main upstream:
* https://github.com/pytorch/pytorch
See also: `README-upstream.md`.
# Disclaimer
I am not a programmer.