Skip to content

Instantly share code, notes, and snippets.

@5lx
Last active November 8, 2018 23:45
Show Gist options
  • Select an option

  • Save 5lx/660396bfaf4c71584a81faf5764c172e to your computer and use it in GitHub Desktop.

Select an option

Save 5lx/660396bfaf4c71584a81faf5764c172e to your computer and use it in GitHub Desktop.
Palmetto deep learning environment conf

Palmetto deep learning environment configuration

Get a compute node:

qsub -I -l select=1:ncpus=4:ngpus=1:gpu_model=k40:mem=16gb,walltime=10:00:00

For Python 2.7

Installation:

# Add modules
module add anaconda/4.3.0
module add cuda-toolkit/8.0.44
module add cuDNN/8.0v6

# Create anaconda environment profile and activate it
conda create -n tfpy2
source activate tfpy2

# Check python version and packages:
# python -V
# pip list

# Install tensorflow-gpu locally
pip install --ignore-installed --upgrade --user tensorflow-gpu==1.4.1

# Finished!
# If you want to deactivate tfpy2, just
# source deactivate tfpy2

For next time using:

module add anaconda/4.3.0
module add cuda-toolkit/8.0.44
module add cuDNN/8.0v6
source activate tfpy2
python
...

For Python 3.4

Installation:

# Add modules
module add anaconda3/4.3.0
module add cuda-toolkit/8.0.44
module add cuDNN/8.0v6

# Create anaconda environment profile and activate it
conda create -n tfpy3
source activate tfpy3

# Install tensorflow-gpu locally
pip install --ignore-installed --upgrade --user tensorflow-gpu==1.4.1

# Finished!
# If you want to deactivate tfpy3, just
# source deactivate tfpy3

For next time using:

module add anaconda3/4.3.0
module add cuda-toolkit/8.0.44
module add cuDNN/8.0v6
source activate tfpy3
python
...

Testing Code:

import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
@5lx
Copy link
Author

5lx commented Mar 27, 2018

MUSE

Install

# env:
module add anaconda/4.3.0 cuda-toolkit/9.0.176 cuDNN/9.0v7 git/2.6.3 intel/17.0
conda create -n muse python=2.7
source activate muse
pip install http://download.pytorch.org/whl/cu90/torch-0.3.1-cp27-cp27mu-linux_x86_64.whl
pip install torchvision
pip install scipy
conda install faiss-gpu -c pytorch

Test GPU

import torch
import torch.cuda
import torch.cuda.comm as comm
torch.cuda.is_available()

import faiss

Run it

# source and data:
git clone https://github.com/facebookresearch/MUSE
cd MUSE/data
./get_evaluation.sh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment