Skip to content

Instantly share code, notes, and snippets.

@shadiakiki1986

shadiakiki1986/README.md

Last active May 29, 2018 07:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save shadiakiki1986/71f49c4051b68a2a9d6e449b62d00a8e to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save shadiakiki1986/71f49c4051b68a2a9d6e449b62d00a8e to your computer and use it in GitHub Desktop.
Download ZIP
H2O driverless AI installation instructions
Raw
README.md

Installation scripts for H2O driverless AI

Launch order

  • install.sh (requires sudo)
    • need to exit and re-ssh into machine for "docker usage without sudo" to take effect
  • for GPU, refer to init-gpu.sh
  • dropbox.sh (optional)
  • start.sh
Raw
dropbox.sh
# Note 1: At the step of installing dropbox, open link displayed in screen to link your dropbox account
# for convenience, link the driverless AI folders to a dropbox folder
# This script is not meant to be run automatically in one shot
# Instead, run it manually one step at a time, modifying steps as needed
#############################
# install requirements for dropbox headless for convenience to upload files into driverless AI
# https://www.dropbox.com/install-linux
sudo apt-get -qq -y install \
python
sudo wget https://www.dropbox.com/download?dl=packages/dropbox.py -O /usr/bin/dropbox
sudo chmod +x /usr/bin/dropbox
# mount a volume from /dev/xvdf into /mnt/ec2vol
# and put the dropbox folder there
# to format a new volume: mkfs -t ext4 /dev/xvdf
sudo mkdir /mnt/ec2vol
sudo mount /dev/xvdf /mnt/ec2vol
sudo chown ubuntu:ubuntu /mnt/ec2vol -R
# This is where the prompt in Note 1 will show up
# move dropbox folder to /mnt/ec2vol
cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
~/.dropbox-dist/dropboxd # manually click the link, when done linking the account, press Ctrl+C to terminate
# selective sync
# http://rkulla.blogspot.com/2014/03/headless-dropbox.html
dropbox exclude add "AEE-Office"
dropbox exclude add "AEE-Stores"
dropbox exclude add "Akiki VHS"
dropbox exclude add "Lebanon2014_Jim"
# start again
dropbox start
#------------------------------------------------------------------------------------
# migrate an existing dropbox folder
# NOT SURE IF THIS WORKS. HAVE FAILED EVERYTIME SO FAR
watch dropbox status # wait till it starts syncing, then do dropbox stop
mv ~/Dropbox /mnt/ec2vol/Dropbox
ln -s /mnt/ec2vol/Dropbox ~/Dropbox
# At this step, make sure no files get "deleted" by dropbox unintentionally
# If it happens, log into the web interface, click "show deleted files", select them, and click "restore"
dropbox start
#------------------------------------------------------------------------------------
# link the dropbox folder to the /var/lib/h2o-driverless-ai folder that will be mount into the docker image
mkdir ~/Dropbox/Shadi_Datasets/h2o-driverless-ai/{mkdir,log,license}
cd /var/lib
sudo mv driverless-ai-dir driverless-ai-dir.bkp
sudo ln -s ~/Dropbox/Shadi_Datasets/h2o-driverless-ai/ .
cd -
Raw
install.sh
# Script to install driverless AI from H2O
# Usage
# wget https://gist.github.com/shadiakiki1986/71f49c4051b68a2a9d6e449b62d00a8e/raw/install.sh -O -|sudo bash
#
# Requirements
# needs to be used with sudo to run
# minimum free disk space 13 GB
# ubuntu 16 or 17
# A dropbox account for simplifying of uploading files into driverless AI's data folder
# Adds the user "ubuntu" to the docker group to be able to run docker without sudo
#
# Note: permalink from https://gist.github.com/atenni/5604615
######################################################################################################################################################################
set -e
export DEBIAN_FRONTEND=noninteractive
# install docker
# prepend h2o's documentation with getting the docker's official gpg key
# https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#os-requirements
apt-get -qq -y remove docker docker-engine docker.io
apt-get -qq -y update
apt-get -qq -y install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
apt-key fingerprint 0EBFCD88
# continue with h2o's documentation
add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
apt-get -qq -y update
apt-get -qq -y install docker-ce
docker run hello-world
# add user to docker group
gpasswd -a ubuntu docker
# Install nvidia-docker. Required only if GPU is used
wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
dpkg -i /tmp/nvidia-docker*.deb
rm /tmp/nvidia-docker*.deb
# without this, the nvidia-docker run... below would fail
apt-get -qq -y install nvidia-modprobe
# download the driverless ai docker image (2GB)
wget -P /tmp https://s3-us-west-2.amazonaws.com/h2o-internal-release/docker/driverless-ai-docker-runtime-rel-0.8.2.gz
# load it in docker (slow command)
sleep 1 # is this necessary for the wget to move the file to /tmp ? Not sure
docker load < /tmp/driverless-ai-docker-runtime-rel*.gz # <<< does this only work with bash and not /bin/sh?
rm /tmp/driverless-ai-docker-runtime-rel*.gz
# prepare folders
mkdir -p /var/lib/h2o-driverless-ai/{data,log,license}
Raw
launch-h2o-ai.sh
#!/bin/bash
# launch h2o.ai locally
# mounts the data/log/license directories from /var/lib/h2o-driverless-ai
# publishes to port 3001
# cannot use this with nvidia-modprobe failing with no GPU
# set -e
ROOT=/var/lib/h2o-driverless-ai
RUNNER=nvidia-docker
# the below will exit with non-0 if no gpu
nvidia-modprobe
if [ $? -eq 1 ]; then
echo "no gpu found"
RUNNER=docker
fi
echo "launch on port 3001. Check http://ip:3001"
$RUNNER run \
--rm \
-u `id -u`:`id -g` \
-p 3001:12345 \
-v $ROOT/data:/data \
-v $ROOT/log:/log \
-v $ROOT/license:/license \
opsh2oai/h2oai-runtime
Raw
scorers.sh
# This is an example script of requirements for running a scorer.zip file
# exported from h2o driverless-ai
# Assumes filename is scorer.zip
#############################################
# install python3.6 requirement
sudo add-apt-repository ppa:jonathonf/python-3.6
sudo apt-get update
sudo apt-get install python3.6 python3.6-dev
# install some openmp stuff, and fix a missing link
sudo apt-get install libomp5
sudo ln -s /usr/lib/x86_64-linux-gnu/libomp.so.5 /usr/lib/x86_64-linux-gnu/libomp.so
unzip scorer.zip # could use -d /path/to/another/folder
cd python_module
# instead of using the run_example.sh which uses virtualenv
# use pew, because pew is awesome
# https://github.com/berdario/pew
sudo pip3 install pew
pew new -r requirements.txt --python=python3.6 H2O_SCORER
python example.py
@shadiakiki1986
Copy link
Author

shadiakiki1986 commented May 29, 2018

You're welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment