Skip to content

Instantly share code, notes, and snippets.

@dipta007

dipta007/deploy_sglang.sh

Last active February 21, 2025 01:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dipta007/7677e05c44733b872882788e5d6c42a3 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dipta007/7677e05c44733b872882788e5d6c42a3 to your computer and use it in GitHub Desktop.
Download ZIP
SGLang deploy using udocker
Raw
deploy_sglang.sh
# pip install udocker
udocker pull lmsysorg/sglang:latest
udocker create --name=sglang lmsysorg/sglang:latest
udocker setup --nvidia sglang
udocker run \
--volume="/${PWD}:/workspace" \
--env "HF_TOKEN=<HF_TOKEN>" \
sglang \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
# you can add any arguments from: https://docs.vllm.ai/en/latest/serving/engine_args.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment