Skip to content

Instantly share code, notes, and snippets.

View merrymercy's full-sized avatar
:octocat:

Lianmin Zheng merrymercy

:octocat:
3.8k followers · 169 following
View GitHub Profile
@merrymercy
merrymercy / time_stats.md
Last active January 21, 2026 07:21
  1. remove all ad-hoc classes: Remove APIServingTimingMixin, RequestTimingMetricsMixin, TimeStats. Unify to the new API.
    • e.g. many time staus related in ReqState: finished_time
  2. remove ad-hoc code in the critical path. e.g.,
                # Record response sent time right before we send response.
              if not state.response_sent_to_client_ts:
                  state.response_sent_to_client_ts = time.time()
                  out["meta_info"][
                      "response_sent_to_client_ts"
] = state.response_sent_to_client_ts
@merrymercy
merrymercy / activation.py
Created September 14, 2024 21:22
"""
Copyright 2023-2024 SGLang Team
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
@merrymercy
merrymercy / gist:e3041f0029db232adbe845cf995b3290
Last active July 17, 2024 22:00
intern-lm-perf

model: InternLM-7B

python3 benchmark_serving.py --backend openai --host 127.0.0.1 --port 30000 --dataset ShareGPT_V3_unfiltered_cleaned_split.json --model internlm/internlm2-chat-7b --tokenizer internlm/internlm2-chat-7b --num-prompts 3000 --trust-remote-code

H100 w/o streaming

SGLang

Successful requests: 3000
@merrymercy
merrymercy / event_overlap.py
Created May 9, 2024 19:58
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
import jax.numpy as jnp
nbytes = 1024*1024*32
data_type = cp.float32
@merrymercy
merrymercy / two_recv_overlap.py
Created May 9, 2024 14:06
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
import jax.numpy as jnp
nbytes = 1024*1024*32
data_type = cp.float32
@merrymercy
merrymercy / interleaved_overlap.py
Last active May 9, 2024 11:53
# mpirun -np 2 python p2p-nonblocking.py
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
import jax.numpy as jnp
@merrymercy
merrymercy / test_cp_nccl_jax_compute.py
Created May 9, 2024 05:19
# mpirun -np 2 python p2p-nonblocking.py
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
import jax.numpy as jnp
@merrymercy
merrymercy / test_nccl_overlap.py
Last active May 8, 2024 17:09
# mpirun -np 2 python p2p-nonblocking.py
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
nbytes = 1024*1024*32
data_type = cp.float32
@merrymercy
merrymercy / test_nccl.py
Created May 2, 2024 01:08
# mpirun -np 2 python p2p-nonblocking.py
import cupy as cp
import cupy.cuda.nccl as nccl
from mpi4py import MPI
import time
import os
nbytes = 1024*1024*32
data_type = cp.float32
@merrymercy
merrymercy / redirect.py
Created March 21, 2023 00:57
Permanently redirect to another url.
from fastapi import FastAPI
from starlette.responses import RedirectResponse
app = FastAPI()
@app.get("/")
async def redirect():
response = RedirectResponse(url="https://alpa-projects.github.io/opt", status_code=301)
return response