Let's get Tanuki-8x8B running | NEXTY Electronics, an electronic components and semiconductor trading company

Introduction

"Tanuki" is a local LLM program from Japan that is attracting attention in the industry.

Hello, this is a member of the Development Department at Nexty Electronics.
I am in charge of technical support for GPU Advanced Test Drive (GAT), which allows for highly flexible trials of various GPU servers.

GPU Advanced Test Drive (GAT)

I'm sure you've all witnessed the recent rise in language generation AI such as ChatGPT, Gemini, and Claude.
Meanwhile, local LLMs (Large Language Models), which run these language models in a local environment, are also making rapid progress.

The other day, the purely domestically produced Tanuki was announced for this local LLM.
This model is available for commercial use and is attracting a lot of attention for its high accuracy in responding to Japanese questions.

Tanuki

In this column, we will provide detailed instructions on how to actually run the high-precision version of the two publicly available models, "Tanuki-8x8B," on a PC, and also provide a simple benchmark based on actual trials on various GPU servers handled by GAT.

Required Hardware

The following PC and GPU are required:

The GPU we are targeting this time is made by NVIDIA.
The GPU memory will be 8x8B (AWQ quantized version), so about 40G is required.
If you don't have enough GPU memory, it may work with Tanuki-8B, though this article will not cover that.
If multiple GPUs are installed, the total recommended GPU memory is approximately 40GB.
The OS is based on Ubuntu, but it can run on Ubuntu with Windows WSL2.

The PC environment used to check operation on Nexty Electronics is as follows.

Ubuntu 20.04.6 LTS
13th Gen Intel Core i5-13600K
128GB Memory
RTX6000Ada(48GB) Driver Version 535.183.01

Preparation (Building a Docker environment)

As a preliminary step, please install docker (with GPU environment).

1. For Ubuntu

We recommend the official apt installation instructions.

Official apt installation instructions

After that, you need to install nvidia-container-toolkit (this will allow you to use the GPU from Docker)

2. For Windows

Please install docker-desktop (Windows version)
(If you enable "WSL2 Integration" during installation, you can use it from Ubuntu on WSL.)

On Windows, no additional installation of nvidia-container-toolkit is required.
Please make sure you can use ubuntu20.04 or ubuntu22.04 with wsl2.

Building a virtual vllm environment with Docker

Tanuki-8x8B uses a modified version of vllm.
We will prepare a virtual environment using Docker in which we can run this.

Modified version of vllm

Save the following text as "Dockerfile".
(It's long, but you can easily copy and paste it from here on out without needing to include prompts.)

FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu20.04
ENV DEBIAN_FRONTEND noninteractive
ENV CONDA_DIR /opt/conda
ENV PATH /usr/local/cuda/bin:$CONDA_DIR/bin:$PATH

#Env(for in docker build)
ENV CPATH /usr/local/cuda-12.1/include:$CPATH
ENV LIBRARY_PATH /usr/local/cuda-12.1/lib64:$LIBRARY_PATH
ENV LD_LIBRARY_PATH /usr/local/cuda-12.1/lib64:$LD_LIBRARY_PATH
ENV PATH /usr/local/cuda-12.1/bin:$PATH
ENV CUDA_HOME /usr/local/cuda-12.1

# System dependencies
RUN apt-get update && apt-get install -y \
wget \
build-essential g++ gcc \
libgl1-mesa-glx libglib2.0-0 \
openmpi-bin openmpi-common libopenmpi-dev libgtk2.0-dev git \
gnupg2 curl software-properties-common \
cmake\
protobuf-compiler\
libprotobuf-dev\
&& rm -rf /var/lib/apt/lists/*

# Install Miniconda
RUN wget --quiet https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh && \
/bin/bash ~/miniconda.sh -b -p $CONDA_DIR

# Python packages
RUN conda install -y python=3.10

# Python packages2
RUN pip install transformers accelerate bitsandbytes

# vLLM(modified) build and install
RUN git clone https://github.com/team-hatakeyama-phase2/vllm.git && \
cd vllm && \
LD_LIBRARY_PATH="" MAX_JOBS=16 pip install -e . && \
pip install --no-build-isolation flash_attn

A brief explanation of the Dockerfile is given below.

I chose this because there was an example of operation confirmation in the cuda12 + python3.10 environment.

Example of operation check in cuda12 + python3.10 environment

The modified vllm is cloned in Docker and installed in Docker.

Create a Docker image with the following command:
The -t part can be anything, but it will give a nickname to the image that will be created.
(This article will explain using "vllm-nexty")

docker build . -f Dockerfile -t vllm-nexty

Running Tanuki-8x8B

Basic script

Save the script that performs the following basic execution as a file such as vllm_test.py. The destination directory is assumed to be "/home/xxx/tanuki".

To create this script, I referred to the official inference method explanation page and the page Trying out Tanuki-8x8B-dpo-v1.0 on WSL2.

from time import time
from vllm import LLM, SamplingParams

#Depending on your environment, you may need to change it to the following:
#from vllm.entrypoints.llm import LLM
#from vllm.sampling_params import SamplingParams

#Model name AWQ quantized version used
model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ"
#model_name = "weblab-GENIAC/Tanuki-8x8B-dpo-v1.0"
#model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-GPTQ-4bit"
#model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-GPTQ-8bit"

#Starting vllm
#Adjust the gpu_memory_utilization part according to your environment (reduce the percentage when overflow occurs)
vllm = LLM(model_name, trust_remote_code=True, tensor_parallel_size=1, gpu_memory_utilization=0.8) # For 1 GPU
#vllm = LLM(model_name, trust_remote_code=True, tensor_parallel_size=2) # When using 2 GPUs
tokenizer = vllm.get_tokenizer()

#Input settings
messages = [
{"role": "system", "content": "The following are instructions that describe the task. Write a response that satisfies the request appropriately."},
{"role": "user", "content": "What are some recommended tourist spots in Japan?"}
]

inputs_text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(f"inputs_text: {inputs_text}")

sampling_params = SamplingParams(temperature=0.0, max_tokens=1024, seed=1, repetition_penalty=1.1)

#execution
start = time()
outputs = vllm.generate(inputs_text, sampling_params=sampling_params, use_tqdm=False)
end = time()

#display
outputs_text = outputs[0].outputs[0].text
print(f"outputs_text: {outputs_text}")
print(f"Elapsed time: {(end - start):.4f} sec.")

#Display execution speed etc.
input_tokens = len(outputs[0].prompt_token_ids)
output_tokens = len(outputs[0].outputs[0].token_ids)
total_time = end - start
tps = output_tokens / total_time
print(f"prompt tokens = {input_tokens:.7g}")
print(f"output tokens = {output_tokens:.7g} ({tps:f} [tps])")
print(f" total time = {total_time:f} [s]")

#end of this program

When you run this program, the following will happen automatically:

1. (First time only) The trained model data for Tanuki-8x8B (AWQ quantized version) will be automatically loaded from the huggingface site into ~/.cache/.
2. The trained model data is deployed on the GPU (approximately 25GB is required).
3. The first message specifies the role (as officially specified), and the second contains an example of an actual question (recommended tourist spots in Japan), which will be used as input for VLLM.
4. The output for the input is displayed as text, and the execution time, tokens per second (TPS), etc. are also displayed.

This will run in an environment where the modified version of vllm created with the Dockerfile above is installed. Let's run it right away.

Starting Docker and running basic scripts

Start the docker image you just created with the following script:

#!/bin/bash D_WORK_DIR=/home/xxx/tanuki docker run --rm --name vllm-nexty-run --gpus all -it --network host \ -w "${D_WORK_DIR}" \ -v "${D_WORK_DIR}":"${D_WORK_DIR}" \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --shm-size 64g vllm-nexty /bin/bash

Below is an explanation of the docker startup script.

In D_WORK_DIR, the directory containing the above basic scripts is set to be shared with Docker.
In addition, by sharing ~/.cache with the Docker side, the storage of trained model data is shared with outside of Docker.
By starting with the --rm option, the setting will not be left behind when the program is shut down (only used as the operating environment for the modified version of vllm).
We launch a Docker image called vllm-nexty and give it the nickname vllm-nexty-run.

If it starts successfully, you will see the root prompt in Docker like below.

root@machine:/home/xxx/tanuki#

Now, run "vllm_test.py", which is located in the shared startup directory.

python vllm_test.py

Run it and if you get the following output, it's successful.

INFO 09-06 07:08:42 awq_marlin.py:89] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 09-06 07:08:42 llm_engine.py:184] Initializing an LLM engine (v0.5.4) with config: model='team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ', speculative_config=None, tokenizer='team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 09-06 07:08:48 model_runner.py:886] Starting to load model team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ...
INFO 09-06 07:08:49 weight_utils.py:231] Using model weights format ['*.safetensors']
533M/533M [03:02<00:00, 2.23MB/s]
4.74G/4.74G [28:34<00:00, 2.67MB/s]
4.98G/4.98G [30:06<00:00, 2.68MB/s]
model-00002-of-00006.safetensors: 100%|██ █ 2.66MB/s]
model-00003-of-00006.safetensors: 100%|██ █ 2.60MB/s]
model-00001-of-00006.safetensors: 100%|██ █ 2.62MB/s]
model.safetensors.index.json: 100%|███████████████████████████████████████████████████████████████████| 273k/273k [00:00<00:00, 603kB/s]
Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00 <?, ?it/s]██████████████████████| 4.97G/4.97G [30:47<00:00, 6.64MB/s]
INFO 09-06 07:44:37 awq_marlin.py:89] The model is convertible to awq_marlin during runtime. Using awq_marlin kernel.
INFO 09-06 07:44:37 llm_engine.py:184] Initializing an LLM engine (v0.5.4) with config: model='team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ', speculative_config=None, tokenizer='team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=awq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ, use_v2_block_manager=False, enable_prefix_caching=False)
INFO 09-06 07:44:37 model_runner.py:886] Starting to load model team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ...
INFO 09-06 07:44:38 weight_utils.py:231] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards: 0% Completed | 0/6 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 17% Completed | 1/6 [00:00<00:03, 1.50it/s]
Loading safetensors checkpoint shards: 33% Completed | 2/6 [00:01<00:02, 1.36it/s]
Loading safetensors checkpoint shards: 50% Completed | 3/6 [00:02<00:02, 1.39it/s]
Loading safetensors checkpoint shards: 67% Completed | 4/6 [00:02<00:01, 1.39it/s]
Loading safetensors checkpoint shards: 83% Completed | 5/6 [00:03<00:00, 1.94it/s]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:03<00:00, 1.78it/s]
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:03<00:00, 1.63it/s]

INFO 09-06 07:44:43 model_runner.py:898] Loading model weights took 23.4735 GB
INFO 09-06 07:44:46 gpu_executor.py:103] # GPU blocks: 6800, # CPU blocks: 2048
INFO 09-06 07:44:48 model_runner.py:1193] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 09-06 07:44:48 model_runner.py:1197] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 09-06 07:45:02 model_runner.py:1394] Graph capturing finished in 14 secs.
inputs_text: Below are instructions that describe the task. Write a response that satisfies the requirement appropriately.

### Instructions:
What are some recommended tourist spots in Japan?

### Response:

outputs_text: Of course. There are many attractive tourist destinations in Japan, but here are some places we particularly recommend.

1. Kyoto
- Kiyomizu-dera Temple: A historic temple registered as a World Heritage Site, it features beautiful wooden architecture and gardens.
- Kinkakuji (Rokuonji): The golden building reflected in the pond is a spectacular sight.
- Fushimi Inari Taisha Shrine: Famous for its tunnel of red torii gates known as "Senbon Torii."

2. Tokyo
- Sensoji Temple: One of Tokyo's oldest temples, it also features the Kaminarimon Gate and Nakamise Shopping Street.
- Meiji Shrine: This shrine is surrounded by a vast forest and has a charming, tranquil atmosphere.
- Odaiba: A futuristic urban landscape and entertainment venues.

3. Hokkaido
- Sapporo Snow Festival: A huge winter event featuring giant ice and snow sculptures.
- Furano and Biei: Beautiful scenery with lavender fields and seasonal flowers.
- Otaru Canal: Enjoy the romantic night view and historic warehouse district.

4. Okinawa
- Churaumi Aquarium: Home to one of the world's largest aquariums, you can get up close and personal with marine life, including whale sharks.
- Shuri Castle: A magnificent castle ruin where you can feel the history of the Ryukyu Kingdom.
- Kouri Island: A small island surrounded by emerald green waters, where you can enjoy driving and strolling along the beach.

5. Nara
- Todaiji Temple: Don't miss the huge Buddha statue enshrined in the Great Buddha Hall.
- Kasuga Taisha Shrine: Registered as a World Heritage Site, it boasts a beautiful group of shrine buildings.
- Mount Yoshino: Known as a famous cherry blossom viewing spot, the area is lined with beautiful rows of cherry blossom trees in spring.

Each of these destinations has its own unique charm and is worth a visit, so choose the one that best suits your travel needs and the season.
Elapsed time: 17.8904 sec.
prompt tokens = 54
output tokens = 511 (28.562876 [tps])
total time = 17.890356 [s]

The model data was automatically downloaded and answered all the questions we asked!

Advanced: Chat-style responses

The basic script will exit after answering the pre-set questions.
Although this is a simple test program on CUI, we will modify it a little so that it can respond continuously to text input (in this script, it will always be in the form of the first question). The modified version is described below, so save it as "vllm_chat.py" and try running it.

from time import time
from vllm import LLM, SamplingParams

#Depending on your environment, you may need to change it to the following:
#from vllm.entrypoints.llm import LLM
#from vllm.sampling_params import SamplingParams

#Model name AWQ quantized version used
model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-AWQ"
#model_name = "weblab-GENIAC/Tanuki-8x8B-dpo-v1.0"
#model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-GPTQ-4bit"
#model_name = "team-hatakeyama-phase2/Tanuki-8x8B-dpo-v1.0-GPTQ-8bit"

#Starting vllm
#Adjust the gpu_memory_utilization part according to your environment (reduce the percentage when overflow occurs)
vllm = LLM(model_name, trust_remote_code=True, tensor_parallel_size=1, gpu_memory_utilization=0.8) # For 1 GPU
#vllm = LLM(model_name, trust_remote_code=True, tensor_parallel_size=2) # When using 2 GPUs
tokenizer = vllm.get_tokenizer()

default_message = {"role": "system", "content": "The following are instructions that describe the task. Write a response that satisfies the requirements appropriately."}

#main loop =====================================================
while True:
print("input?>", end="")
key_input = input()

tmp_mes = [default_message, {"role": "user", "content": key_input}]
inputs_text = tokenizer.apply_chat_template(tmp_mes, tokenize=False, add_generation_prompt=True)
print(f"inputs_text: {inputs_text}")

sampling_params = SamplingParams(temperature=0.0, max_tokens=1024, seed=1, repetition_penalty=1.1)

#execution
start = time()
outputs = vllm.generate(inputs_text, sampling_params=sampling_params, use_tqdm=False)
end = time()

#output
outputs_text = outputs[0].outputs[0].text
print(f"outputs_text: {outputs_text}")
print(f"Elapsed time: {(end - start):.4f} sec.")

input_tokens = len(outputs[0].prompt_token_ids)
output_tokens = len(outputs[0].outputs[0].token_ids)
total_time = end - start
tps = output_tokens / total_time
print(f"prompt tokens = {input_tokens:.7g}")
print(f"output tokens = {output_tokens:.7g} ({tps:f} [tps])")
print(f" total time = {total_time:f} [s]")
print()

#end of this program

Below is an example of execution.

(Omitted)

input?>hello
inputs_text: Below are instructions that describe the task. Write a response that satisfies the requirement appropriately.

### Instructions:
Hello

### Response:

outputs_text: Hello! How can I help you today? Please feel free to reach out with any questions or concerns you may have.
Elapsed time: 1.0484 sec.
prompt tokens = 43
output tokens = 29 (27.660848 [tps])
total time = 1.048413 [s]

input?>What are some recommended tourist spots in the US?
inputs_text: Below are instructions that describe the task. Write a response that satisfies the requirement appropriately.

### Instructions:
What are your recommended tourist spots in the US?

### Response:

outputs_text: Of course. There are many attractive tourist destinations in the United States, but here are some places we particularly recommend.

1. New York City - This iconic American city is home to many attractions, including Times Square, Central Park, and the Statue of Liberty. Don't miss the Broadway musicals.

2. Grand Canyon National Park - Located in Arizona, the spectacular natural scenery is a breathtaking sight, perfect for hiking and photography.

3. Los Angeles - The heart of the film industry, Los Angeles also offers a variety of entertainment options, including Santa Monica Beach, Beverly Hills, and Disneyland Resort.

4. San Francisco - Beautiful scenery and historical sites abound, including the Golden Gate Bridge, Alcatraz Island, and Fisherman's Wharf. Don't miss CABLES cars.

5. Washington DC - The capital of the United States, it is a very interesting political and cultural center, home to the White House, the Capitol, and the Smithsonian Museums.

6. Seattle - Home to many unique tourist attractions, including the Space Needle, Pike Place Market, and the original Starbucks, the city also boasts Mount Rainier National Park, perfect for those who love outdoor activities.

7. New Orleans - Enjoy jazz music, the vibrant atmosphere of the French Quarter, and the famous Mardi Gras festival.

8. Chicago - Known as the "Windy City," the skyscrapers of Chicago are worth a visit, with views from Willis Tower. Millennium Park and Navy Pier are also worth a visit.

Each of these places has its own unique charm and will provide visitors with an unforgettable experience, so choose one that suits your travel purpose and interests.
Elapsed time: 13.8894 sec.
prompt tokens = 54
output tokens = 396 (28.510975 [tps])
total time = 13.889388 [s]

input?>

You can respond continuously just like in a chat (press Ctrl-C to end).
When executed locally, the trained model is deployed in GPU memory before the loop begins, resulting in very fast response times.

Speed on GAT servers

Nexty Electronics offers the GPU Advanced Test Drive (GAT) service, which allows for highly flexible testing of various GPU servers. This service can be used for pre-testing before deploying applications like local LLMs on-premises.

GPU Advanced Test Drive (GAT) Service

This time, we tried running Tanuki-8x8B (AWQ quantized version) on various servers deployed in the GAT service. We ran the basic script and created a table showing the TPS (tokens per second).

Server name	Number of parallel GPUs	TPS (Token per second)
DGX-H100	8	140.1555
DGX-H100	4	105.6157
DGX-H100	2	69.62028
DGX-H100	1	42.23749
DGX-A100	1	29.92136
GSV (RTX6000Ada+A800)	2	48.57265
GSV (RTX6000Ada+A800)	1	28.22094

The higher the TPS number, the faster the speed. The fastest DGX-H100, with 8 parallel connections, achieved a high execution speed of 140 tokens per second, and in the case of the recommended tourist spots in Japan mentioned earlier, it was able to respond with that many characters in machine gun-like speech in just 3.6 seconds.

Conclusion

This time, we actually ran "Tanuki-8x8B," a domestically produced LLM for local use. We explained how to run it and how to execute scripts. Running it yourself can give you a deeper understanding and allow you to see the finer details. Furthermore, with this level of precision and speed, we think various applications and developments are possible.

The NEXTY Electronics Development Department investigates cutting-edge technology trends and conducts numerous trials at the edge, including on PCs, GPU servers, and NVIDIA DRIVE Orin.

Furthermore, Nexty Electronics also provides technical support related to AI, as well as server trial environment services (GPU Advanced Test Drive: GAT) such as the DGX-H100/A100 for training and execution. Please feel free Inquiry.

Server trial environment service (GPU Advanced Test Drive: GAT)

Let's run Tanuki-8x8B