파란크리스마스

출처

• Qwen3.8 27B | Jetson AI Lab
• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

llama.cpp 설치 확인

bluesanta@localhost:~$ llama-cli --version
version: 0.1.2-dev (build 10547, commit 749f688fc)
built with GNU 13.3.0 for Linux aarch64

llama.cpp 설치 확인

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ llama-cli -m ./models/Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj ./models/Qwen3.8-27B-mmproj-F16.gguf -ngl 999 --spec-type draft-mtp -p "한글은 누가 만들었어?"
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b10547-749f688fc
model      : ./models/Qwen3.8-27B-UD-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text, vision, video
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
  /image        add an image file
  /video        add a video file
 
 
 
> 한글은 누가 만들었어?
 
[Start thinking]
 
The user is asking in Korean: "Who created Hangul?" This is a straightforward factual question about the history of the Korean writing system.
 
Hangul (한글) was created by King Sejong the Great (세종대왕) and his scholars at the Jiphyeonjeon (집현전) in 1443 (Joseon dynasty). It was officially promulgated in 1446 in the document "Hunminjeongeum" (훈민정음, The Proper Sounds for the Instruction of the People).
 
Let me respond in Korean since the question was asked in Korean.
[End thinking]
 
한글은 조선 제4대 왕인 **세종대왕**(1397~1450)이 1443년에 창제하고, 1446년 **《훈민정음》** 을 반포하면서 세상에 알려졌습니다.
 
세종대왕은 백성들이 한자를 배우기 어려워하는 현실을 안타깝게 여겨, 배우기 쉽면서도 음가를 정확히 표기할 수 있는 문자를 만들 것을 지시했습니다. 세종대왕을 중심으로 **예종(예하), 성삼문, 신숙주, 정인지, 박팽년, 이개** 등 집현전 학자들이 함께 연구·창제에 참여했습니다.
 
한글은 자음의 모양이 발음 기관의 형태를 닮고(예: ㄱ = 혀가 윗니를 막는 모양), 모음의 모양이 하늘·땅·사람의 이치를 담은 **원리 문자**라는 점에서, 문자학적으로도 매우 독창적인 체계로 평가받고 있습니다.
 
[ Prompt: 61.8 t/s | Generation: 9.7 t/s ]
 
> 

서비스 등록

작업 디렉토리

bbluesanta@localhost:~$ sudo mkdir -p /opt/llama.cpp
bluesanta@localhost:~$ sudo chown bluesanta:bluesanta -R /opt/llama.cpp

서비스 파일 생성

bluesanta@localhost:~$ sudo vi /etc/systemd/system/llama.service

[Unit]
Description=Llama.cpp Server Service
After=network.target

[Service]
# 사용자 계정
User=bluesanta
Group=bluesanta
LimitMEMLOCK=infinity
WorkingDirectory=/opt/llama.cpp

# 최적화된 실행 명령어
# --ctx-size 131072 -> 262144 -> 196608
# --spec-type ngram-mod,draft-mtp --spec-draft-n-max 4
ExecStart=/usr/local/bin/llama-server \
    -m /home/bluesanta/llm/models/Qwen3.8-27B-UD-Q4_K_M.gguf \
    --mmproj /home/bluesanta/llm/models/Qwen3.8-27B-mmproj-F16.gguf \
    -ngl all \
    --spec-type draft-mtp \
    --temp 1.0 \
    --top-k 20 \
    --min-p 0.0 \
    --ctx-size 196608 \
    --n-gpu-layers 99 \
    --flash-attn on \
    --mlock \
    --cont-batching \
    --metrics \
    --image-min-tokens 1024 \
    --reasoning-preserve \
    --host 0.0.0.0 \
    --port 8000

# 프로세스 종료 시 자동 재시작 설정
# Restart=always
# RestartSec=5

[Install]
WantedBy=multi-user.target

서비스 등록

bluesanta@localhost:~$ sudo systemctl enable llama.service
Created symlink /etc/systemd/system/multi-user.target.wants/llama.service → /etc/systemd/system/llama.service.

서비스 갱신

bluesanta@localhost:~$ sudo systemctl daemon-reload

서비스 실행

bluesanta@localhost:~$ sudo systemctl start llama

서비스 상태 확인

bluesanta@localhost:~$ sudo systemctl status llama

서비스 로그 확인

bluesanta@localhost:~$ sudo journalctl -u llama.service -f

확인

bluesanta@localhost:~$ curl http://localhost:8000/completion -H "Content-Type: application/json" -d '{
  "prompt": "Jetson AGX Orin의 장점 3가지는?",
  "n_predict": 256
}'
{"index":0,"content":"\n\n\nThe user is asking about 3 advantages of the Jetson AGX Orin. Let me think about the key advantages of this NVIDIA embedded AI computing platform.\n\nNVIDIA Jetson AGX Orin is an embedded AI computing module/platform. Let me recall its key advantages:\n\n1. **High AI Performance (TOPS)**: The Jetson AGX Orin delivers up to 275 TOPS of AI performance (the developer kit), making it one of the most powerful embedded AI processors available. It uses a custom Arm Cortex-A78AE CPU and Ampere GPU with 2048 CUDA cores and 64 Tensor cores.\n\n2. **Power Efficiency**: It achieves this high performance while being power efficient, consuming power in the range of 15W to 60W (configurable). This makes it suitable for edge/embedded applications where power is constrained (robots, drones, autonomous vehicles).\n\n3. **Full Software Ecosystem / Developer Support**: It runs on the NVIDIA JetPack SDK which includes CUDA, cuDNN, TensorRT, and the broader NVIDIA AI software stack. There's extensive developer support, documentation, and a large community. It supports multiple AI frameworks (PyTorch,","tokens":[],"id_slot":3,"stop":true,"model":"/home/bluesanta/llm/models/Qwen3.8-27B-UD-Q4_K_M.gguf","tokens_predicted":256,"tokens_evaluated":13,"generation_settings":{"seed":4294967295,"temperature":1.0,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":20,"top_p":0.949999988079071,"min_p":0.0,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":64,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"adaptive_target":-1.0,"adaptive_decay":0.8999999761581421,"stop":[],"max_tokens":256,"n_predict":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"generation_prompt":"","samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.types":"none,draft-mtp","timings_per_token":false,"post_sampling_probs":false,"backend_sampling":false,"lora":[]},"prompt":"Jetson AGX Orin의 장점 3가지는?","has_new_line":true,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":268,"timings":{"cache_n":0,"prompt_n":13,"prompt_ms":719.45,"prompt_per_token_ms":55.3423076923077,"prompt_per_second":18.069358537771908,"predicted_n":256,"predicted_ms":23792.995,"predicted_per_token_ms":93.30586274509804,"predicted_per_second":10.717440154129399,"draft_n":306,"draft_n_accepted":153}}

출처

• GitHub - hasso5703/dgx-spark-qwen38: Fastest measured Qwen3.8-27B config for DGX Spark (GB10): SGLang + NVFP4 + DFlash2, deterministic boots, 50 tok/s greedy median, 148 tok/s at 8 streams, 258 at 32. One command, everything pinned, lossless. · GitHub

SGLang 설치

bluesanta@gx10-3b16:~/llm$ git clone https://github.com/sgl-project/sglang.git
bluesanta@gx10-3b16:~/llm$ cd sglang

Rust 툴체인 설치

(.venv) bluesanta@gx10-3b16:~/llm/sglang$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
 
Rust is installed now. Great!
 
To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).
 
To configure your current shell, you need to source
the corresponding env file under $HOME/.cargo.
 
This is usually done by running one of the following (note the leading DOT):
. "$HOME/.cargo/env"            # For sh/bash/zsh/ash/dash/pdksh
source "$HOME/.cargo/env.fish"  # For fish
source "~/.cargo/env.nu"  # For nushell
source "$HOME/.cargo/env.tcsh"  # For tcsh
. "$HOME/.cargo/env.ps1"        # For pwsh
source "$HOME/.cargo/env.xsh"   # For xonsh

Rust 툴체인 설치

(.venv) bluesanta@gx10-3b16:~/llm/sglang$ source $HOME/.cargo/env

SGLang Python 래퍼 설치

(.venv) bluesanta@gx10-3b16:~/llm/sglang$ pip install -e "python"

GB10 전용 커널 설치 (sglang-kernel)

(.venv) bluesanta@gx10-3b16:~/llm/sglang$ pip install sglang-kernel --index-url https://docs.sglang.io/whl/cu130 --upgrade

sglang 실행

python -m sglang.launch_server \
  --model-path /home/bluesanta/llm/models/radixArkQwen3.8-27B-NVFP4 --tp-size 1 \
  --speculative-draft-model-path /home/bluesanta/llm/models/z-labQwen3.8-27B-DFlash2 \
  --served-model-name qwen3.8-27b \
  --mem-fraction-static 0.50 \
  --attention-backend flashinfer --chunked-prefill-size 8192 \
  --disable-prefill-cuda-graph --cuda-graph-max-bs-decode 4 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --max-mamba-cache-size 96 --max-running-requests 8 \
  --num-continuous-decode-steps 2 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --api-key "bluesanta" \
  --host 0.0.0.0 --port 8000

sglang 테스트

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer bluesanta" \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant. Always answer in Korean."},
      {"role": "user", "content": "양자역학의 불확정성 원리에 대해 초등학생이 이해할 수 있게 짧고 쉽게 설명해줄래?"}
    ],
    "max_tokens": 1024,
    "temperature": 0.5,
    "repetition_penalty": 1.15
  }'

서비스 등록

SGLang 실행 파일 생성

bluesanta@gx10-3b16:~/llm$ vi ~/llm/start_sglang.sh

start_sglang.sh

#!/bin/bash

# 1. 가상환경 활성화
source /home/bluesanta/llm/.venv/bin/activate

# 2. SGLang 서버 실행 (OpenAI 호환 API 서버가 자동으로 켜집니다)
exec python -m sglang.launch_server \
  --model-path /home/bluesanta/llm/models/radixArkQwen3.8-27B-NVFP4 --tp-size 1 \
  --speculative-draft-model-path /home/bluesanta/llm/models/z-labQwen3.8-27B-DFlash2 \
  --served-model-name qwen3.8-27b \
  --mem-fraction-static 0.50 \
  --attention-backend flashinfer --chunked-prefill-size 8192 \
  --disable-prefill-cuda-graph --cuda-graph-max-bs-decode 4 \
  --mamba-radix-cache-strategy extra_buffer_lazy --mamba-ssm-dtype bfloat16 \
  --max-mamba-cache-size 96 --max-running-requests 8 \
  --num-continuous-decode-steps 2 \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --api-key "bluesanta" \
  --host 0.0.0.0 --port 8000

start_sglang.sh 실행 권한을 부여

(.venv) bluesanta@gx10-3b16:~/llm$ chmod +x ~/llm/start_sglang.sh

서비스 등록 파일 생성

(.venv) bluesanta@gx10-3b16:~/llm$ sudo vi /etc/systemd/system/sglang.service

vsglang.service

[Unit]
Description=SGLang API Server (Qwen3.8)
After=network.target

[Service]
User=bluesanta
WorkingDirectory=/home/bluesanta/llm
ExecStart=/home/bluesanta/llm/start_sglang.sh

Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

서비스 등록

(.venv) bluesanta@gx10-3b16:~/llm$ sudo systemctl enable sglang.service
Created symlink /etc/systemd/system/multi-user.target.wants/sglang.service → /etc/systemd/system/sglang.service.

서비스 시작

(.venv) bluesanta@gx10-3b16:~/llm$ sudo systemctl start sglang

서비스 로그 확인

(.venv) bluesanta@gx10-3b16:~/llm$ sudo journalctl -u sglang -f

출처

• 千问3.8-27B · 模型库
• Qwen/Qwen3.8-27B | vLLM Recipes
• Models | Jetson AI Lab
• TensorRT-LLM/examples/models/core/qwen/README.md at main · NVIDIA/TensorRT-LLM

CUDA 버전 확인

bluesanta@gx10-3b16:~$ uname -a
Linux gx10-3b16 6.17.0-1029-nvidia #29-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul  1 00:13:52 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@gx10-3b16:~$ cat /etc/dgx-release
DGX_NAME="DGX Spark"
DGX_PRETTY_NAME="NVIDIA DGX Spark"
DGX_SWBUILD_DATE="2026-01-26-16-04-58"
DGX_SWBUILD_VERSION="7.4.0"
DGX_COMMIT_ID="6c74fd3"
DGX_PLATFORM="GX10"
DGX_SERIAL_NUMBER="TAMSAG002278D8C"
 
DGX_OTA_VERSION="7.5.0"
DGX_OTA_DATE="Mon May 18 23:15:23 KST 2026"
bluesanta@gx10-3b16:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
NVIDIA GB10, 12.1, 580.173.02
bluesanta@gx10-3b16:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Fri_May_08_10:52:52_AM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.86
Build cuda_13.2.r13.2/compiler.37953736_0

vLLM

vLLM 설치

(.venv) bluesanta@gx10-3b16:~/llm$ pip install vllm==0.27.1 --no-deps

vLLM 실행

exec /home/bluesanta/llm/.venv/bin/vllm serve /home/bluesanta/llm/models/Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.8 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 16384 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 3}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 4, "video": 0}' \
  --mm-processor-cache-type shm \
  --mm-processor-kwargs '{"max_pixels": 602112, "min_pixels": 3136}'

vLLM 실행 테스트

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"qwen3.8-27b",
  "messages":[{"role":"user","content":"1부터 30까지 소수의 합은?"}],
  "temperature":0.6,"top_p":0.95,"top_k":20,"max_tokens":800}' \
| python3 -c "import sys,json;m=json.load(sys.stdin)['choices'][0]['message'];print('THINK:',(m.get('reasoning_content') or '')[:200]);print('ANSWER:',m.get('content'))"

(.venv) bluesanta@gx10-3b16:~/llm$ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model":"qwen3.8-27b",
  "messages":[{"role":"user","content":"한글은 누가 만들었어?"}],
  "temperature":0.6,"top_p":0.95,"top_k":20,"max_tokens":800}' | python3 -c "import sys,json;m=json.load(sys.stdin)['choices'][0]['message'];print('THINK:',(m.get('reasoning_content') or '')[:200]);print('ANSWER:',m.get('content'))"
THINK: 
ANSWER: 한글(한자음)은 **조선 제4대 왕인 세종대왕**이 주도로 제작한 문자입니다.
 
세종대왕은 백성들이 한자를 배우기 어렵다는 점을 안타깝게 여겨, 1443년(세종 25년)에 훈도(訓度)라는 이름으로 문자 제작을 명했고, 1446년(세종 28년)에 **『훈민정음』**이라는 제목으로 정식으로 반포했습니다.
 
이 문자는 과학적이고 체계적인 원리(자음의 모양이 발음 기관의 형태를 반영, 모음의 모양이 음운의 원리를 반영)에 기반하여 설계되었으며, 이후 '한글'이라는 이름으로 널리 사용되고 있습니다.

서비스 등록

vLLM 실행 파일 생성

bluesanta@gx10-3b16:~/llm$ vi ~/llm/start_vllm.sh

start_vllm.sh

#!/bin/bash

# 사용자의 기본 환경 변수 로드 (CUDA 경로 등이 .bashrc에 있다면)
# source /home/bluesanta/.bashrc

# 가상환경 활성화
source /home/bluesanta/llm/.venv/bin/activate

# vLLM 실행 (exec를 사용하여 프로세스를 대체해야 systemd가 정상적으로 PID를 추적합니다)
exec vllm serve /home/bluesanta/llm/models/Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.72 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 16384 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}' \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --limit-mm-per-prompt '{"image": 4, "video": 0}' \
  --mm-processor-cache-type shm \
  --mm-processor-kwargs '{"max_pixels": 602112, "min_pixels": 3136}' \
  --safetensors-load-strategy=prefetch

서비스 등록 파일 생성

bluesanta@gx10-3b16:~$ sudo vi /etc/systemd/system/vllm-qwen38.service

vllm-qwen38.service

[Unit]
Description=vLLM Qwen3.8-27B Service
After=network.target

[Service]
User=bluesanta
WorkingDirectory=/home/bluesanta/llm

# 이전에 설정한 환경 변수 파일이 있다면 주석을 해제하고 사용하세요.
# EnvironmentFile=/home/bluesanta/llm/vllm.env

# Python 대신 쉘 스크립트를 직접 실행하도록 변경
ExecStart=/home/bluesanta/llm/start_vllm.sh

Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

서비스 등록

bluesanta@gx10-3b16:~$ sudo systemctl enable vllm-qwen38.service 
Created symlink /etc/systemd/system/multi-user.target.wants/vllm-qwen38.service → /etc/systemd/system/vllm-qwen38.service.

서비스 시작

bluesanta@gx10-3b16:~$ sudo systemctl start vllm-qwen38

서비스 로그 확인

(.venv) bluesanta@gx10-3b16:~/llm$ sudo journalctl -u vllm-qwen38 -f

출처

• TensorRT 11.x Download | NVIDIA Developer

현재 설치된 버전 확인

bluesanta@localhost:~$ dpkg -l | grep nvinfer
ii  libnvinfer-bin                                     10.16.2.10-1+cuda13.2                            arm64        TensorRT binaries
ii  libnvinfer-dev                                     10.16.2.10-1+cuda13.2                            arm64        TensorRT development libraries
ii  libnvinfer-dispatch-dev                            10.16.2.10-1+cuda13.2                            arm64        TensorRT development dispatch runtime libraries
ii  libnvinfer-dispatch10                              10.16.2.10-1+cuda13.2                            arm64        TensorRT dispatch runtime library
ii  libnvinfer-headers-dev                             10.16.2.10-1+cuda13.2                            arm64        TensorRT development headers
ii  libnvinfer-headers-plugin-dev                      10.16.2.10-1+cuda13.2                            arm64        TensorRT plugin headers
ii  libnvinfer-headers-python-plugin-dev               10.16.2.10-1+cuda13.2                            arm64        TensorRT Python plugin development headers
ii  libnvinfer-lean-dev                                10.16.2.10-1+cuda13.2                            arm64        TensorRT lean runtime libraries
ii  libnvinfer-lean10                                  10.16.2.10-1+cuda13.2                            arm64        TensorRT lean runtime library
ii  libnvinfer-plugin-dev                              10.16.2.10-1+cuda13.2                            arm64        TensorRT plugin libraries
ii  libnvinfer-plugin10                                10.16.2.10-1+cuda13.2                            arm64        TensorRT plugin libraries
ii  libnvinfer-safe-headers-dev                        10.16.2.10-1+cuda13.2                            arm64        TensorRT safety development headers
ii  libnvinfer-vc-plugin-dev                           10.16.2.10-1+cuda13.2                            arm64        TensorRT vc-plugin library
ii  libnvinfer-vc-plugin10                             10.16.2.10-1+cuda13.2                            arm64        TensorRT vc-plugin library
ii  libnvinfer10                                       10.16.2.10-1+cuda13.2                            arm64        TensorRT runtime libraries
ii  python3-libnvinfer                                 10.16.2.10-1+cuda13.2                            arm64        Python 3 bindings for TensorRT standard runtime
ii  python3-libnvinfer-dev                             10.16.2.10-1+cuda13.2                            arm64        Python 3 development package for TensorRT standard runtime
ii  python3-libnvinfer-dispatch                        10.16.2.10-1+cuda13.2                            arm64        Python 3 bindings for TensorRT dispatch runtime
ii  python3-libnvinfer-lean                            10.16.2.10-1+cuda13.2                            arm64        Python 3 bindings for TensorRT lean runtime

NVIDIA TensorRT 11.x 설치

다운로드

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/tensorrt/11.2.1/local_installers/nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3_1.0-1_arm64.deb

deb 로컬 리포지토리 등록

bluesanta@localhost:~$ sudo dpkg -i nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3_1.0-1_arm64.deb
Selecting previously unselected package nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3.
(Reading database ... 204342 files and directories currently installed.)
Preparing to unpack nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3_1.0-1_arm64.deb ...
Unpacking nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3 (1.0-1) ...
Setting up nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3 (1.0-1) ...
 
The public nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3 GPG key does not appear to be installed.
To install the key, run this command:
sudo cp /var/nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3/nv-tensorrt-local-019AD499-keyring.gpg /usr/share/keyrings/

GPG 키링 복사

bluesanta@localhost:~$ sudo cp /var/nv-tensorrt-local-repo-ubuntu2404-11.2.1-cuda-13.3/nv-tensorrt-local-019AD499-keyring.gpg /usr/share/keyrings/

패키지 인덱스 업데이트 및 TensorRT 11 설치

bluesanta@localhost:~$ sudo apt updateㅍ
bluesanta@localhost:~$ sudo apt remove --purge -y "libnvinfer*" "python3-libnvinfer*" "tensorrt*"
bluesanta@localhost:~$ sudo apt-get autoremove -y
bluesanta@localhost:~$ sudo apt-get install -y libnvinfer11=11.2.1.2-1+cuda13.3 libnvinfer-dev=11.2.1.2-1+cuda13.3 libnvinfer-plugin11=11.2.1.2-1+cuda13.3 libnvinfer-plugin-dev=11.2.1.2-1+cuda13.3 libnvinfer-headers-dev=11.2.1.2-1+cuda13.3 libnvinfer-headers-plugin-dev=11.2.1.2-1+cuda13.3 libnvinfer-safe-headers-dev=11.2.1.2-1+cuda13.3 libnvinfer-lean11=11.2.1.2-1+cuda13.3 libnvinfer-lean-dev=11.2.1.2-1+cuda13.3 libnvinfer-dispatch11=11.2.1.2-1+cuda13.3 libnvinfer-dispatch-dev=11.2.1.2-1+cuda13.3 libnvinfer-vc-plugin11=11.2.1.2-1+cuda13.3 libnvinfer-vc-plugin-dev=11.2.1.2-1+cuda13.3 libnvonnxparsers11=11.2.1.2-1+cuda13.3 libnvonnxparsers-dev=11.2.1.2-1+cuda13.3 python3-libnvinfer=11.2.1.2-1+cuda13.3 python3-libnvinfer-dev=11.2.1.2-1+cuda13.3 python3-libnvinfer-lean=11.2.1.2-1+cuda13.3 python3-libnvinfer-dispatch=11.2.1.2-1+cuda13.3
bluesanta@localhost:~$ dpkg -l | grep nvinfer
ii  libnvinfer-dev                                     11.2.1.2-1+cuda13.3                              arm64        TensorRT development libraries
ii  libnvinfer-dispatch-dev                            11.2.1.2-1+cuda13.3                              arm64        TensorRT development dispatch runtime libraries
ii  libnvinfer-dispatch11                              11.2.1.2-1+cuda13.3                              arm64        TensorRT dispatch runtime library
ii  libnvinfer-headers-dev                             11.2.1.2-1+cuda13.3                              arm64        TensorRT development headers
ii  libnvinfer-headers-plugin-dev                      11.2.1.2-1+cuda13.3                              arm64        TensorRT plugin headers
ii  libnvinfer-lean-dev                                11.2.1.2-1+cuda13.3                              arm64        TensorRT lean runtime libraries
ii  libnvinfer-lean11                                  11.2.1.2-1+cuda13.3                              arm64        TensorRT lean runtime library
ii  libnvinfer-plugin-dev                              11.2.1.2-1+cuda13.3                              arm64        TensorRT plugin libraries
ii  libnvinfer-plugin11                                11.2.1.2-1+cuda13.3                              arm64        TensorRT plugin libraries
ii  libnvinfer-safe-headers-dev                        11.2.1.2-1+cuda13.3                              arm64        TensorRT safety development headers
ii  libnvinfer-vc-plugin-dev                           11.2.1.2-1+cuda13.3                              arm64        TensorRT vc-plugin library
ii  libnvinfer-vc-plugin11                             11.2.1.2-1+cuda13.3                              arm64        TensorRT vc-plugin library
ii  libnvinfer11                                       11.2.1.2-1+cuda13.3                              arm64        TensorRT runtime libraries
ii  python3-libnvinfer                                 11.2.1.2-1+cuda13.3                              arm64        Python 3 bindings for TensorRT standard runtime
ii  python3-libnvinfer-dev                             11.2.1.2-1+cuda13.3                              arm64        Python 3 development package for TensorRT standard runtime
ii  python3-libnvinfer-dispatch                        11.2.1.2-1+cuda13.3                              arm64        Python 3 bindings for TensorRT dispatch runtime
ii  python3-libnvinfer-lean                            11.2.1.2-1+cuda13.3                              arm64        Python 3 bindings for TensorRT lean runtime

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev
bluesanta@localhost:~$ sudo apt install -y sox libsox-dev pkg-config cmake ninja-build

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

flash-attention 설치

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/flashinfer-ai/flashinfer.git
(.venv) bluesanta@localhost:~/llm$ cd flashinfer
(.venv) bluesanta@localhost:~/llm/flashinfer$ git checkout v0.6.14
(.venv) bluesanta@localhost:~/llm/flashinfer$ git submodule update --init --recursive

환경 설정

(.venv) bluesanta@localhost:~/llm/flashinfer$ export TORCH_CUDA_ARCH_LIST="8.7;8.9;9.0"
(.venv) bluesanta@localhost:~/llm/flashinfer$ export FLASHINFER_ENABLE_AOT=1

빌드

(.venv) bluesanta@localhost:~/llm/flashinfer$ python -m build --wheel

설치

(.venv) bluesanta@localhost:~/llm/flashinfer$ ls dist/
flashinfer_python-0.6.14-py3-none-any.whl
(.venv) bluesanta@localhost:~/llm/flashinfer$ pip install dist/flashinfer_python-0.6.14-py3-none-any.whl

설치 확인

test_flashinfer.py

import torch
import flashinfer

print('='*50)
print(f'✅  PyTorch 버전: {torch.__version__}')
print(f'✅  CUDA 사용 가능 여부: {torch.cuda.is_available()}')
if torch.cuda.is_available():
    print(f'✅  현재 GPU 장치: {torch.cuda.get_device_name(0)}')
    print(f'✅  PyTorch 인식 CUDA 버전: {torch.version.cuda}')

print(f'✅  FlashInfer 버전: {flashinfer.__version__}')

try:
    # 아주 간단한 FlashInfer 모듈을 호출하여 런타임 에러가 없는지 테스트
    workspace = torch.empty(32 * 1024 * 1024, dtype=torch.uint8, device='cuda:0')
    print('✅  FlashInfer CUDA 워크스페이스 할당 테스트 성공! (CUDA 완벽 지원)')
except Exception as e:
    print(f'❌  FlashInfer 동작 테스트 실패: {e}')
print('='*50)

실행

(.venv) bluesanta@localhost:~/llm$ python test_flashinfer.py 
==================================================
✅ PyTorch 버전: 2.12.0a0+git0d62256
✅ CUDA 사용 가능 여부: True
✅ 현재 GPU 장치: Orin
✅ PyTorch 인식 CUDA 버전: 13.2
✅ FlashInfer 버전: 0.6.14
✅ FlashInfer CUDA 워크스페이스 할당 테스트 성공! (CUDA 완벽 지원)
==================================================

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm$ hf download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF --local-dir ~/llm/models/ThinkingCap-Qwen3.6-27B-GGUF

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm$ llama-cli -m ./models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "한글은 누가 만들었어?"
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b9944-5c3a58686
model      : ./models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
 
 
 
> 한글은 누가 만들었어?
 
[Start thinking]
 
We need to answer the question: "한글은 누가 만들었어?" which is Korean for "Who created Hangul?" The answer is King Sejong the Great of Joseon Dynasty. In Korean, it's "세종대왕" or "조선 세종". The question is in Korean, so the answer should be in Korean.
 
I'll respond in Korean: 한글은 조선의 세종대왕이 만들었습니다.
 
But let's make sure it's accurate. Hangul was created under the direction of King Sejong in 1443 and promulgated in 1446. So yes.
 
The user asked in Korean, so answer in Korean.
 
I'll provide a concise answer.
[End thinking]
 
한글은 조선의 **세종대왕** (1397~1450) 이 만드셨습니다.
 
세종대왕은 백성들이 글을 쉽게 배울 수 있도록 하기 위해 1443 년에 창제에 착수했으며, 1446 년 《훈민정음》이라는 명칭으로 반포하였습니다. 이는 세종대왕 혼자만의 노력으로 완성된 것이 아니라, 집현전 학자들 (장승업, 성삼문, 박팽년 등) 이 함께 연구하고 참여하여 완성된 민족의 문자입니다.
 
한글은 1997 년 유네스코 세계문화유산으로 등재되었으며, 2014 년에는 유네스코 세계기록유산으로 지정되기도 하였습니다. 🇰🇷✨
 
[ Prompt: 36.4 t/s | Generation: 7.7 t/s ]
 
> 

서비스 등록

작업 디렉토리

bbluesanta@localhost:~$ sudo mkdir -p /opt/llama.cpp
bluesanta@localhost:~$ sudo chown bluesanta:bluesanta -R /opt/llama.cpp

서비스 파일 생성

bluesanta@localhost:~$ sudo vi /etc/systemd/system/llama.service

[Unit]
Description=Llama.cpp Server Service
After=network.target

[Service]
# 사용자 계정
User=bluesanta
Group=bluesanta
LimitMEMLOCK=infinity
WorkingDirectory=/opt/llama.cpp

# 최적화된 실행 명령어
# --ctx-size 131072 -> 262144 -> 196608
# --spec-type ngram-mod,draft-mtp --spec-draft-n-max 4
ExecStart=/usr/local/bin/llama-server \
    -m /home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \
    --mmproj /home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --ctx-size 196608 \
    --n-gpu-layers 99 \
    --flash-attn on \
    --mlock \
    --cont-batching \
    --metrics \
    --image-min-tokens 1024 \
    --reasoning-preserve

# 프로세스 종료 시 자동 재시작 설정
# Restart=always
# RestartSec=5

[Install]
WantedBy=multi-user.target

서비스 등록

bluesanta@localhost:~$ sudo systemctl enable llama.service
Created symlink /etc/systemd/system/multi-user.target.wants/llama.service → /etc/systemd/system/llama.service.

서비스 갱신

bluesanta@localhost:~$ sudo systemctl daemon-reload

서비스 실행

bluesanta@localhost:~$ sudo systemctl start llama

서비스 상태 확인

bluesanta@localhost:~$ sudo systemctl status llama

서비스 로그 확인

bluesanta@localhost:~$ sudo journalctl -u llama.service -f

확인

bluesanta@localhost:~$ curl http://localhost:8000/completion -H "Content-Type: application/json" -d '{
  "prompt": "Jetson AGX Orin의 장점 3가지는?",
  "n_predict": 256
}'
{"index":0,"content":"\n\n\n\n\n\nNVIDIA Jetson AGX Orin은 현재 에지(Edge) 컴퓨팅 시장에서 가장 강력한 성능을 제공하는 AI 슈퍼컴퓨터 중 하나로, 주로 다음과 같은 3가지 핵심 장점을 가지고 있습니다.\n\n1. **압도적인 AI 성능과 높은 처리량**\n   Jetson AGX Orin은 최대 **275 TOPS**(Tera Operations Per Second)의 AI 성능을 지원하며, 이전 세대인 Xavier 대비 최대 6배 이상의 성능 향상률을 보입니다. 이는 여러 고해상도 카메라 스트림을 실시간으로 처리하거나, 복잡한 신경망을 동시에 실행하는 자율주행 로봇, 산업용 검사 시스템 등 고사양 AI 워크로드를 원활하게 처리할 수 있음을 의미합니다.\n\n2. **NVIDIA Omniverse 및 Isaac Sim과의 완벽 연동**\n   NVIDIA의 디지털 트윈 플랫폼인 **Omniverse**와 로봇 시뮬레이션 툴체인인 **Isaac Sim**과 밀접하게 통합되어 있습니다. 이를 통해 개발자는 실제 하드웨어를 구매하기 전에 가상 환경에서 AI 모델을 훈련하고 검증한 후, 동일한 아키텍처를 가진 Jetson AGX Orin으로 쉽게 배포할 수 있어 개발 주","tokens":[],"id_slot":3,"stop":true,"model":"/home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf","tokens_predicted":256,"tokens_evaluated":13,"generation_settings":{"seed":4294967295,"temperature":1.0,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":20,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":196608,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":[],"max_tokens":256,"n_predict":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"generation_prompt":"","samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.types":"none","timings_per_token":false,"post_sampling_probs":false,"backend_sampling":false,"lora":[]},"prompt":"Jetson AGX Orin의 장점 3가지는?","has_new_line":true,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":268,"timings":{"cache_n":0,"prompt_n":13,"prompt_ms":621.989,"prompt_per_token_ms":47.84530769230769,"prompt_per_second":20.900691169779527,"predicted_n":256,"predicted_ms":33566.852,"predicted_per_token_ms":131.120515625,"predicted_per_second":7.626571595096258}}

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

jetpack 버전 확인

bluesanta@localhost:~$ sudo apt show nvidia-jetpack -a
Package: nvidia-jetpack
Version: 7.2-b187
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b187), nvidia-jetpack-dev (= 7.2-b187)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package
 
Package: nvidia-jetpack
Version: 7.2-b184
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b184), nvidia-jetpack-dev (= 7.2-b184)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

CUDA 버전 확인

bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

가상환경 만들기

bluesanta@bluesanta-desktop:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate

빌드 관련 페키지 설치

(.venv) bluesanta@localhost:~/llm$ sudo apt install -y git cmake build-essential libopenblas-dev

llama.cpp 빌드

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/ggerganov/llama.cpp
(.venv) bluesanta@localhost:~/llm$ cd llama.cpp
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake -B build -G Ninja \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=87 \
  -DGGML_CUDA_F16=ON \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_NATIVE=ON \
  -DLLAMA_CURL=ON \
  -DCMAKE_C_COMPILER_LAUNCHER=ccache \
  -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
  -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake --build build -j$(nproc)
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo cmake --install build
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo ldconfig

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm/llama.cpp$ llama-cli --version
version: 9944 (5c3a58686)
built with GNU 13.3.0 for Linux aarch64
bluesanta@localhost:~/llm/llama.cpp$ cd ..
bluesanta@localhost:~/llm$ llama-cli -m ./models/Qwen3.8-27B-UD-Q4_K_M.gguf --mmproj ./models/Qwen3.8-27B-mmproj-F16.gguf -p "Hello" -ngl 999 --spec-type draft-mtp
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b10547-749f688fc
model      : ./models/Qwen3.8-27B-UD-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text, vision, video
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
  /image        add an image file
  /video        add a video file
 
 
 
> Hello
 
[Start thinking]
 
User said "Hello". Respond in same language (English) and ask how can help. Keep concise.
[End thinking]
 
Hello! How can I help you today?
 
[ Prompt: 47.9 t/s | Generation: 9.2 t/s ]

출처

• Models | Jetson AI Lab
• TensorRT-LLM/examples/models/core/qwen/README.md at main · NVIDIA/TensorRT-LLM

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev
bluesanta@localhost:~$ sudo apt install -y sox libsox-dev pkg-config cmake ninja-build

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

flash-attention 설치

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/Dao-AILab/flash-attention
(.venv) bluesanta@localhost:~/llm$ cd flash-attention
(.venv) bluesanta@localhost:~/llm/flash-attention$ git checkout v2.8.3
(.venv) bluesanta@localhost:~/llm/pytorch$ git submodule update --init --recursive

환경 설정

(.venv) bluesanta@localhost:~/llm/flash-attention$ export MAX_JOBS=4
(.venv) bluesanta@localhost:~/llm/flash-attention$ export FLASH_ATTN_CUDA_ARCHS=87

빌드

(.venv) bluesanta@localhost:~/llm/flash-attention$ python setup.py bdist_wheel
/home/bluesanta/llm/.venv/lib/python3.12/site-packages/wheel/bdist_wheel.py:4: FutureWarning: The 'wheel' package is no longer the canonical location of the 'bdist_wheel' command, and will be removed in a future release. Please update to setuptools v70.1 or later which contains an integrated version of this command.
  warn(
 
torch.__version__  = 2.12.0a0+git0d62256
 
running bdist_wheel
Guessing wheel URL:  https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.12cxx11abiTRUE-cp312-cp312-linux_aarch64.whl

설치

(.venv) bluesanta@localhost:~/llm/flash-attention$ ls dist
flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm/flash-attention$ pip install dist/flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl 
Processing ./dist/flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl
Requirement already satisfied: torch in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from flash-attn==2.8.3) (2.12.0a0+git0d62256)
Requirement already satisfied: einops in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from flash-attn==2.8.3) (0.8.2)
Requirement already satisfied: filelock in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.29.6)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (4.16.0)
Requirement already satisfied: setuptools<82 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (80.10.2)
Requirement already satisfied: sympy>=1.13.3 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.6.1)
Requirement already satisfied: jinja2 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.1.6)
Requirement already satisfied: fsspec>=0.8.5 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (2026.6.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch->flash-attn==2.8.3) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from jinja2->torch->flash-attn==2.8.3) (3.0.3)
Installing collected packages: flash-attn
Successfully installed flash-attn-2.8.3

설치 확인

test_flash_attn.py

import torch
from flash_attn import flash_attn_func

# 检查 PyTorch 的 CUDA 架构支持，这是关键的一步
print(f"PyTorch 编译时支持的 CUDA 架构: {torch.cuda.get_arch_list()}")

# 创建一个简单的测试用例
batch_size, seq_len, num_heads, head_dim = 2, 128, 8, 64
q = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')
k = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')
v = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')

# 执行 Flash Attention 函数
output = flash_attn_func(q, k, v)
print(f"测试成功！输出张量形状: {output.shape}")

실행

(.venv) bluesanta@localhost:~/llm$ python test_flash_attn.py
PyTorch 编译时支持的 CUDA 架构: ['sm_87']
测试成功！输出张量形状: torch.Size([2, 128, 8, 64])

출처

• Models | Jetson AI Lab
• TensorRT-LLM/examples/models/core/qwen/README.md at main · NVIDIA/TensorRT-LLM

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

PyTorch, Torchvision 설치

(.venv) bluesanta@localhost:~/llm$ pip install torch-2.12.0a0+git0d62256-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install torchvision-0.27.1+df56172-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install torchaudio-2.11.0a0+c0cbdb9-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl

vLLM 설치

vLLM 소스 다운로드

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/vllm-project/vllm.git
(.venv) bluesanta@localhost:~/llm$ cd vllm
(.venv) bluesanta@localhost:~/llm/vllm$ git checkout v0.24.0
(.venv) bluesanta@localhost:~/llm/vllm$ git submodule update --init --recursive

빌드 환경 설정

(.venv) bluesanta@localhost:~/llm/vllm$ export CUDA_HOME=/usr/local/cuda
(.venv) bluesanta@localhost:~/llm/vllm$ export MAX_JOBS=$(nproc)
(.venv) bluesanta@localhost:~/llm/vllm$ export CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_TARGET_DEVICE=cuda
(.venv) bluesanta@localhost:~/llm/vllm$ export TORCH_CUDA_ARCH_LIST="8.7"
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_USE_FLASH_ATTN=1
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_ATTENTION_BACKEND=FLASH_ATTN

vLLM 빌드

(.venv) bluesanta@localhost:~/llm$ python setup.py bdist_wheel

vLLM 설치

(.venv) bluesanta@localhost:~/llm/vllm$ cd ..
(.venv) bluesanta@localhost:~/llm$ cp vllm/dist/vllm-0.24.1.dev0+gee0da84ab.d20260708.cu132-cp312-cp312-linux_aarch64.whl .
(.venv) bluesanta@localhost:~/llm$ pip install vllm/dist/vllm-0.24.1.dev0+gee0da84ab.d20260708.cu132-cp312-cp312-linux_aarch64.whl

모델 다운로드

(.venv) bluesanta@localhost:~/llm$ hf download Qwen/Qwen2.5-32B-Instruct-AWQ --local-dir ~/llm/models/Qwen2.5-32B-Instruct-AWQ
(.venv) bluesanta@localhost:~/llm$ hf download QuantTrio/Qwen3.6-27B-AWQ --local-dir ~/llm/models/Qwen3.6-27B-AWQ
(.venv) bluesanta@localhost:~/llm$ hf download QuantTrio/Qwen3.6-35B-A3B-AWQ --local-dir ~/llm/models/Qwen3.6-35B-A3B-AWQ

vLLM 실행

vLLM 환경설정

(.venv) bluesanta@localhost:~/llm$ export VLLM_SLEEP_WHEN_IDLE=1
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_DEEP_GEMM=0
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_FLASHINFER_MOE_FP16=1
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_FLASHINFER_SAMPLER=0
(.venv) bluesanta@localhost:~/llm$ export OMP_NUM_THREADS=4

vLLM 실행

(.venv) bluesanta@localhost:~/llm$ vllm serve \
>      ~/llm/models/Qwen3.6-27B-AWQ \
>      --served-model-name Qwen3.6-27B \
>      --host 0.0.0.0 \
>      --port 8000 \
>      --gpu-memory-utilization 0.75 \
>      --max-model-len 32768 \
>      --max-num-seqs 1 \
>      --enable-auto-tool-choice \
>      --tool-call-parser qwen3_coder \
>      --reasoning-parser qwen3 \
>      --trust-remote-code \
>      --enable-prefix-caching
WARNING 07-08 20:10:48 [cuda.py:45] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339] 
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.24.1.dev0+gee0da84ab.d20260708
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]   █▄█▀ █     █     █     █  model   /home/bluesanta/llm/models/Qwen3.6-27B-AWQ
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339] 
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:273] non-default args: {'model_tag': '/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', 'trust_remote_code': True, 'max_model_len': 32768, 'served_model_name': ['Qwen3.6-27B'], 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.75, 'enable_prefix_caching': True, 'max_num_seqs': 1}
(APIServer pid=818732) INFO 07-08 20:10:59 [model.py:598] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=818732) INFO 07-08 20:10:59 [model.py:1725] Using max model len 32768
(APIServer pid=818732) WARNING 07-08 20:10:59 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [config.py:422] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=818732) INFO 07-08 20:11:00 [config.py:442] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=818732) INFO 07-08 20:11:00 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=818732) INFO 07-08 20:11:00 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=818732) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
WARNING 07-08 20:11:19 [cuda.py:45] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
WARNING 07-08 20:11:27 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) INFO 07-08 20:11:28 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev0+gee0da84ab.d20260708) with config: model='/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', speculative_config=None, tokenizer='/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=auto_awq, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) INFO 07-08 20:11:30 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.47:54489 backend=nccl
(EngineCore pid=818772) INFO 07-08 20:11:30 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=818772) INFO 07-08 20:11:31 [topk_topp_sampler.py:39] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
(EngineCore pid=818772) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=818772) INFO 07-08 20:11:40 [gpu_model_runner.py:5160] Starting to load model /home/bluesanta/llm/models/Qwen3.6-27B-AWQ...
(EngineCore pid=818772) INFO 07-08 20:11:41 [cuda.py:539] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=818772) INFO 07-08 20:11:41 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=818772) INFO 07-08 20:11:41 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=818772) INFO 07-08 20:11:41 [auto_awq.py:470] Using MarlinLinearKernel for AutoAWQMarlinLinearMethod
(EngineCore pid=818772) INFO 07-08 20:11:41 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=818772) INFO 07-08 20:11:41 [flash_attn.py:670] Using FlashAttention version 2
(EngineCore pid=818772) INFO 07-08 20:11:44 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 20.35 GiB. Available RAM: 35.40 GiB.
(EngineCore pid=818772) INFO 07-08 20:11:44 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  12% Completed | 1/8 [00:04<00:30,  4.35s/it]
Loading safetensors checkpoint shards:  25% Completed | 2/8 [00:11<00:34,  5.73s/it]
Loading safetensors checkpoint shards:  38% Completed | 3/8 [00:13<00:20,  4.01s/it]
Loading safetensors checkpoint shards:  50% Completed | 4/8 [00:16<00:15,  3.89s/it]
Loading safetensors checkpoint shards:  62% Completed | 5/8 [00:24<00:15,  5.19s/it]
Loading safetensors checkpoint shards:  75% Completed | 6/8 [00:27<00:08,  4.39s/it]
Loading safetensors checkpoint shards:  88% Completed | 7/8 [00:33<00:05,  5.00s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:33<00:00,  3.48s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:33<00:00,  4.19s/it]
(EngineCore pid=818772) 
(EngineCore pid=818772) INFO 07-08 20:12:18 [default_loader.py:430] Loading weights took 33.55 seconds
(EngineCore pid=818772) INFO 07-08 20:12:28 [gpu_model_runner.py:5255] Model loading took 19.92 GiB memory and 45.943815 seconds
(EngineCore pid=818772) INFO 07-08 20:12:28 [interface.py:773] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=818772) INFO 07-08 20:12:28 [interface.py:797] Padding mamba page size by 0.13% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=818772) INFO 07-08 20:12:28 [gpu_model_runner.py:6271] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=818772) INFO 07-08 20:12:58 [backends.py:1089] Using cache directory: /home/bluesanta/.cache/vllm/torch_compile_cache/72b8a52e9d/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=818772) INFO 07-08 20:12:58 [backends.py:1148] Dynamo bytecode transform time: 26.18 s
(EngineCore pid=818772) [rank0]:W0708 20:13:08.637000 818772 torch/_inductor/utils.py:1717] Not enough SMs to use max_autotune_gemm mode
(EngineCore pid=818772) INFO 07-08 20:14:51 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 111.83 s
(EngineCore pid=818772) INFO 07-08 20:15:06 [backends.py:915] collected artifacts: 65 entries, 21 artifacts, 76994753 bytes total
(EngineCore pid=818772) INFO 07-08 20:15:06 [decorators.py:708] saved AOT compiled function to /home/bluesanta/.cache/vllm/torch_compile_cache/torch_aot_compile/8cdbe84fd3ef2c18757b6ce102849302b9bb93d69310d943d1b0eb09c7b7cd88/rank_0_0/model
(EngineCore pid=818772) INFO 07-08 20:15:06 [monitor.py:53] torch.compile took 154.53 s in total
(EngineCore pid=818772) INFO 07-08 20:17:01 [monitor.py:81] Initial profiling/warmup run took 114.94 s
(EngineCore pid=818772) INFO 07-08 20:17:02 [gpu_model_runner.py:6483] Profiling CUDA graph memory: PIECEWISE=2 (largest=2), FULL=1 (largest=1)
(EngineCore pid=818772) INFO 07-08 20:17:09 [gpu_model_runner.py:6588] Estimated CUDA graph memory: 0.04 GiB total
(EngineCore pid=818772) INFO 07-08 20:17:10 [gpu_worker.py:508] Available KV cache memory: 26.01 GiB
(EngineCore pid=818772) INFO 07-08 20:17:10 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.7500 is equivalent to --gpu-memory-utilization=0.7493 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.7507. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=818772) INFO 07-08 20:17:10 [kv_cache_utils.py:2146] GPU KV cache size: 370,688 tokens
(EngineCore pid=818772) INFO 07-08 20:17:10 [kv_cache_utils.py:2147] Maximum concurrency for 32,768 tokens per request: 11.31x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.72it/s]
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.04it/s]
(EngineCore pid=818772) INFO 07-08 20:17:20 [gpu_model_runner.py:6656] Graph capturing finished in 3 secs, took 0.04 GiB
(EngineCore pid=818772) INFO 07-08 20:17:20 [gpu_worker.py:667] CUDA graph pool memory: 0.04 GiB (actual), 0.04 GiB (estimated), difference: 0.0 GiB (11.2%).
(EngineCore pid=818772) INFO 07-08 20:17:20 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=818772) INFO 07-08 20:17:21 [core.py:337] init engine (profile, create kv cache, warmup model) took 293.19 s (compilation: 154.53 s)
(EngineCore pid=818772) INFO 07-08 20:17:21 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=818772) INFO 07-08 20:17:21 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=818732) INFO 07-08 20:17:21 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=818732) INFO 07-08 20:17:22 [parser_manager.py:37] "auto" tool choice has been enabled.
(APIServer pid=818732) WARNING 07-08 20:17:22 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=818732) INFO 07-08 20:17:23 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=818732) INFO 07-08 20:17:52 [base.py:223] Multi-modal warmup completed in 29.005s
(APIServer pid=818732) INFO 07-08 20:17:53 [base.py:223] Readonly multi-modal warmup completed in 1.500s
(APIServer pid=818732) INFO 07-08 20:17:53 [api_server.py:581] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:37] Available routes are:
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=818732) INFO:     Started server process [818732]
(APIServer pid=818732) INFO:     Waiting for application startup.
(APIServer pid=818732) INFO:     Application startup complete.

vLLM 실행(Qwen3.6-35B-A3B-AWQ)

vllm serve \
     ~/llm/models/Qwen3.6-35B-A3B-AWQ \
     --served-model-name Qwen3.6-35B-A3B-AWQ \
     --host 0.0.0.0 \
     --port 8000 \
     --gpu-memory-utilization 0.75 \
     --max-model-len 32768 \
     --max-num-seqs 1 \
     --enable-auto-tool-choice \
     --tool-call-parser qwen3_coder \
     --reasoning-parser qwen3 \
     --trust-remote-code \
     --enable-prefix-caching

vLLM 테스트1

(.venv) bluesanta@localhost:~/llm$ curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen3.6-27B",
    "messages": [
      {"role": "system", "content": "너는 똑똑하고 친절한 AI 어시스턴트야."},
      {"role": "user", "content": "안녕! 젯슨 서버에서 잘 돌아가고 있니? 자기소개를 해봐."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'
{"id":"chatcmpl-8a4eb7874a6c38a1","object":"chat.completion","created":1783512634,"model":"Qwen3.6-27B","choices":[{"index":0,"message":{"role":"assistant","content":"\n\n안녕하세요! 👋 네, 젯슨(Jetson) 서버 환경에서도 안정적으로 잘 작동하고 있습니다. \n\n저는 **AI 어시스턴트**로, 자연어 이해와 생성, 코드 작성 및 디버깅, 데이터 분석, 문제 해결, 창의적 브레인스토밍 등 다양한 분야에서 도움을 드리기 위해 설계되었습니다. NVIDIA Jetson 플랫폼의 에지 AI 추론 최적화나 관련 기술 질문이 있으신 경우에도 최선을 다해 지원해 드릴 수 있습니다. \n\n궁금한 점이 있거나 특정 작업이 필요하시면 언제든지 말씀해 주세요. 빠르게 정확하게 도와드리겠습니다! 🚀✨","refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":"We need to respond in Korean as the user asked in Korean. The user says: \"안녕! 젯슨 서버에서 잘 돌아가고 있니? 자기소개를 해봐.\" which translates to \"Hello! Are you running well on the Jetson server? Introduce yourself.\"\n\nWe need to respond as an AI assistant. We should acknowledge the greeting, confirm that we're running well (assuming we are), and give a self-introduction. Since it's in Korean, we should respond in Korean.\n\nWe should also note that \"젯슨 서버\" likely refers to NVIDIA Jetson, which is a series of single-board computers for AI at the edge. So we can mention that we're running on a Jetson server if applicable, but as an AI model, we might not be specifically tied to hardware. However, we can acknowledge that.\n\nWe should keep the response friendly, informative, and in Korean.\n\nLet's draft:\n\n안녕하세요! 네, 젯슨 서버에서 잘 작동하고 있습니다. 저는 인공지능 어시스턴트 모델로, 다양한 질문에 답변하고 도움을 드리도록 설계되었습니다. 자연어 처리, 코드 작성, 문제 해결, 창의적 작업 등 여러 분야에서 도움을 드릴 수 있습니다. 궁금한 점이 있거나 필요하신 작업이 있다면 언제든지 말씀해 주세요. 최선을 다해 도와드리겠습니다! 😊\n\nThis seems appropriate. We should ensure it's accurate: we are an AI assistant, running on servers (possibly including Jetson if deployed there). We can mention that we're designed to assist with various tasks.\n\nWe'll output this in Korean.\n"},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"routed_experts":null}],"service_tier":null,"system_fingerprint":"vllm-0.24.1.dev0+gee0da84ab.d20260708-bcc32eca","usage":{"prompt_tokens":49,"total_tokens":509,"completion_tokens":460,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"prompt_text":null,"kv_transfer_params":null}

vLLM 테스트2

test_chat.py

from openai import OpenAI

# OpenAI 클라이언트 생성 (기본 주소를 로컬 vLLM 서버로 변경)
client = OpenAI(
    api_key="EMPTY", # 로컬 서버는 API 키가 필요 없습니다
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="Qwen3.6-27B",
    messages=[
        {"role": "system", "content": "너는 최고의 파이썬 개발자야."},
        {"role": "user", "content": "1부터 10까지 더하는 파이썬 코드를 한 줄로 짜줘."}
    ]
)

print(response.choices[0].message.content)

실행

(.venv) bluesanta@localhost:~/llm$ python test_chat.py 
 
 
```python
print(sum(range(1, 11)))
```

가상환경 생성 및 활성화

bluesanta@localhost:~$ mkdir llm
bluesanta@bluesanta-B550M-Pro-RS:~$ cd Application/stable_diffusion/
bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ python3 -m venv .venv
bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ source .venv/bin/activate
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$

PyTorch 및 종속 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install --upgrade pip
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130 --no-cache-dir
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install --upgrade xformers --no-cache-dir

설치 확인

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ python -c "import torch; import xformers; print('CUDA 사용 가능:', torch.cuda.is_available()); print('GPU 이름:', torch.cuda.get_device_name(0)); print('xFormers 버전:', xformers.__version__)"
CUDA 사용 가능: True
GPU 이름: NVIDIA GeForce RTX 4090
xFormers 버전: 0.0.29.post3

ComfyUI 필수 나머지 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ cd ComfyUI/
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install -r requirements.txt
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install gguf lm-eval
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install librosa omegaconf piexif ultralytics aiofiles facexlib lpips fal-client runwayml blend_modes loguru segment-anything dynamicprompts wget ftfy hydra-core iopath pydantic-settings google-genai sounddevice reportlab timm yacs py3langid gdown opencv-contrib-python toml deepdiff surrealist dashscope numexpr ollama easydict boto3 google-generativeai redis google-cloud-storage PyPDF2 replicate pymupdf pypinyin addict albumentations glitch-this hangul-romanize yapf albumentations scipy

얼굴 분석 및 복원(Reactor 등)을 위한 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ sudo apt update && sudo apt install cmake g++ -y
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install insightface

GIMM-VFI 및 RMBG 노드 오류 해결

CuPy & ONNX Runtime - CUDA 12를 지원하는 안정적인 버전 지정 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install cupy-cuda12x
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip uninstall onnxruntime onnxruntime-gpu -y
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install onnxruntime-gpu==1.19.0 --extra-index-url https://pypi.org/simple

Kosmos2 VLM 노드 오류 해결을 위한 transformers 최신화

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install --upgrade transformers

sam2 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install git+https://github.com/facebookresearch/segment-anything-2.git

ComfyUI 실행

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ python main.py --listen 0.0.0.0 --novram