파란크리스마스

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev
bluesanta@localhost:~$ sudo apt install -y sox libsox-dev pkg-config cmake ninja-build

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

flash-attention 설치

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/flashinfer-ai/flashinfer.git
(.venv) bluesanta@localhost:~/llm$ cd flashinfer
(.venv) bluesanta@localhost:~/llm/flashinfer$ git checkout v0.6.14
(.venv) bluesanta@localhost:~/llm/flashinfer$ git submodule update --init --recursive

환경 설정

(.venv) bluesanta@localhost:~/llm/flashinfer$ export TORCH_CUDA_ARCH_LIST="8.7;8.9;9.0"
(.venv) bluesanta@localhost:~/llm/flashinfer$ export FLASHINFER_ENABLE_AOT=1

빌드

(.venv) bluesanta@localhost:~/llm/flashinfer$ python -m build --wheel

설치

(.venv) bluesanta@localhost:~/llm/flashinfer$ ls dist/
flashinfer_python-0.6.14-py3-none-any.whl
(.venv) bluesanta@localhost:~/llm/flashinfer$ pip install dist/flashinfer_python-0.6.14-py3-none-any.whl

설치 확인

test_flashinfer.py

import torch
import flashinfer

print('='*50)
print(f'✅  PyTorch 버전: {torch.__version__}')
print(f'✅  CUDA 사용 가능 여부: {torch.cuda.is_available()}')
if torch.cuda.is_available():
    print(f'✅  현재 GPU 장치: {torch.cuda.get_device_name(0)}')
    print(f'✅  PyTorch 인식 CUDA 버전: {torch.version.cuda}')

print(f'✅  FlashInfer 버전: {flashinfer.__version__}')

try:
    # 아주 간단한 FlashInfer 모듈을 호출하여 런타임 에러가 없는지 테스트
    workspace = torch.empty(32 * 1024 * 1024, dtype=torch.uint8, device='cuda:0')
    print('✅  FlashInfer CUDA 워크스페이스 할당 테스트 성공! (CUDA 완벽 지원)')
except Exception as e:
    print(f'❌  FlashInfer 동작 테스트 실패: {e}')
print('='*50)

실행

(.venv) bluesanta@localhost:~/llm$ python test_flashinfer.py 
==================================================
✅ PyTorch 버전: 2.12.0a0+git0d62256
✅ CUDA 사용 가능 여부: True
✅ 현재 GPU 장치: Orin
✅ PyTorch 인식 CUDA 버전: 13.2
✅ FlashInfer 버전: 0.6.14
✅ FlashInfer CUDA 워크스페이스 할당 테스트 성공! (CUDA 완벽 지원)
==================================================

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm$ hf download bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF --local-dir ~/llm/models/ThinkingCap-Qwen3.6-27B-GGUF

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm$ llama-cli -m ./models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf -p "한글은 누가 만들었어?"
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b9944-5c3a58686
model      : ./models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
 
 
 
> 한글은 누가 만들었어?
 
[Start thinking]
 
We need to answer the question: "한글은 누가 만들었어?" which is Korean for "Who created Hangul?" The answer is King Sejong the Great of Joseon Dynasty. In Korean, it's "세종대왕" or "조선 세종". The question is in Korean, so the answer should be in Korean.
 
I'll respond in Korean: 한글은 조선의 세종대왕이 만들었습니다.
 
But let's make sure it's accurate. Hangul was created under the direction of King Sejong in 1443 and promulgated in 1446. So yes.
 
The user asked in Korean, so answer in Korean.
 
I'll provide a concise answer.
[End thinking]
 
한글은 조선의 **세종대왕** (1397~1450) 이 만드셨습니다.
 
세종대왕은 백성들이 글을 쉽게 배울 수 있도록 하기 위해 1443 년에 창제에 착수했으며, 1446 년 《훈민정음》이라는 명칭으로 반포하였습니다. 이는 세종대왕 혼자만의 노력으로 완성된 것이 아니라, 집현전 학자들 (장승업, 성삼문, 박팽년 등) 이 함께 연구하고 참여하여 완성된 민족의 문자입니다.
 
한글은 1997 년 유네스코 세계문화유산으로 등재되었으며, 2014 년에는 유네스코 세계기록유산으로 지정되기도 하였습니다. 🇰🇷✨
 
[ Prompt: 36.4 t/s | Generation: 7.7 t/s ]
 
> 

서비스 등록

작업 디렉토리

bbluesanta@localhost:~$ sudo mkdir -p /opt/llama.cpp
bluesanta@localhost:~$ sudo chown bluesanta:bluesanta -R /opt/llama.cpp

서비스 파일 생성

bluesanta@localhost:~$ sudo vi /etc/systemd/system/llama.service

[Unit]
Description=Llama.cpp Server Service
After=network.target

[Service]
# 사용자 계정
User=bluesanta
Group=bluesanta
LimitMEMLOCK=infinity
WorkingDirectory=/opt/llama.cpp

# 최적화된 실행 명령어
# --ctx-size 131072 -> 262144 -> 196608
# --spec-type ngram-mod,draft-mtp --spec-draft-n-max 4
ExecStart=/usr/local/bin/llama-server \
    -m /home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf \
    --mmproj /home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/mmproj-ThinkingCap-Qwen3.6-27B-f16.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --ctx-size 196608 \
    --n-gpu-layers 99 \
    --flash-attn on \
    --mlock \
    --cont-batching \
    --metrics \
    --image-min-tokens 1024 \
    --reasoning-preserve

# 프로세스 종료 시 자동 재시작 설정
# Restart=always
# RestartSec=5

[Install]
WantedBy=multi-user.target

서비스 등록

bluesanta@localhost:~$ sudo systemctl enable llama.service
Created symlink /etc/systemd/system/multi-user.target.wants/llama.service → /etc/systemd/system/llama.service.

서비스 갱신

bluesanta@localhost:~$ sudo systemctl daemon-reload

서비스 실행

bluesanta@localhost:~$ sudo systemctl start llama

서비스 상태 확인

bluesanta@localhost:~$ sudo systemctl status llama

서비스 로그 확인

bluesanta@localhost:~$ sudo journalctl -u llama.service -f

확인

bluesanta@localhost:~$ curl http://localhost:8000/completion -H "Content-Type: application/json" -d '{
  "prompt": "Jetson AGX Orin의 장점 3가지는?",
  "n_predict": 256
}'
{"index":0,"content":"\n\n\n\n\n\nNVIDIA Jetson AGX Orin은 현재 에지(Edge) 컴퓨팅 시장에서 가장 강력한 성능을 제공하는 AI 슈퍼컴퓨터 중 하나로, 주로 다음과 같은 3가지 핵심 장점을 가지고 있습니다.\n\n1. **압도적인 AI 성능과 높은 처리량**\n   Jetson AGX Orin은 최대 **275 TOPS**(Tera Operations Per Second)의 AI 성능을 지원하며, 이전 세대인 Xavier 대비 최대 6배 이상의 성능 향상률을 보입니다. 이는 여러 고해상도 카메라 스트림을 실시간으로 처리하거나, 복잡한 신경망을 동시에 실행하는 자율주행 로봇, 산업용 검사 시스템 등 고사양 AI 워크로드를 원활하게 처리할 수 있음을 의미합니다.\n\n2. **NVIDIA Omniverse 및 Isaac Sim과의 완벽 연동**\n   NVIDIA의 디지털 트윈 플랫폼인 **Omniverse**와 로봇 시뮬레이션 툴체인인 **Isaac Sim**과 밀접하게 통합되어 있습니다. 이를 통해 개발자는 실제 하드웨어를 구매하기 전에 가상 환경에서 AI 모델을 훈련하고 검증한 후, 동일한 아키텍처를 가진 Jetson AGX Orin으로 쉽게 배포할 수 있어 개발 주","tokens":[],"id_slot":3,"stop":true,"model":"/home/bluesanta/llm/models/ThinkingCap-Qwen3.6-27B-GGUF/ThinkingCap-Qwen3.6-27B-Q4_K_M.gguf","tokens_predicted":256,"tokens_evaluated":13,"generation_settings":{"seed":4294967295,"temperature":1.0,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":20,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":196608,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":[],"max_tokens":256,"n_predict":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"generation_prompt":"","samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.types":"none","timings_per_token":false,"post_sampling_probs":false,"backend_sampling":false,"lora":[]},"prompt":"Jetson AGX Orin의 장점 3가지는?","has_new_line":true,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":268,"timings":{"cache_n":0,"prompt_n":13,"prompt_ms":621.989,"prompt_per_token_ms":47.84530769230769,"prompt_per_second":20.900691169779527,"predicted_n":256,"predicted_ms":33566.852,"predicted_per_token_ms":131.120515625,"predicted_per_second":7.626571595096258}}

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

jetpack 버전 확인

bluesanta@localhost:~$ sudo apt show nvidia-jetpack -a
Package: nvidia-jetpack
Version: 7.2-b187
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b187), nvidia-jetpack-dev (= 7.2-b187)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package
 
Package: nvidia-jetpack
Version: 7.2-b184
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b184), nvidia-jetpack-dev (= 7.2-b184)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

CUDA 버전 확인

bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

가상환경 만들기

bluesanta@bluesanta-desktop:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate

빌드 관련 페키지 설치

(.venv) bluesanta@localhost:~/llm$ sudo apt install -y git cmake build-essential libopenblas-dev

llama.cpp 빌드

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/ggerganov/llama.cpp
(.venv) bluesanta@localhost:~/llm$ cd llama.cpp
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake --build build -j$(nproc)
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo cmake --install build
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo ldconfig

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm/llama.cpp$ llama-cli --version
version: 9944 (5c3a58686)
built with GNU 13.3.0 for Linux aarch64
(.venv) bluesanta@localhost:~/llm/llama.cpp$ llama-cli -m ../models/Qwen3.6-35B-A3B-Q5_K_M.gguf --mmproj ../models/Qwen3.6-35B-A3B-mmproj-f16.gguf -p "Hello" -ngl 999
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b9944-5c3a58686
model      : ../models/Qwen3.6-35B-A3B-Q5_K_M.gguf
ftype      : Q5_K - Medium
modalities : text, vision, video
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
  /image        add an image file
  /video        add a video file
 
 
 
> Hello
 
[Start thinking]
 
Here's a thinking process:
 
1.  **Analyze User Input:**
   - User said: "Hello"
   - This is a simple greeting.
 
2.  **Identify Intent:**
   - The user is initiating a conversation.
   - No specific question or task is provided.
 
3.  **Determine Response Strategy:**
   - Acknowledge the greeting warmly.
   - Offer assistance.
   - Keep it concise and open-ended to encourage further interaction.
 
4.  **Draft Response (Mental):**
   - "Hello! How can I assist you today?"
   - Alternative: "Hi there! What can I do for you?"
   - Both are appropriate. I'll go with a friendly, professional tone.
 
5.  **Refine Response:**
   - "Hello! How can I help you today?" (Simple, clear, open-ended)
 
6.  **Final Output Generation:** (Matches the refined response)✅
[End thinking]
 
Hello! How can I help you today?
 
[ Prompt: 39.4 t/s | Generation: 31.6 t/s ]

출처

• Models | Jetson AI Lab
• TensorRT-LLM/examples/models/core/qwen/README.md at main · NVIDIA/TensorRT-LLM

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev
bluesanta@localhost:~$ sudo apt install -y sox libsox-dev pkg-config cmake ninja-build

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

flash-attention 설치

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/Dao-AILab/flash-attention
(.venv) bluesanta@localhost:~/llm$ cd flash-attention
(.venv) bluesanta@localhost:~/llm/flash-attention$ git checkout v2.8.3
(.venv) bluesanta@localhost:~/llm/pytorch$ git submodule update --init --recursive

환경 설정

(.venv) bluesanta@localhost:~/llm/flash-attention$ export MAX_JOBS=4
(.venv) bluesanta@localhost:~/llm/flash-attention$ export FLASH_ATTN_CUDA_ARCHS=87

빌드

(.venv) bluesanta@localhost:~/llm/flash-attention$ python setup.py bdist_wheel
/home/bluesanta/llm/.venv/lib/python3.12/site-packages/wheel/bdist_wheel.py:4: FutureWarning: The 'wheel' package is no longer the canonical location of the 'bdist_wheel' command, and will be removed in a future release. Please update to setuptools v70.1 or later which contains an integrated version of this command.
  warn(
 
torch.__version__  = 2.12.0a0+git0d62256
 
running bdist_wheel
Guessing wheel URL:  https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3/flash_attn-2.8.3+cu12torch2.12cxx11abiTRUE-cp312-cp312-linux_aarch64.whl

설치

(.venv) bluesanta@localhost:~/llm/flash-attention$ ls dist
flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm/flash-attention$ pip install dist/flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl 
Processing ./dist/flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl
Requirement already satisfied: torch in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from flash-attn==2.8.3) (2.12.0a0+git0d62256)
Requirement already satisfied: einops in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from flash-attn==2.8.3) (0.8.2)
Requirement already satisfied: filelock in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.29.6)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (4.16.0)
Requirement already satisfied: setuptools<82 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (80.10.2)
Requirement already satisfied: sympy>=1.13.3 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (1.14.0)
Requirement already satisfied: networkx>=2.5.1 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.6.1)
Requirement already satisfied: jinja2 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (3.1.6)
Requirement already satisfied: fsspec>=0.8.5 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from torch->flash-attn==2.8.3) (2026.6.0)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from sympy>=1.13.3->torch->flash-attn==2.8.3) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/bluesanta/llm/.venv/lib/python3.12/site-packages (from jinja2->torch->flash-attn==2.8.3) (3.0.3)
Installing collected packages: flash-attn
Successfully installed flash-attn-2.8.3

설치 확인

test_flash_attn.py

import torch
from flash_attn import flash_attn_func

# 检查 PyTorch 的 CUDA 架构支持，这是关键的一步
print(f"PyTorch 编译时支持的 CUDA 架构: {torch.cuda.get_arch_list()}")

# 创建一个简单的测试用例
batch_size, seq_len, num_heads, head_dim = 2, 128, 8, 64
q = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')
k = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')
v = torch.randn(batch_size, seq_len, num_heads, head_dim, dtype=torch.float16, device='cuda')

# 执行 Flash Attention 函数
output = flash_attn_func(q, k, v)
print(f"测试成功！输出张量形状: {output.shape}")

실행

(.venv) bluesanta@localhost:~/llm$ python test_flash_attn.py
PyTorch 编译时支持的 CUDA 架构: ['sm_87']
测试成功！输出张量形状: torch.Size([2, 128, 8, 64])

출처

• Models | Jetson AI Lab
• TensorRT-LLM/examples/models/core/qwen/README.md at main · NVIDIA/TensorRT-LLM

CUDA 버전 확인

bluesanta@localhost:~$ uname -a
Linux localhost.localdomain 6.8.12-1021-tegra #1 SMP PREEMPT Mon Jun  1 13:25:46 PDT 2026 aarch64 aarch64 aarch64 GNU/Linux
bluesanta@localhost:~$ cat /etc/nv_tegra_release
## R39 (release), REVISION: 2.0, GCID: 45755727, BOARD: generic, EABI: aarch64, DATE: Mon Jun  1 09:28:48 PM UTC 2026
## KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia
bluesanta@localhost:~$ nvidia-smi --query-gpu=name,compute_cap,driver_version --format=csv
name, compute_cap, driver_version
Orin (nvgpu), 8.7, 595.78
bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

빌드 도구 설치

bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install -y cmake ninja-build gcc g++ git build-essential git cmake ninja-build libopenblas-dev libopenmpi-dev openmpi-bin libatlas-base-dev libprotobuf-dev protobuf-compiler libssl-dev zlib1g-dev libffi-dev python3-pip libopenblas-dev ccache git-lfs libjpeg-dev libpng-dev libtiff-dev

NCCL 설치

bluesanta@localhost:~$ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo dpkg -i cuda-keyring_1.1-1_all.deb
bluesanta@localhost:~$ sudo apt update
bluesanta@localhost:~$ sudo apt install libnccl2 libnccl-dev

Python 가상 환경 생성 및 활성화

bluesanta@localhost:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate
(.venv) bluesanta@localhost:~/llm$ python --version
Python 3.12.3
(.venv) bluesanta@localhost:~/llm$ pip install --upgrade pip

PyTorch, Torchvision 설치

(.venv) bluesanta@localhost:~/llm$ pip install torch-2.12.0a0+git0d62256-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install torchvision-0.27.1+df56172-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install torchaudio-2.11.0a0+c0cbdb9-cp312-cp312-linux_aarch64.whl
(.venv) bluesanta@localhost:~/llm$ pip install flash_attn-2.8.3-cp312-cp312-linux_aarch64.whl

vLLM 설치

vLLM 소스 다운로드

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/vllm-project/vllm.git
(.venv) bluesanta@localhost:~/llm$ cd vllm
(.venv) bluesanta@localhost:~/llm/vllm$ git checkout v0.24.0
(.venv) bluesanta@localhost:~/llm/vllm$ git submodule update --init --recursive

빌드 환경 설정

(.venv) bluesanta@localhost:~/llm/vllm$ export CUDA_HOME=/usr/local/cuda
(.venv) bluesanta@localhost:~/llm/vllm$ export MAX_JOBS=$(nproc)
(.venv) bluesanta@localhost:~/llm/vllm$ export CMAKE_BUILD_PARALLEL_LEVEL=$(nproc)
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_TARGET_DEVICE=cuda
(.venv) bluesanta@localhost:~/llm/vllm$ export TORCH_CUDA_ARCH_LIST="8.7"
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_USE_FLASH_ATTN=1
(.venv) bluesanta@localhost:~/llm/vllm$ export VLLM_ATTENTION_BACKEND=FLASH_ATTN

vLLM 빌드

(.venv) bluesanta@localhost:~/llm$ python setup.py bdist_wheel

vLLM 설치

(.venv) bluesanta@localhost:~/llm/vllm$ cd ..
(.venv) bluesanta@localhost:~/llm$ cp vllm/dist/vllm-0.24.1.dev0+gee0da84ab.d20260708.cu132-cp312-cp312-linux_aarch64.whl .
(.venv) bluesanta@localhost:~/llm$ pip install vllm/dist/vllm-0.24.1.dev0+gee0da84ab.d20260708.cu132-cp312-cp312-linux_aarch64.whl

모델 다운로드

(.venv) bluesanta@localhost:~/llm$ hf download Qwen/Qwen2.5-32B-Instruct-AWQ --local-dir ~/llm/models/Qwen2.5-32B-Instruct-AWQ
(.venv) bluesanta@localhost:~/llm$ hf download QuantTrio/Qwen3.6-27B-AWQ --local-dir ~/llm/models/Qwen3.6-27B-AWQ
(.venv) bluesanta@localhost:~/llm$ hf download QuantTrio/Qwen3.6-35B-A3B-AWQ --local-dir ~/llm/models/Qwen3.6-35B-A3B-AWQ

vLLM 실행

vLLM 환경설정

(.venv) bluesanta@localhost:~/llm$ export VLLM_SLEEP_WHEN_IDLE=1
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_DEEP_GEMM=0
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_FLASHINFER_MOE_FP16=1
(.venv) bluesanta@localhost:~/llm$ export VLLM_USE_FLASHINFER_SAMPLER=0
(.venv) bluesanta@localhost:~/llm$ export OMP_NUM_THREADS=4

vLLM 실행

(.venv) bluesanta@localhost:~/llm$ vllm serve \
>      ~/llm/models/Qwen3.6-27B-AWQ \
>      --served-model-name Qwen3.6-27B \
>      --host 0.0.0.0 \
>      --port 8000 \
>      --gpu-memory-utilization 0.75 \
>      --max-model-len 32768 \
>      --max-num-seqs 1 \
>      --enable-auto-tool-choice \
>      --tool-call-parser qwen3_coder \
>      --reasoning-parser qwen3 \
>      --trust-remote-code \
>      --enable-prefix-caching
WARNING 07-08 20:10:48 [cuda.py:45] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339] 
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.24.1.dev0+gee0da84ab.d20260708
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]   █▄█▀ █     █     █     █  model   /home/bluesanta/llm/models/Qwen3.6-27B-AWQ
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:339] 
(APIServer pid=818732) INFO 07-08 20:10:59 [api_utils.py:273] non-default args: {'model_tag': '/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'model': '/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', 'trust_remote_code': True, 'max_model_len': 32768, 'served_model_name': ['Qwen3.6-27B'], 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.75, 'enable_prefix_caching': True, 'max_num_seqs': 1}
(APIServer pid=818732) INFO 07-08 20:10:59 [model.py:598] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=818732) INFO 07-08 20:10:59 [model.py:1725] Using max model len 32768
(APIServer pid=818732) WARNING 07-08 20:10:59 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(APIServer pid=818732) WARNING 07-08 20:11:00 [config.py:422] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=818732) INFO 07-08 20:11:00 [config.py:442] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=818732) INFO 07-08 20:11:00 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=818732) INFO 07-08 20:11:00 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=818732) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
WARNING 07-08 20:11:19 [cuda.py:45] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
WARNING 07-08 20:11:27 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) INFO 07-08 20:11:28 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev0+gee0da84ab.d20260708) with config: model='/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', speculative_config=None, tokenizer='/home/bluesanta/llm/models/Qwen3.6-27B-AWQ', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=auto_awq, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=Qwen3.6-27B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 2, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) WARNING 07-08 20:11:28 [cuda.py:230] Failed to import from vllm._qutlass_C: ModuleNotFoundError("No module named 'vllm._qutlass_C'")
(EngineCore pid=818772) INFO 07-08 20:11:30 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.47:54489 backend=nccl
(EngineCore pid=818772) INFO 07-08 20:11:30 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=818772) INFO 07-08 20:11:31 [topk_topp_sampler.py:39] FlashInfer top-p/top-k sampling disabled via VLLM_USE_FLASHINFER_SAMPLER=0.
(EngineCore pid=818772) [transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(EngineCore pid=818772) INFO 07-08 20:11:40 [gpu_model_runner.py:5160] Starting to load model /home/bluesanta/llm/models/Qwen3.6-27B-AWQ...
(EngineCore pid=818772) INFO 07-08 20:11:41 [cuda.py:539] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=818772) INFO 07-08 20:11:41 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=818772) INFO 07-08 20:11:41 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=818772) INFO 07-08 20:11:41 [auto_awq.py:470] Using MarlinLinearKernel for AutoAWQMarlinLinearMethod
(EngineCore pid=818772) INFO 07-08 20:11:41 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=818772) INFO 07-08 20:11:41 [flash_attn.py:670] Using FlashAttention version 2
(EngineCore pid=818772) INFO 07-08 20:11:44 [weight_utils.py:849] Filesystem type for checkpoints: EXT4. Checkpoint size: 20.35 GiB. Available RAM: 35.40 GiB.
(EngineCore pid=818772) INFO 07-08 20:11:44 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/8 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  12% Completed | 1/8 [00:04<00:30,  4.35s/it]
Loading safetensors checkpoint shards:  25% Completed | 2/8 [00:11<00:34,  5.73s/it]
Loading safetensors checkpoint shards:  38% Completed | 3/8 [00:13<00:20,  4.01s/it]
Loading safetensors checkpoint shards:  50% Completed | 4/8 [00:16<00:15,  3.89s/it]
Loading safetensors checkpoint shards:  62% Completed | 5/8 [00:24<00:15,  5.19s/it]
Loading safetensors checkpoint shards:  75% Completed | 6/8 [00:27<00:08,  4.39s/it]
Loading safetensors checkpoint shards:  88% Completed | 7/8 [00:33<00:05,  5.00s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:33<00:00,  3.48s/it]
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:33<00:00,  4.19s/it]
(EngineCore pid=818772) 
(EngineCore pid=818772) INFO 07-08 20:12:18 [default_loader.py:430] Loading weights took 33.55 seconds
(EngineCore pid=818772) INFO 07-08 20:12:28 [gpu_model_runner.py:5255] Model loading took 19.92 GiB memory and 45.943815 seconds
(EngineCore pid=818772) INFO 07-08 20:12:28 [interface.py:773] Setting attention block size to 784 tokens to ensure that attention page size is >= mamba page size.
(EngineCore pid=818772) INFO 07-08 20:12:28 [interface.py:797] Padding mamba page size by 0.13% to ensure that mamba page size and attention page size are exactly equal.
(EngineCore pid=818772) INFO 07-08 20:12:28 [gpu_model_runner.py:6271] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
(EngineCore pid=818772) INFO 07-08 20:12:58 [backends.py:1089] Using cache directory: /home/bluesanta/.cache/vllm/torch_compile_cache/72b8a52e9d/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=818772) INFO 07-08 20:12:58 [backends.py:1148] Dynamo bytecode transform time: 26.18 s
(EngineCore pid=818772) [rank0]:W0708 20:13:08.637000 818772 torch/_inductor/utils.py:1717] Not enough SMs to use max_autotune_gemm mode
(EngineCore pid=818772) INFO 07-08 20:14:51 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 111.83 s
(EngineCore pid=818772) INFO 07-08 20:15:06 [backends.py:915] collected artifacts: 65 entries, 21 artifacts, 76994753 bytes total
(EngineCore pid=818772) INFO 07-08 20:15:06 [decorators.py:708] saved AOT compiled function to /home/bluesanta/.cache/vllm/torch_compile_cache/torch_aot_compile/8cdbe84fd3ef2c18757b6ce102849302b9bb93d69310d943d1b0eb09c7b7cd88/rank_0_0/model
(EngineCore pid=818772) INFO 07-08 20:15:06 [monitor.py:53] torch.compile took 154.53 s in total
(EngineCore pid=818772) INFO 07-08 20:17:01 [monitor.py:81] Initial profiling/warmup run took 114.94 s
(EngineCore pid=818772) INFO 07-08 20:17:02 [gpu_model_runner.py:6483] Profiling CUDA graph memory: PIECEWISE=2 (largest=2), FULL=1 (largest=1)
(EngineCore pid=818772) INFO 07-08 20:17:09 [gpu_model_runner.py:6588] Estimated CUDA graph memory: 0.04 GiB total
(EngineCore pid=818772) INFO 07-08 20:17:10 [gpu_worker.py:508] Available KV cache memory: 26.01 GiB
(EngineCore pid=818772) INFO 07-08 20:17:10 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.7500 is equivalent to --gpu-memory-utilization=0.7493 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.7507. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=818772) INFO 07-08 20:17:10 [kv_cache_utils.py:2146] GPU KV cache size: 370,688 tokens
(EngineCore pid=818772) INFO 07-08 20:17:10 [kv_cache_utils.py:2147] Maximum concurrency for 32,768 tokens per request: 11.31x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████| 2/2 [00:00<00:00,  4.72it/s]
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.04it/s]
(EngineCore pid=818772) INFO 07-08 20:17:20 [gpu_model_runner.py:6656] Graph capturing finished in 3 secs, took 0.04 GiB
(EngineCore pid=818772) INFO 07-08 20:17:20 [gpu_worker.py:667] CUDA graph pool memory: 0.04 GiB (actual), 0.04 GiB (estimated), difference: 0.0 GiB (11.2%).
(EngineCore pid=818772) INFO 07-08 20:17:20 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=818772) INFO 07-08 20:17:21 [core.py:337] init engine (profile, create kv cache, warmup model) took 293.19 s (compilation: 154.53 s)
(EngineCore pid=818772) INFO 07-08 20:17:21 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=818772) INFO 07-08 20:17:21 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=818732) INFO 07-08 20:17:21 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=818732) INFO 07-08 20:17:22 [parser_manager.py:37] "auto" tool choice has been enabled.
(APIServer pid=818732) WARNING 07-08 20:17:22 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 1.0, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=818732) INFO 07-08 20:17:23 [hf.py:548] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=818732) INFO 07-08 20:17:52 [base.py:223] Multi-modal warmup completed in 29.005s
(APIServer pid=818732) INFO 07-08 20:17:53 [base.py:223] Readonly multi-modal warmup completed in 1.500s
(APIServer pid=818732) INFO 07-08 20:17:53 [api_server.py:581] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:37] Available routes are:
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=818732) INFO 07-08 20:17:53 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=818732) INFO:     Started server process [818732]
(APIServer pid=818732) INFO:     Waiting for application startup.
(APIServer pid=818732) INFO:     Application startup complete.

vLLM 실행(Qwen3.6-35B-A3B-AWQ)

vllm serve \
     ~/llm/models/Qwen3.6-35B-A3B-AWQ \
     --served-model-name Qwen3.6-35B-A3B-AWQ \
     --host 0.0.0.0 \
     --port 8000 \
     --gpu-memory-utilization 0.75 \
     --max-model-len 32768 \
     --max-num-seqs 1 \
     --enable-auto-tool-choice \
     --tool-call-parser qwen3_coder \
     --reasoning-parser qwen3 \
     --trust-remote-code \
     --enable-prefix-caching

vLLM 테스트1

(.venv) bluesanta@localhost:~/llm$ curl http://localhost:8000/v1/chat/completions   -H "Content-Type: application/json"   -d '{
    "model": "Qwen3.6-27B",
    "messages": [
      {"role": "system", "content": "너는 똑똑하고 친절한 AI 어시스턴트야."},
      {"role": "user", "content": "안녕! 젯슨 서버에서 잘 돌아가고 있니? 자기소개를 해봐."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'
{"id":"chatcmpl-8a4eb7874a6c38a1","object":"chat.completion","created":1783512634,"model":"Qwen3.6-27B","choices":[{"index":0,"message":{"role":"assistant","content":"\n\n안녕하세요! 👋 네, 젯슨(Jetson) 서버 환경에서도 안정적으로 잘 작동하고 있습니다. \n\n저는 **AI 어시스턴트**로, 자연어 이해와 생성, 코드 작성 및 디버깅, 데이터 분석, 문제 해결, 창의적 브레인스토밍 등 다양한 분야에서 도움을 드리기 위해 설계되었습니다. NVIDIA Jetson 플랫폼의 에지 AI 추론 최적화나 관련 기술 질문이 있으신 경우에도 최선을 다해 지원해 드릴 수 있습니다. \n\n궁금한 점이 있거나 특정 작업이 필요하시면 언제든지 말씀해 주세요. 빠르게 정확하게 도와드리겠습니다! 🚀✨","refusal":null,"annotations":null,"audio":null,"function_call":null,"reasoning":"We need to respond in Korean as the user asked in Korean. The user says: \"안녕! 젯슨 서버에서 잘 돌아가고 있니? 자기소개를 해봐.\" which translates to \"Hello! Are you running well on the Jetson server? Introduce yourself.\"\n\nWe need to respond as an AI assistant. We should acknowledge the greeting, confirm that we're running well (assuming we are), and give a self-introduction. Since it's in Korean, we should respond in Korean.\n\nWe should also note that \"젯슨 서버\" likely refers to NVIDIA Jetson, which is a series of single-board computers for AI at the edge. So we can mention that we're running on a Jetson server if applicable, but as an AI model, we might not be specifically tied to hardware. However, we can acknowledge that.\n\nWe should keep the response friendly, informative, and in Korean.\n\nLet's draft:\n\n안녕하세요! 네, 젯슨 서버에서 잘 작동하고 있습니다. 저는 인공지능 어시스턴트 모델로, 다양한 질문에 답변하고 도움을 드리도록 설계되었습니다. 자연어 처리, 코드 작성, 문제 해결, 창의적 작업 등 여러 분야에서 도움을 드릴 수 있습니다. 궁금한 점이 있거나 필요하신 작업이 있다면 언제든지 말씀해 주세요. 최선을 다해 도와드리겠습니다! 😊\n\nThis seems appropriate. We should ensure it's accurate: we are an AI assistant, running on servers (possibly including Jetson if deployed there). We can mention that we're designed to assist with various tasks.\n\nWe'll output this in Korean.\n"},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null,"routed_experts":null}],"service_tier":null,"system_fingerprint":"vllm-0.24.1.dev0+gee0da84ab.d20260708-bcc32eca","usage":{"prompt_tokens":49,"total_tokens":509,"completion_tokens":460,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"prompt_text":null,"kv_transfer_params":null}

vLLM 테스트2

test_chat.py

from openai import OpenAI

# OpenAI 클라이언트 생성 (기본 주소를 로컬 vLLM 서버로 변경)
client = OpenAI(
    api_key="EMPTY", # 로컬 서버는 API 키가 필요 없습니다
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="Qwen3.6-27B",
    messages=[
        {"role": "system", "content": "너는 최고의 파이썬 개발자야."},
        {"role": "user", "content": "1부터 10까지 더하는 파이썬 코드를 한 줄로 짜줘."}
    ]
)

print(response.choices[0].message.content)

실행

(.venv) bluesanta@localhost:~/llm$ python test_chat.py 
 
 
```python
print(sum(range(1, 11)))
```

가상환경 생성 및 활성화

bluesanta@localhost:~$ mkdir llm
bluesanta@bluesanta-B550M-Pro-RS:~$ cd Application/stable_diffusion/
bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ python3 -m venv .venv
bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ source .venv/bin/activate
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$

PyTorch 및 종속 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install --upgrade pip
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130 --no-cache-dir
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ pip install --upgrade xformers --no-cache-dir

설치 확인

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ python -c "import torch; import xformers; print('CUDA 사용 가능:', torch.cuda.is_available()); print('GPU 이름:', torch.cuda.get_device_name(0)); print('xFormers 버전:', xformers.__version__)"
CUDA 사용 가능: True
GPU 이름: NVIDIA GeForce RTX 4090
xFormers 버전: 0.0.29.post3

ComfyUI 필수 나머지 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion$ cd ComfyUI/
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install -r requirements.txt
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install gguf lm-eval
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install librosa omegaconf piexif ultralytics aiofiles facexlib lpips fal-client runwayml blend_modes loguru segment-anything dynamicprompts wget ftfy hydra-core iopath pydantic-settings google-genai sounddevice reportlab timm yacs py3langid gdown opencv-contrib-python toml deepdiff surrealist dashscope numexpr ollama easydict boto3 google-generativeai redis google-cloud-storage PyPDF2 replicate pymupdf pypinyin addict albumentations glitch-this hangul-romanize yapf albumentations scipy

얼굴 분석 및 복원(Reactor 등)을 위한 패키지 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ sudo apt update && sudo apt install cmake g++ -y
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install insightface

GIMM-VFI 및 RMBG 노드 오류 해결

CuPy & ONNX Runtime - CUDA 12를 지원하는 안정적인 버전 지정 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install cupy-cuda12x
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip uninstall onnxruntime onnxruntime-gpu -y
(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install onnxruntime-gpu==1.19.0 --extra-index-url https://pypi.org/simple

Kosmos2 VLM 노드 오류 해결을 위한 transformers 최신화

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install --upgrade transformers

sam2 설치

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ pip install git+https://github.com/facebookresearch/segment-anything-2.git

ComfyUI 실행

(.venv) bluesanta@bluesanta-B550M-Pro-RS:~/Application/stable_diffusion/ComfyUI$ python main.py --listen 0.0.0.0 --novram

출처

• How to Install OpenClaude in 10 Minutes (Open Source Claude Code with ANY AI Model) - YouTube
• GitHub - Gitlawb/openclaude: runs anywhere. uses anything · GitHub

openclaude 설치

C:\Users\bluesanta>npm install -g @gitlawb/openclaude

openclaude 최신버전으로 설치

C:\Users\bluesanta>npm install -g @gitlawb/openclaude@latest

설치된 버전 확인

C:\Users\bluesanta>openclaude --version
0.13.0 (OpenClaude)

환경설정

C:\test>set CLAUDE_CODE_USE_OPENAI=1
C:\test>set OPENAI_API_KEY=sk-jetson-qwen36
C:\test>set OPENAI_BASE_URL=http://192.168.0.235:8000/v1
C:\test>set OPENAI_MODEL=Qwen3.6-35B-A3B-Q5_K_M.gguf

실행

C:\test>openclaude
Warning: ignoring saved provider profile. Codex auth is required for codexplan. Set CODEX_API_KEY, choose Codex OAuth in /provider or put auth.json at C:\Users\bluesanta\.codex\auth.json.
 
  ████████╗ ████████╗ ████████╗ ██╗  ██╗
  ██╔═══██║ ██╔═══██║ ██╔═════╝ ███╗ ██║
  ██║   ██║ ████████║ ██████╗   ████╗██║
  ██║   ██║ ██╔═════╝ ██╔═══╝   ██╔████║
  ████████║ ██║       ████████╗ ██║ ╚███║
  ╚═══════╝ ╚═╝       ╚═══════╝ ╚═╝  ╚══╝
 
  ████████╗ ██╗      ████████╗ ██╗   ██╗ ███████╗  ████████╗
  ██╔═════╝ ██║      ██╔═══██║ ██║   ██║ ██╔═══██╗ ██╔═════╝
  ██║       ██║      ████████║ ██║   ██║ ██║   ██║ ██████╗
  ██║       ██║      ██╔═══██║ ██║   ██║ ██║   ██║ ██╔═══╝
  ████████╗ ████████╗██║   ██║ ╚██████╔╝ ███████╔╝ ████████╗
  ╚═══════╝ ╚═══════╝╚═╝   ╚═╝  ╚═════╝  ╚══════╝  ╚═══════╝
 
  ✦ Any model. Every tool. Zero limits. ✦
 
╔════════════════════════════════════════════════════════════╗
│ Provider  OpenAI                                           │
│ Model     Qwen3.6-35B-A3B-Q5_K_M.gguf                      │
│ Endpoint  http://192.168.0.235:8000/v1                     │
╠════════════════════════════════════════════════════════════╣
│ ● cloud    Ready — type /help to begin                     │
╚════════════════════════════════════════════════════════════╝
  openclaude v0.13.0
 
 
─────────────────────────────────────────────────────────────────────────────────────────────
> 
─────────────────────────────────────────────────────────────────────────────────────────────
  ? for shortcuts

bun 설치

PS C:\WINDOWS\System32> powershell -c "irm bun.sh/install.ps1 | iex"

openclaude 소스 다운로드

C:\test>git clone https://github.com/Gitlawb/openclaude.git

openclaude-vscode 디렉토리로 이동

C:\test>cd openclaude
C:\test\openclaude>cd vscode-extension
C:\test\openclaude\vscode-extension>cd openclaude-vscode

빌드

C:\test\openclaude\vscode-extension\openclaude-vscode>bun add -d @vscode/vsce
bun add v1.3.14 (0d9b296a)
 
installed @vscode/vsce@3.9.1 with binaries:
 - vsce
 
293 packages installed [50.82s]
 
Blocked 1 postinstall. Run `bun pm untrusted` for details.
 
C:\test\openclaude\vscode-extension\openclaude-vscode>bunx vsce package
 WARNING  LICENSE, LICENSE.md, or LICENSE.txt not found
Do you want to continue? [y/N] y
 INFO  Files included in the VSIX:
openclaude-vscode-0.2.0.vsix
├─ [Content_Types].xml
├─ extension.vsixmanifest
└─ extension/
   ├─ package.json [4.99 KB]
   ├─ readme.md [2.34 KB]
   ├─ media/
   │  └─ openclaude.svg [0.43 KB]
   ├─ src/
   │  ├─ extension.js [38.7 KB]
   │  ├─ presentation.js [5.92 KB]
   │  ├─ state.js [10.93 KB]
   │  └─ chat/
   │     ├─ chatProvider.js [21.92 KB]
   │     ├─ chatRenderer.js [47.16 KB]
   │     ├─ diffController.js [2.66 KB]
   │     ├─ messageParser.js [4.5 KB]
   │     ├─ processManager.js [5.7 KB]
   │     ├─ protocol.js [4.64 KB]
   │     └─ sessionManager.js [8.61 KB]
   └─ themes/
      └─ OpenClaude-Terminal-Black.json [2.76 KB]
 
 DONE  Packaged: C:\test\openclaude\vscode-extension\openclaude-vscode\openclaude-vscode-0.2.0.vsix (16 files, 42.77 KB)

-

-

출처

• havenoammo/Qwen3.6-35B-A3B-MTP-GGUF · Hugging Face
• unsloth/Qwen3.6-35B-A3B-MTP-GGUF · Hugging Face
• Introducing Unsloth Studio | Unsloth Documentation

havenoammo/Qwen3.6-35B-A3B-MTP-GGUF

llama.cpp 소스 다운로드

(.venv) bluesanta@ubuntu:~/llm$ git clone https://github.com/ggml-org/llama.cpp.git
(.venv) bluesanta@ubuntu:~/llm$ cd llama.cpp

최신 원격 변경 사항을 가져오기

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ git fetch origin
From https://github.com/ggml-org/llama.cpp
 * [new tag]             b9151      -> b9151

PR #22673을 로컬 브랜치로 가져오기

PR #22673("llama + spec: MTP 지원")은 speculative decoding 기능을 추가

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ git fetch origin pull/22673/head:pr-22673
From https://github.com/ggml-org/llama.cpp
 * [new tag]             b9156      -> b9156

Checkout master and reset to latest remote

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ git checkout master
Already on 'master'
Your branch is up to date with 'origin/master'.
(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ git reset --hard 856c3adac
HEAD is now at 856c3adac hexagon: eliminate scalar VTCM loads via HVX splat helpers (#22993)

Merge the PR on top (non-fast-forward)

(.venv) bluesanta@ubuntu:~/llm/llama.cpp# git merge --no-ff pr-22673 -m "Merge [PR #22673](https://github.com/ggml-org/llama.cpp/pull/22673): llama + spec: MTP Support"
Merge made by the 'ort' strategy.
 .devops/intel.Dockerfile                                                     |    20 +-
 .editorconfig                                                                |     8 -
 .gitattributes                                                               |     4 -
 
 delete mode 100644 tools/server/public/index.html
 delete mode 100644 tools/server/public/loading.html

빌드 llama-server

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ cmake -B build -DGGML_CUDA=ON
bluesanta@ubuntu:~/llm/llama.cpp$ cmake --build build --config Release -j$(nproc)

llama-server 설치

bluesanta@ubuntu:~/llm/llama.cpp$ sudo cmake --install build

llama-server 버전 확인

bluesanta@ubuntu:~/llm/llama.cpp$ ./build/bin/llama-server --version
version: 9173 (0672285b2)
built with GNU 11.4.0 for Linux aarch64

unsloth/Qwen3.6-35B-A3B-MTP-GGUF

소스 다운로드

bluesanta@ubuntu:~/llm$ git clone https://github.com/ggml-org/llama.cpp.git llama.cpp-22673-mtp
bluesanta@ubuntu:~/llm$ cd llama.cpp-22673-mtp
bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ git fetch origin pull/22673/head:pr-22673-mtp
remote: Enumerating objects: 158, done.
remote: Counting objects: 100% (128/128), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 158 (delta 119), reused 119 (delta 119), pack-reused 30 (from 2)
Receiving objects: 100% (158/158), 158.55 KiB | 12.20 MiB/s, done.
Resolving deltas: 100% (124/124), completed with 32 local objects.
From https://github.com/ggml-org/llama.cpp
 * [new ref]             refs/pull/22673/head -> pr-22673-mtp
bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ git checkout pr-22673-mtp
Switched to branch 'pr-22673-mtp'
bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ git status
On branch pr-22673-mtp
nothing to commit, working tree clean
bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ git merge pr-22673-mtp -m "Merge MTP support from PR #22673"

빌드

bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ cmake --build build -j$(nproc)

llama-server 설치

bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ sudo cmake --install build

llama-server 버전 확인

bluesanta@ubuntu:~/llm/llama.cpp-22673-mtp$ ./build/bin/llama-server --version
version: 9173 (0672285b2)
built with GNU 11.4.0 for Linux aarch64

출처

• GitHub - antirez/ds4: DeepSeek 4 Flash local inference engine for Metal and CUDA · GitHub
• antirez/ds4 - Metal용 DeepSeek V4 Flash 로컬 추론 엔진 | GeekNews

소수 다운로드

(.venv) bluesanta@gx10-3b16:~/llm$ git clone https://github.com/antirez/ds4.git
(.venv) bluesanta@gx10-3b16:~/llm$ cd ds4/

모델 다운로드

(.venv) bluesanta@gx10-3b16:~/llm/ds4$ ./download_model.sh q2-imatrix 
Downloading DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
from https://huggingface.co/antirez/deepseek-v4-gguf
If the download stops, run the same command again to resume it.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1479  100  1479    0     0    279      0  0:00:05  0:00:05 --:--:--   344
100 80.7G  100 80.7G    0     0  51.7M      0  0:26:37  0:26:37 --:--:-- 51.4M
Linked ./ds4flash.gguf -> /home/bluesanta/llm/ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf
 
Done.

빌드

(.venv) bluesanta@gx10-3b16:~/llm/ds4$ make cuda-spark

실행

bluesanta@gx10-3b16:~/llm/ds4$ ./ds4 -p "모델 이름 알려죠"
ds4: context buffers 751.71 MiB (ctx=32768, backend=cuda, prefill_chunk=2048, raw_kv_rows=2304, compressed_kv_rows=8194)
ds4: CUDA backend initialized on NVIDIA GB10 (sm_121)
ds4: CUDA registered 80.76 GiB model mapping for device access

ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 0.000s
ds4: cuda backend initialized for graph diagnostics
We need to answer the user's query. The user asked "모델 이름 알려죠" which is Korean for "Tell me the model name" or "What's your model name?" So we need to respond with the model name. The assistant should state its name. Typically, the assistant might say something like "저는 DeepSeek입니다." But need to check the context. The user didn't specify which model. Probably the assistant is a DeepSeek model. So answer accordingly.
저는 DeepSeek 모델입니다. 도움이 필요하시면 언제든지 물어보세요! 😊
ds4: prefill: 9.25 t/s, generation: 4.39 t/s

서비스 파일 생성

bluesanta@gx10-3b16:~/llm/ds4$ sudo vi /etc/systemd/system/ds4-server.service

[Unit]
Description=DS4 LLM Server
After=network.target

[Service]
Type=simple
User=bluesanta
WorkingDirectory=/home/bluesanta/llm/ds4
ExecStart=/home/bluesanta/llm/ds4/ds4-server --host 0.0.0.0 --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-space-mb 8192
Restart=on-failure
RestartSec=10
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

# 로깅 설정
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

서비스 등록

bluesanta@gx10-3b16:~/llm/ds4$ sudo systemctl enable ds4-server
Created symlink /etc/systemd/system/multi-user.target.wants/ds4-server.service → /etc/systemd/system/ds4-server.service.

서비스 실행

bluesanta@gx10-3b16:~/llm/ds4$ sudo systemctl start ds4-server

서비스 상태 확인

bluesanta@gx10-3b16:~/llm/ds4$ sudo systemctl start ds4-server
bluesanta@gx10-3b16:~/llm/ds4$ 
bluesanta@gx10-3b16:~/llm/ds4$ 
bluesanta@gx10-3b16:~/llm/ds4$ sudo systemctl status ds4-server
● ds4-server.service - DS4 LLM Server
     Loaded: loaded (/etc/systemd/system/ds4-server.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-05-21 23:15:40 KST; 19s ago
   Main PID: 953632 (ds4-server)
      Tasks: 3 (limit: 153548)
     Memory: 634.1M (peak: 634.1M)
        CPU: 5.399s
     CGroup: /system.slice/ds4-server.service
             └─953632 /home/bluesanta/llm/ds4/ds4-server --host 0.0.0.0 --ctx 100000 --kv-disk-dir /tmp/ds4-kv --kv-disk-sp>
 
 5월 21 23:15:40 gx10-3b16 ds4-server[953632]: ds4: CUDA host registration skipped: operation not supported
 5월 21 23:15:41 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors into device cache
 5월 21 23:15:44 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors 16.02 GiB cached
 5월 21 23:15:48 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors 32.06 GiB cached
 5월 21 23:15:52 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors 48.02 GiB cached
 5월 21 23:15:55 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors 64.06 GiB cached
 5월 21 23:15:59 gx10-3b16 ds4-server[953632]: ds4: CUDA loading model tensors 80.04 GiB cached
 5월 21 23:15:59 gx10-3b16 ds4-server[953632]: ds4: CUDA startup model cache prepared 80.76 GiB of tensor spans in 19.009s
 5월 21 23:15:59 gx10-3b16 ds4-server[953632]: ds4: cuda backend initialized for graph diagnostics
 5월 21 23:15:59 gx10-3b16 ds4-server[953632]: 0521 23:15:59 ds4-server: context buffers 1896.58 MiB (ctx=100000, backend=c>

서비스 로그 확인

bluesanta@gx10-3b16:~/llm/ds4$ sudo journalctl -u ds4-server -f

확인

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"deepseek-v4-flash",
    "messages":[{"role":"user","content":"List three Redis design principles."}],
    "stream":true
  }'

openclaude 설정

export CLAUDE_CODE_USE_OPENAI=1
export OPENAI_API_KEY=deepseek-v4-flash
export OPENAI_BASE_URL=http://192.168.0.240:8000/v1
export OPENAI_MODEL="DeepSeek V4 Flash"

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

jetpack 버전 확인

bluesanta@ubuntu:~$ sudo apt show nvidia-jetpack
Package: nvidia-jetpack
Version: 6.2.2+b24
Priority: standard
Section: metapackages
Source: nvidia-jetpack (6.2.2)
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-jetpack-runtime (= 6.2.2+b24), nvidia-jetpack-dev (= 6.2.2+b24)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.3 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r36.5/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

CUDA 버전 확인

bluesanta@ubuntu:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Wed_Aug_14_10:14:07_PDT_2024
Cuda compilation tools, release 12.6, V12.6.68
Build cuda_12.6.r12.6/compiler.34714021_0

가상환경 만들기

bluesanta@bluesanta-desktop:~$ cd llm
bluesanta@bluesanta-desktop:~/llm$ python -m venv .venv
bluesanta@bluesanta-desktop:~/llm$ source .venv/bin/activate
(.venv) bluesanta@bluesanta-desktop:~/llm$ 

모델 다운로드 (Hugging Face)

(.venv) bluesanta@ubuntu:~/llm$ pip install -U "huggingface_hub[cli]"
(.venv) bluesanta@ubuntu:~/llm$ hf download Qwen/Qwen3.6-35B-A3B --local-dir ~/llm/models/Qwen3.6-35B-A3B

원본 모델을 GGUF(FP16)로 변환

(.venv) bluesanta@gx10-3b16:~/llm/llama.cpp$ python convert_hf_to_gguf.py ~/llm/models/Qwen3.6-35B-A3B --outfile ~/llm/models/Qwen3.6-35B-A3B-F16.gguf --outtype f16
 
INFO:gguf.gguf_writer:Writing the following files:
INFO:gguf.gguf_writer:/home/bluesanta/llm/models/Qwen3.6-35B-A3B-F16.gguf: n_tensors = 733, total_size = 69.4G
Writing: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69.4G/69.4G [01:36<00:00, 718Mbyte/s]
INFO:hf-to-gguf:Model successfully exported to /home/bluesanta/llm/models/Qwen3.6-35B-A3B-F16.gguf

양자화 (Q5_K_M)

(.venv) bluesanta@gx10-3b16:~/llm/llama.cpp$ llama-quantize --leave-output-tensor ~/llm/models/Qwen3.6-35B-A3B-F16.gguf ~/llm/models/Qwen3.6-35B-A3B-Q5_K_M.gguf Q5_K_M
 
[ 731/ 733] blk.39.ffn_up_exps.weight            - [  2048,    512,    256,      1], type =    f16, converting to q5_K .. size =   512.00 MiB ->   176.00 MiB
[ 732/ 733] blk.39.ffn_up_shexp.weight           - [  2048,    512,      1,      1], type =    f16, converting to q5_K .. size =     2.00 MiB ->     0.69 MiB
[ 733/ 733] blk.39.post_attention_norm.weight    - [  2048,      1,      1,      1], type =    f32, size =    0.008 MiB
llama_model_quantize_impl: model size  = 66152.24 MiB (16.01 BPW)
llama_model_quantize_impl: quant size  = 24145.21 MiB (5.84 BPW)
 
main: quantize time = 473736.51 ms
main:    total time = 473736.51 ms

멀티모달 프로젝터 (mmproj) 파일 생성

(.venv) bluesanta@gx10-3b16:~/llm/llama.cpp$ python convert_hf_to_gguf.py ~/llm/models/Qwen3.6-35B-A3B --outfile  ~/llm/models/Qwen3.6-35B-A3B-mmproj-f16.gguf --outtype f16 --mmproj
INFO:hf-to-gguf:Loading model: Qwen3.6-35B-A3B
INFO:hf-to-gguf:Model architecture: Qwen3_5MoeForConditionalGeneration
 
INFO:gguf.gguf_writer:/home/bluesanta/llm/models/Qwen3.6-35B-A3B-mmproj-f16.gguf: n_tensors = 334, total_size = 899.3M
Writing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 899M/899M [00:00<00:00, 1.48Gbyte/s]
INFO:hf-to-gguf:Model successfully exported to /home/bluesanta/llm/models/Qwen3.6-35B-A3B-mmproj-f16.gguf

llama.cpp 이용한 확인

(.venv) bluesanta@ubuntu:~/llm$ llama-cli -m ~/llm/models/Qwen3.6-35B-A3B-Q5_K_M.gguf  -n 512 --color on -ngl 99 -p "당신은 유능한 AI 어시스턴트입니다. Jetson AGX Orin에 대해 설명해주세요."
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 62827 MiB):
  Device 0: Orin, compute capability 8.7, VMM: yes, VRAM: 62827 MiB
 
Loading model...  
 
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b9123-927dada6c
model      : Qwen3.6-35B-A3B-Q5_K_M.gguf
modalities : text
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
 
 
> 당신은 유능한 AI 어시스턴트입니다. Jetson AGX Orin에 대해 설명해주세요.
 
[Start thinking]
Here's a thinking process:
 
1.  **Analyze User Input:**
   - **User:** "당신은 유능한 AI 어시스턴트입니다. Jetson AGX Orin 에 대해 설명해주세요."
   - **Language:** Korean
   - **Topic:** Jetson AGX Orin (NVIDIA's edge AI computing platform)
   - **Request:** Explain/describe it
 
2.  **Identify Key Information Needed:**
   - What is it? (Overview/Definition)
   - Key Specifications (CPU, GPU, Memory, AI Performance, I/O, Power)
   - Target Applications/Use Cases
   - Key Features/Advantages
   - Ecosystem/Software Support
   - Comparison/Context (if relevant, but keep it concise)
   - Structure the response logically in Korean
 
3.  **Gather Facts (Internal Knowledge up to 2024):**
   - *Manufacturer:* NVIDIA
   - *Series:* Jetson family (edge AI)
   - *Architecture:* Ampere GPU architecture, Carmel CPU cores
   - *AI Performance:* Up to 275 TOPS (INT8)
   - *GPU:* 1024-core NVIDIA Ampere architecture GPU with 32 Tensor cores
   - *CPU:* 12-core ARM64 CPU (8 high-performance + 4 high-efficiency)
   - *Memory:* 64GB LPDDR5 (102.4 GB/s bandwidth)
   - *Storage/Connectivity:* PCIe 4.0, NVMe SSD support, USB 3.2, Ethernet, MIPI CSI-2, etc.
   - *Power:* 60W max (configurable), TDP 60W
   - *Use Cases:* Autonomous mobile robots, smart cameras, medical devices, industrial automation, drones, etc.
   - *Software:* JetPack SDK, CUDA, cuDNN, TensorRT, DeepStream, Isaac ROS, Docker support, Linux-based
   - *Form Factors:* Developer Kit, Industrial Module (for OEMs), Edge Cloud Gateway
   - *Successor/Context:* Follows Xavier AGX, successor to Jetson AGX Xavier, part of NVIDIA's edge AI lineup
 
4.  **Structure the Response (in Korean):**
   - 소개 (Overview)
   - 주요 스펙 (Key Specs
 
[ Prompt: 111.5 t/s | Generation: 30.9 t/s ]
 
> 

API 서버 실행

(.venv) bluesanta@bluesanta-desktop:~/llm/llama.cpp$ llama-server -m ~/llm/models/Qwen3.6-35B-A3B-Q4_K_M.gguf --host 0.0.0.0 --port 8080 -ngl 99 -c 32768 -np 1
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 62840 MiB):
  Device 0: Orin, compute capability 8.7, VMM: yes, VRAM: 62840 MiB
build_info: b8851-e365e658f
system_info: n_threads = 12 (n_threads_batch = 12) / 12 | CUDA : ARCHS = 500,610,700,750,800,860,890 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : NEON = 1 | ARM_FMA = 1 | FP16_VA = 1 | DOTPROD = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
init: using 11 threads for HTTP server
start: binding port with default address family
 
srv    load_model: loading model '/home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q4_K_M.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 21098 MiB of device memory vs. 52336 MiB of free device memory
llama_params_fit_impl: will leave 31238 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 1.10 seconds
llama_model_load_from_file_impl: using device CUDA0 (Orin) (0000:00:00.0) - 52362 MiB free
llama_model_loader: loaded meta data with 44 key-value pairs and 733 tensors from /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q4_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
 
init: chat template, example_format: '<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

'
srv          init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle

서비스 파일 생성

(.venv) bluesanta@bluesanta-desktop:~/llm$ sudo vi /etc/systemd/system/llama.service

텍스트 전용

[Unit]
Description=Llama.cpp Server Service
After=network.target

[Service]
# 사용자 계정
User=bluesanta
Group=bluesanta
LimitMEMLOCK=infinity
WorkingDirectory=/opt/llama.cpp

# 최적화된 실행 명령어
# -m /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
# -m /home/bluesanta/llm/models/gemma-4-26b-Q4_K_M.gguf
# --ctx-size 262144
ExecStart=/usr/local/bin/llama-server \
    -m /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q5_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --ctx-size 262144 \
    --n-gpu-layers 99 \
    --flash-attn on \
    --spec-draft-n-max 2 \
    --mlock \
    --cont-batching \
    --metrics
    
# 프로세스 종료 시 자동 재시작 설정
# Restart=always
# RestartSec=5

[Install]
WantedBy=multi-user.target

--mmproj 적용

[Unit]
Description=Llama.cpp Server Service
After=network.target

[Service]
# 사용자 계정
User=bluesanta
Group=bluesanta
LimitMEMLOCK=infinity
WorkingDirectory=/opt/llama.cpp

# 최적화된 실행 명령어
# -m /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q4_K_M.gguf \
# -m /home/bluesanta/llm/models/gemma-4-26b-Q4_K_M.gguf
# -m /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q5_K_M.gguf \
# --ctx-size 262144
# --ctx-size를 65536 (64k) 또는 32768 (32k)로 변경
# --spec-type ngram-mod,draft-mtp --spec-draft-n-max 4
ExecStart=/usr/local/bin/llama-server \
    -m /home/bluesanta/llm/models/Qwen3.6-35B-A3B-Q5_K_M.gguf \
    --mmproj /home/bluesanta/llm/models/Qwen3.6-35B-A3B-mmproj-f16.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --ctx-size 131072 \
    --n-gpu-layers 99 \
    --flash-attn on \
    --spec-draft-n-max 2 \
    --mlock \
    --cont-batching \
    --metrics


# 프로세스 종료 시 자동 재시작 설정
# Restart=always
# RestartSec=5

[Install]
WantedBy=multi-user.target

서비스 갱신

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ sudo systemctl daemon-reload

서비스 실행

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ sudo systemctl start llama

서비스 상태 확인

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ sudo systemctl status llama

서비스 로그 확인

(.venv) bluesanta@ubuntu:~/llm/llama.cpp$ sudo journalctl -u llama.service -f

확인

(.venv) bluesanta@ubuntu:~/llm$ curl http://localhost:8000/completion -H "Content-Type: application/json" -d '{
  "prompt": "Jetson AGX Orin의 장점 3가지는?",
  "n_predict": 256
}'
{"index":0,"content":"\n\n\n\n\n\nNVIDIA Jetson AGX Orin은 에지 AI(Edge AI) 애플리케이션을 위한 최상위 성능의 임베디드 컴퓨팅 모듈로, 기존 제품 대비 뛰어난 성능과 효율성을 자랑합니다. 주요 장점 3가지는 다음과 같습니다:\n\n1. **뛰어난 AI 추론 성능 (500 TOPS)**  \n   Jetson AGX Orin은 최대 **500 TOPS**(초당 500조 회 연산)의 INT8 추론 성능을 제공합니다. 이는 이전 세대인 Jetson Xavier NX 대비 약 **20배 이상** 향상된 것으로, 대규모 딥러닝 모델(예: YOLO, ResNet, Vision Transformer 등)을 실시간으로 처리할 수 있어 복잡한 비전 AI, 로봇 공학, 자율 주행 등 고성능이 요구되는 애플리케이션에 이상적입니다.\n\n2. **높은 전력 효율성 대비 고성능**  \n   최대 60W까지 전력 소비를 지원하지만, 필요에 따라 **5W부터 60W까지 유연하게 전력 구성**이 가능합니다. 이는 제한된 전력과 열 설계(Power & Thermal Budget) 환경에서도 고성능을 유지하면서도 에너지 효율","tokens":[],"id_slot":3,"stop":true,"model":"Qwen3.6-35B-A3B-Q5_K_M.gguf","tokens_predicted":256,"tokens_evaluated":13,"generation_settings":{"seed":4294967295,"temperature":1.0,"dynatemp_range":0.0,"dynatemp_exponent":1.0,"top_k":20,"top_p":0.949999988079071,"min_p":0.05000000074505806,"top_n_sigma":-1.0,"xtc_probability":0.0,"xtc_threshold":0.10000000149011612,"typical_p":1.0,"repeat_last_n":64,"repeat_penalty":1.0,"presence_penalty":0.0,"frequency_penalty":0.0,"dry_multiplier":0.0,"dry_base":1.75,"dry_allowed_length":2,"dry_penalty_last_n":262144,"dry_sequence_breakers":["\n",":","\"","*"],"mirostat":0,"mirostat_tau":5.0,"mirostat_eta":0.10000000149011612,"stop":[],"max_tokens":256,"n_predict":256,"n_keep":0,"n_discard":0,"ignore_eos":false,"stream":false,"logit_bias":[],"n_probs":0,"min_keep":0,"grammar":"","grammar_lazy":false,"grammar_triggers":[],"preserved_tokens":[],"chat_format":"Content-only","reasoning_format":"deepseek","reasoning_in_content":false,"generation_prompt":"","samplers":["penalties","dry","top_n_sigma","top_k","typ_p","top_p","min_p","xtc","temperature"],"speculative.types":"none","timings_per_token":false,"post_sampling_probs":false,"backend_sampling":false,"lora":[]},"prompt":"Jetson AGX Orin의 장점 3가지는?","has_new_line":true,"truncated":false,"stop_type":"limit","stopping_word":"","tokens_cached":268,"timings":{"cache_n":0,"prompt_n":13,"prompt_ms":434.387,"prompt_per_token_ms":33.41438461538461,"prompt_per_second":29.927230787293357,"predicted_n":256,"predicted_ms":8410.958,"predicted_per_token_ms":32.8553046875,"predicted_per_second":30.436485356364873}}

export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000
export CLAUDE_CODE_USE_OPENAI=1
export OPENAI_API_KEY=sk-jetson-qwen36
export OPENAI_BASE_URL=http://192.168.0.235:8000/v1
export OPENAI_MODEL=Qwen3.6-35B-A3B-Q5_K_M.gguf