Jetson AGX Orin 64GB : llama.cpp

출처

• Jetson AGX Orin에서 PyTorch 설치 ::: 대학원 생존일지

jetpack 버전 확인

bluesanta@localhost:~$ sudo apt show nvidia-jetpack -a
Package: nvidia-jetpack
Version: 7.2-b187
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b187), nvidia-jetpack-dev (= 7.2-b187)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package
 
Package: nvidia-jetpack
Version: 7.2-b184
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 198 kB
Depends: nvidia-jetpack-runtime (= 7.2-b184), nvidia-jetpack-dev (= 7.2-b184)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29.6 kB
APT-Sources: https://repo.download.nvidia.com/jetson/common r39.2/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

CUDA 버전 확인

bluesanta@localhost:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2026 NVIDIA Corporation
Built on Thu_Mar_19_11:11:41_PM_PDT_2026
Cuda compilation tools, release 13.2, V13.2.78
Build cuda_13.2.r13.2/compiler.37668154_0

가상환경 만들기

bluesanta@bluesanta-desktop:~$ cd llm
bluesanta@localhost:~/llm$ python -m venv .venv
bluesanta@localhost:~/llm$ source .venv/bin/activate

빌드 관련 페키지 설치

(.venv) bluesanta@localhost:~/llm$ sudo apt install -y git cmake build-essential libopenblas-dev

llama.cpp 빌드

(.venv) bluesanta@localhost:~/llm$ git clone https://github.com/ggerganov/llama.cpp
(.venv) bluesanta@localhost:~/llm$ cd llama.cpp
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
(.venv) bluesanta@localhost:~/llm/llama.cpp$ cmake --build build -j$(nproc)
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo cmake --install build
(.venv) bluesanta@localhost:~/llm/llama.cpp$ sudo ldconfig

llama.cpp 설치 확인

(.venv) bluesanta@localhost:~/llm/llama.cpp$ llama-cli --version
version: 9944 (5c3a58686)
built with GNU 13.3.0 for Linux aarch64
(.venv) bluesanta@localhost:~/llm/llama.cpp$ llama-cli -m ../models/Qwen3.6-35B-A3B-Q5_K_M.gguf --mmproj ../models/Qwen3.6-35B-A3B-mmproj-f16.gguf -p "Hello" -ngl 999
 
 
Loading model...  
 
▄▄ ▄▄
██ ██
██ ██  ▀▀█▄ ███▄███▄  ▀▀█▄    ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██    ██    ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
                                    ██    ██
                                    ▀▀    ▀▀
 
build      : b9944-5c3a58686
model      : ../models/Qwen3.6-35B-A3B-Q5_K_M.gguf
ftype      : Q5_K - Medium
modalities : text, vision, video
 
available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read         add a text file
  /glob      add text files using globbing pattern
  /image        add an image file
  /video        add a video file
 
 
 
> Hello
 
[Start thinking]
 
Here's a thinking process:
 
1.  **Analyze User Input:**
   - User said: "Hello"
   - This is a simple greeting.
 
2.  **Identify Intent:**
   - The user is initiating a conversation.
   - No specific question or task is provided.
 
3.  **Determine Response Strategy:**
   - Acknowledge the greeting warmly.
   - Offer assistance.
   - Keep it concise and open-ended to encourage further interaction.
 
4.  **Draft Response (Mental):**
   - "Hello! How can I assist you today?"
   - Alternative: "Hi there! What can I do for you?"
   - Both are appropriate. I'll go with a friendly, professional tone.
 
5.  **Refine Response:**
   - "Hello! How can I help you today?" (Simple, clear, open-ended)
 
6.  **Final Output Generation:** (Matches the refined response)✅
[End thinking]
 
Hello! How can I help you today?
 
[ Prompt: 39.4 t/s | Generation: 31.6 t/s ]