Run DeepSeek-R1 Distill Locally on RK3588

Running a reasoning model entirely on an RK3588 development board is now practical when the model, quantization and runtime are chosen for the platform. A useful example is DeepSeek-R1-Distill-Qwen-1.5B converted with Rockchip's RKLLM toolchain and executed through the RK3588 NPU. The result can support offline assistants, equipment documentation search, local text analysis and other privacy-sensitive edge applications without sending every prompt to a cloud API.

“Runs locally” does not mean every RK3588 board will deliver the same speed. Memory capacity and bandwidth, RKLLM version, RKNPU driver, quantization settings, context length, CPU/NPU clocks, cooling and prompt shape all influence the result. This guide therefore documents a reproducible deployment path and keeps the reported benchmark tied to the tested configuration.

The worked example uses a user-supplied YY3588 board configuration and an RKLLM 1.2.2 conversion/runtime stack. Rockchip's public repository has continued to evolve, so a new deployment should first check the current release and its compatibility notes rather than copying the version numbers blindly.

Why the 1.5B distilled model fits this edge target #

DeepSeek released six dense distilled checkpoints based on Qwen and Llama families. DeepSeek-R1-Distill-Qwen-1.5B is the smallest official checkpoint in that group and is based on Qwen2.5-Math-1.5B. DeepSeek says its distilled models were fine-tuned with reasoning samples generated by DeepSeek-R1, allowing a much smaller model to reproduce part of the larger model's reasoning behavior.

That makes the 1.5B checkpoint a sensible first target for RK3588 because it reduces three difficult edge constraints:

• Model weights fit into a much smaller memory budget after quantization.
• Prefill and token generation place less sustained pressure on memory bandwidth.
• Conversion and on-board debugging finish faster than with 7B- or 14B-class models.

The trade-off is capability. A 1.5B distilled model is not equivalent to the full DeepSeek-R1 service, and a benchmark score does not guarantee reliability on industrial instructions, safety decisions or domain-specific questions. Local deployment should include application-specific evaluation, output constraints and a fallback path.

Hardware and software boundary #

Rockchip specifies the RK3588 with four Cortex-A76 and four Cortex-A55 CPU cores and an NPU rated up to 6 TOPS. RKLLM supports the RK3588 series and lists DeepSeek-R1-Distill among its supported model families. Those official statements establish platform capability, while the carrier board determines RAM, storage, thermal design and exposed interfaces.

The illustrated YY3588 configuration exposes dual Ethernet, multiple USB ports, HDMI input/output, serial buses, CAN, wireless expansion and a removable compute module. These labels describe the pictured board revision; they should not be generalized to every RK3588 board. For an order or project release, confirm the current schematic, module revision, RAM/eMMC configuration and BSP image.

For this model class, use the following as planning guidance rather than universal minimums:

The RKLLM deployment pipeline #

RKLLM separates conversion from inference. The host-side toolkit imports a Hugging Face-format checkpoint, applies optimization and quantization, and exports a Rockchip `.rkllm` model. The target-side runtime loads that file and calls the NPU driver from an ARM64 application.

The components must be treated as one compatibility set:

1. RKLLM Toolkit on the x86-64 conversion machine. 2. RKLLM Runtime libraries on the RK3588 board. 3. RKNPU kernel driver delivered with the board BSP. 4. The converted `.rkllm` model, which records target and quantization choices. 5. The C/C++ demo or application linked against the matching runtime.

Mixing an old converted model with an arbitrary new runtime can cause load failures, output changes or crashes. Record the repository tag, wheel filename, runtime library hash, driver version and conversion command with each released model.

Step 1: obtain and verify the official checkpoint #

Use the official `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` model repository rather than a similarly named community conversion. A Hugging Face-style directory normally includes the configuration, tokenizer and one or more weight files.

For procurement and deployment control, record:

• Model repository and exact revision or commit.
• License files for both the distilled model and its base model.
• File hashes after download.
• Whether Git LFS completed rather than leaving pointer files.
• Any tokenizer or configuration modifications required by the selected RKLLM release.

DeepSeek's model card recommends particular generation settings for the R1 family and warns that evaluation should use repeated trials. Preserve those recommendations in the test plan rather than judging the model from one prompt.

Step 2: create an isolated conversion environment #

The supplied deployment record used Ubuntu 24.04, Python 3.12 and the RKLLM Toolkit 1.2.2 wheel built for CPython 3.12 on x86-64 Linux. Because this is a dated test stack rather than a permanent requirement, first check the package matrix in the current Rockchip repository.

An isolated environment prevents the toolkit's dependencies from changing the system Python:

```bash python3 -m venv rkllm_venv source rkllm_venv/bin/activate pip install ./rkllm_toolkit-1.2.2-cp312-cp312-linux_x86_64.whl ```

Do not select a wheel solely because the filename looks close. Match the RKLLM release, Python ABI, operating system and host architecture. Keep the wheel itself in the build archive so the model can be reproduced later.

Step 3: prepare representative quantization data #

The conversion workflow uses a calibration dataset when building a quantized model. Rockchip's example tree includes scripts for generating a `data_quant.json` file from the model directory. The source procedure used:

```bash python3 generate_data_quant.py -m ./DeepSeek-R1-Distill-Qwen-1.5B/ ```

Calibration data influences activation ranges and therefore accuracy. A file generated from generic samples may be acceptable for a first demo, but a production model should be evaluated with prompts representative of its real language, length and domain. Quantization acceptance should compare outputs against the source checkpoint across a fixed evaluation set.

Step 4: build a W8A8 RK3588 model #

The tested conversion used W8A8 group quantization, three NPU cores, a 4096-token maximum context and RK3588 as the target. The essential structure is:

```python from rkllm.api import RKLLM

converter = RKLLM() converter.load_huggingface( model="./DeepSeek-R1-Distill-Qwen-1.5B", model_lora=None, device="cpu", ) converter.build( do_quantization=True, optimization_level=1, quantized_dtype="w8a8_g128", quantized_algorithm="normal", num_npu_core=3, dataset="./data_quant.json", hybrid_rate=0, target_platform="rk3588", max_context=4096, ) converter.export_rkllm(export_path="./deepseek-r1-distill-qwen-1.5b.rkllm") ```

Every return code should be checked in the actual build script. Save the console log, toolkit version and output hash. If optimization level, quantization type, context or hybrid rate changes, create a new model artifact rather than overwriting the previous file; the performance and output quality are no longer directly comparable.

What W8A8 changes #

`W8A8` quantizes weights and activations to 8-bit representations, while `g128` indicates grouped quantization with a group size of 128 in this workflow. It reduces memory and can improve NPU execution efficiency, but it may alter model output. The correct test is not simply whether the model answers—it is whether accuracy, stability, memory and speed remain acceptable for the intended prompts.

Step 5: align the board runtime and driver #

Before copying the model, confirm the NPU driver exposed by the board image. On an RK3588 Linux target, the available debug node depends on the BSP, but the supplied procedure checked the Rockchip NPU driver version through the kernel debug filesystem.

The target application needs the ARM64 RKLLM runtime libraries and the converted model. On Android, the source record placed `libomp.so` and `librkllmrt.so` under a dedicated application directory and built the Rockchip demo with Android NDK r21e. On Debian, library placement and the build path differ, but the same rule applies: the headers, libraries, model and driver must belong to a validated combination.

For a production image, avoid relying on a manually edited shell environment. Package libraries in a controlled path, define the service account and permissions, verify the model hash at startup, and log runtime/driver versions with every inference service launch.

Step 6: run inference and measure the whole system #

The source configuration launched Rockchip's demo with a 1024-token input setting and 2048-token output setting:

```bash RKLLM_LOG_LEVEL=2 \ LD_LIBRARY_PATH=/data/rkllm/lib:$LD_LIBRARY_PATH \ ./llm_demo ./deepseek-r1-distill-qwen-1.5b.rkllm 1024 2048 ```

In that user-supplied test, the runtime log reported approximately 8 output tokens/s, about 1.84 GB peak memory, and NPU activity around 75% during generation. These are observed values for one board, model build, prompt and software stack—not a guaranteed RK3588 specification. The earlier source narrative also mentioned a higher 15.4 tokens/s figure without an equally clear test record, so this article does not use it as a verified benchmark.

A credible benchmark report should include:

“Fluent” is application-dependent. Eight tokens per second may feel responsive for a local maintenance assistant, but a long chain-of-thought response can still take significant time. Measure complete answer latency and task success, not just steady-state decode speed.

Optimization priorities after the first successful run #

Control context length #

KV-cache memory grows with context. Setting a 4096-token maximum does not mean every application should always fill it. Constrain retrieved documents, summarize conversation history and reject oversized requests before they create memory pressure.

Keep the model service warm #

Model loading and first-run initialization can dominate sporadic workloads. A supervised service can load the model once, expose a local API and serialize requests according to the memory budget. Add health checks and a bounded restart policy rather than launching a new process for each prompt.

Separate thermal testing from functional testing #

A five-minute demo may never reach the temperature of a sealed industrial enclosure. Run repeated prompts while camera, storage and network services are active, then inspect CPU/NPU clocks and token latency over time.

Evaluate quantization, not only conversion success #

Build at least one reference evaluation set covering the expected language and tasks. Compare factuality, formatting, refusal behavior and reasoning consistency between the original checkpoint and the `.rkllm` artifact. If a different quantization improves speed, make the quality trade-off visible before release.

Secure the local endpoint #

Offline inference improves data locality, but a listening service can still expose prompts, model outputs or system access. Bind only to required interfaces, authenticate clients, sanitize logs, limit prompt size and treat model-generated commands as untrusted input.

Board sourcing and deployment checks #

An RK3588 board for local LLM work is more than the processor label. Confirm these items before ordering a batch:

• Exact RK3588/RK3588S module and carrier revisions.
• Installed LPDDR capacity and approved memory vendor/part changes.
• eMMC or SSD capacity, endurance, interface and production availability.
• Cooling solution under sustained NPU load.
• Debian/Android BSP version, kernel, RKNPU driver and update policy.
• Availability of the source, device tree and recovery image.
• Power input range and brownout behavior during NPU load transitions.
• Exposed Ethernet, USB, display, camera and industrial interfaces on the ordered revision.
• Model/runtime license obligations and a reproducible build archive.
• Long-term support owner for the board BSP and RKLLM application.

For an RFQ, include board quantity, RAM/eMMC configuration, operating system, required interfaces, temperature range, thermal method and whether the supplier must preload a specific image or model artifact. If performance is contractual, attach the exact benchmark procedure rather than writing only “DeepSeek-R1 must run smoothly.”

Conclusion #

DeepSeek-R1-Distill-Qwen-1.5B is a realistic starting point for local reasoning-model inference on RK3588. Rockchip's RKLLM stack provides the supported conversion and runtime path, while W8A8 quantization brings the model into an edge-friendly memory class. The supplied test demonstrates useful local output at roughly 8 tokens/s with about 1.84 GB peak memory on one YY3588 configuration.

The transferable result is the workflow, not the headline speed: freeze the model revision, align toolkit/runtime/driver versions, build with documented quantization data, measure the entire board under sustained load and validate output quality for the actual application. That is what turns an RK3588 DeepSeek demo into a maintainable edge-AI product.

Official references #

• DeepSeek-R1 official model repository and distilled-model documentation
• DeepSeek-R1 official GitHub repository
• Rockchip RKLLM official repository
• Rockchip RKLLM releases and compatibility notes
• Rockchip RK3588 official development-board announcement
• Rockchip RK3588 brief datasheet