Due to high Vulkan backend demand, I update TensorSharp and release the initial version of GGML Vulkan backend by leveraging external GGML project. The native Vulkan backend will be implemented later. I tested it on Nvidia Geforce RTX 3080 Laptop GPU, and Intel(R) UHD Graphics on Windows. They all work. However, I do not have AMD GPU, so I have no way to get it tested. It's really appreciated if you have AMD GPU and would like to try it out. Any feedback and comment are welcome.
Here is the benchmark I run to compare with llama.cpp:
Performance ratio — TensorSharp vs reference engines
Geomean of TensorSharp's per-scenario speedup over each reference engine on the same backend, across every scenario both engines ran (single-stream, MTP-off). A value > 1.0× means TensorSharp is faster (for decode / prefill throughput) or lower-latency (for TTFT); — = no overlapping cells. Per-scenario ratios are in each model's section below.
Model Comparison decode prefill TTFT
Gemma 4 E4B it (Q8_0, dense multimodal) vs llama.cpp · Vulkan 0.93× 0.96× 0.95×
Gemma 4 12B it (QAT UD-Q4_K_XL, dense) vs llama.cpp · Vulkan 1.18× 0.97× 0.95×
Gemma 4 E4B it (Q8_0, dense multimodal) (gemma4-e4b)
Decode throughput (tok/s)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 41.6 45.3
text_long 40.9 44.5
multi_turn 41.3 43.6
function_call 41.2 44.4
Prefill throughput (tok/s)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 1641.7 1641.1
text_long 1157.0 1718.1
multi_turn 1695.5 1454.3
function_call 1661.2 1531.6
Time to first token (ms, lower is better)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 1203.0 1187.0
text_long 2719.0 1813.0
multi_turn 1235.0 1422.0
function_call 1219.0 1328.0
Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)
Decode throughput
Scenario vs llama.cpp · Vulkan
text_short 0.92×
text_long 0.92×
multi_turn 0.95×
function_call 0.93×
Prefill throughput
Scenario vs llama.cpp · Vulkan
text_short 1.00×
text_long 0.67×
multi_turn 1.17×
function_call 1.08×
Time to first token (latency; > 1.0× = TensorSharp lower)
Scenario vs llama.cpp · Vulkan
text_short 0.99×
text_long 0.67×
multi_turn 1.15×
function_call 1.09×
Gemma 4 12B it (QAT UD-Q4_K_XL, dense) (gemma4-12b)
Decode throughput (tok/s)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 31.3 31.1
text_long 31.4 30.0
multi_turn 30.9 31.6
function_call 60.8 31.9
Prefill throughput (tok/s)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 766.1 729.4
text_long 635.2 647.4
multi_turn 617.5 636.6
function_call 587.4 674.7
Time to first token (ms, lower is better)
Scenario TensorSharp · Vulkan llama.cpp · Vulkan
text_short 2578.0 2672.0
text_long 4953.0 4813.0
multi_turn 3391.0 3250.0
function_call 3531.0 3016.0
Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)
Decode throughput
Scenario vs llama.cpp · Vulkan
text_short 1.01×
text_long 1.05×
multi_turn 0.98×
function_call 1.91×
Prefill throughput
Scenario vs llama.cpp · Vulkan
text_short 1.05×
text_long 0.98×
multi_turn 0.97×
function_call 0.87×
Time to first token (latency; > 1.0× = TensorSharp lower)
Scenario vs llama.cpp · Vulkan
text_short 1.04×
text_long 0.97×
multi_turn 0.96×
function_call 0.85×
In case you didn't know what is TensorSharp, here is an introduction:
TensorSharp is an open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), image edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability (support Cuda, Metal and Vulkan backends). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp
This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implemented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.
I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quantized from llama.cpp and other optimizations for prefill and decode.
Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.
United States
NORTH AMERICA
Related News
🚀 I Built a Dropshipping Automation Pipeline — Here's What I Learned (and What I'd Do Differently)
10h ago
How I Cut My LLM API Bill by 40x: A Freelancer's Migration Story
10h ago

Mattress Firm Coupons: Save up to $600
3h ago
Google Ordered to Pay $2 Billion For Anti-Competitive Practices By Swedish Court
20h ago
The Censorship Wall: Why Every AI Companion App Ends Up Filtering You
20h ago