From Dev.to
Can an 8 GB Edge Device Run a 2 GB Model? Budget Memory at Peak Concurrency
Eight gigabytes of RAM minus a 2 GB model file is not 6 GB of usable inference memory. Calculate the KV cache, then test the longest request and peak concurrency. That subtraction cannot approve a dev...