Same machine is simpler and faster until memory pressure turns your p99 into a random number.
In short
The vector index lives in RAM, the model lives in VRAM — they compete only when the host runs out of system memory. Size the index first, then decide.
A vector database and a model on the same host do not fight over the same memory. The model occupies GPU memory. The index occupies system memory, unless you deliberately built a GPU index.
So the question is not "do they fit in 24 GB" — it is "does the host have enough RAM for the index plus everything else". Those are different budgets, and conflating them is how people end up renting a bigger GPU to fix a RAM problem.
Raw vectors are easy: count × dimensions × 4 bytes for fp32. One million 768-dimension vectors is about 3 GB.
The index structure is the part people forget. HNSW keeps a graph alongside the vectors, and the overhead is substantial — plan for roughly 50% on top, more at high M values. That same million vectors is then closer to 4.5 GB resident.
Then add headroom. A vector database being written to while serving reads needs room for both the current graph and the rebuild. Sitting at 85% RAM utilisation is not "fine" — it is the state right before the OOM killer picks a victim, and the victim is usually the process serving traffic.
| Component | Lives in | Grows with | Fails how |
|---|---|---|---|
| Model weights | VRAM | parameters × precision | hard OOM at load |
| KV cache | VRAM | context × concurrency | OOM under load |
| Vector index | System RAM | vectors × dims × index type | swap, then OOM killer |
| Reranker | VRAM or CPU | batch size | latency spike |
For a single-tenant service with a modest corpus, co-locating is the better engineering choice. The retrieval call becomes a loopback request instead of a network hop, which removes an entire failure mode: the p99 spike that used to arrive with every network hiccup stops happening.
It is also cheaper and easier to reason about. One machine to monitor, one set of logs, one deploy. At small scale that operational simplicity is worth more than the theoretical isolation you gave up.
# RSS is what matters -- VIRT includes mapped files that are
# not resident and will make you think headroom is smaller
# than it is.
ps -o pid,rss,vsz,cmd -p $(pgrep -d, -f vector)
# Available, not free. `free` counts page cache as used,
# which makes a healthy host look nearly dead.
free -m | awk 'NR==2 {print "available: " $7 " MB"}'Yes, and it is fast — but on the same GPU the model already occupies. A GPU index trades VRAM for latency, which is the wrong trade when model memory is your constraint. Use it only for small corpora with tight latency budgets.
No. Retrieval quality depends on the embedding model and chunking, not on where the index runs. What changes is failure behaviour under memory pressure.
Enough that the index plus CPU-side model buffers stay below 80% of total. Past that you are one concurrent rebuild away from the kernel picking a process to kill — and it does not pick the one you would have.