Every organization that decides to run AI on-premise reaches the same question: what kind of server do we need? Too big wastes the budget; too small stalls the project. Fortunately, the basic logic is simple.
1. Model size sets the memory
A language model's size is expressed in parameters. To fit in GPU memory, a model needs roughly parameter count × bytes per parameter. At 16-bit precision each parameter takes 2 bytes, so a 7-billion-parameter model needs about 14 GB just for its weights. Compressed to 4 bits (quantization), that need drops to roughly a quarter.
2. Users need extra memory
Once the model fits, every conversation needs memory too; long documents and long conversations increase it. That is why how many people use the system at once, and with how much text, matters as much as model size.
3. One model is not enough
An enterprise installation usually runs several models together:
- a language model for chat and writing,
- an embedding model to make documents searchable,
- a vision model for scanned documents and images.
Their memory must be calculated together.
4. Leave room to grow
Usage grows after the system is installed. New teams join, new work is added. Leaving headroom above today's need, or choosing a server that can take more GPUs later, avoids a new purchase within the first year.
5. Measure first, then buy
The safest path is a small trial with samples of your real work, measuring response times. The calculation on paper shows the direction; measurement settles the decision.
With the Bizz AI Ready service, we size the server with you and deliver it with the models and Bizz AI Core installed.