If you are using Ollama in production to serve LLM modelsYou've probably already realized that it's not just a matter of "install and discard." Between choosing the right model, quantization, getting the right amount of VRAM, the number of simultaneous requests, and multi-stage agents, it's very easy to end up with slow responses, 503 errors, or even crashes due to insufficient memory.
The good news is that, knowing how it manages Don't la concurrency, queues, and memoryBy applying a few good architectural and system practices, you can significantly improve performance on both GPUs and CPUs. Furthermore, this can be done without sacrificing data privacy or the flexibility of local models.
Ollama and llama.cpp in production: parts and roles
Before fine-tuning anything, it's important to understand who does what. llama.cpp is the inference engine, extremely optimized in C++ to get the most out of the hardware (CPU, Apple Silicon, NVIDIA, AMD). Ollama is the high-level “wrapping” that orchestrates that engine and other backends (such as vLLM in some cases), exposing a simple CLI and a ready-to-use REST API.
In practice, when you launch a model with Ollama, what happens is that the application (written in Go) It launches a child process that executes llama.cpp (or another compatible runtime), manages weight offloading, GPU/CPU configuration, context size, and model lifetime in memory. This greatly simplifies production operations compared to using llama.cpp directly, where you would have to compile it, manage routes, and parameters such as –n-gpu-layersquantization, etc.
If we think in terms of analogies, llama.cpp is the tensor surgery centerMinimalist, fine-tuned to get the most out of every CPU/GPU cycle. Ollama is the "IKEA" of local AI: it gives you a pre-packaged system with model management, standard API, request queue, and hardware auto-tuningIdeal for production environments where you don't want to fight with every compilation flag.
Hardware requirements and model selection for production
A key part of ensuring things run smoothly is not to be overly ambitious with the model size relative to your hardware. The combination model parameters + quantization type + context length Determines RAM, VRAM, and inference times.
As a guideline, for production with Ollama on a single machine, these ranges are usually used:
- 8 GB of RAMSmall models (1B, 3B, 7B quantized). Suitable for prototypes and small services, but fluency may suffer under heavy loads.
- 16 GB of RAMA reasonable point for quantized 7B and 13B models of type Q4_K_M. Real service can be provided if concurrency is well controlled.
- 32 GB or moreRecommended if you want to play with 30B, 40B or 70B models, or if you plan to serve several models in parallel.
On GPUs, the pattern is similar: The more VRAM you have, the more layers you can offload to the graphics card. And you'll achieve more throughput. With a 16GB GPU, you can comfortably serve well-quantized 7B-13B models, while for 70B you're already talking about very high-end hardware or multiple GPUs.
Regarding storage, it's important to keep in mind that “Small” quantized models can occupy 2 GBMedium-sized SSDs range from 5 GB or more to very large ones with tens or even hundreds of gigabytes. An NVMe SSD makes a difference when loading or swapping models.
Finally, the CPU still matters, especially if you're doing inference using only the processor or combined with the GPU. 4 cores is the minimum acceptableFor a stable service with several concurrent requests, 8 cores or more are ideal.
Quantization and model formats: how to gain performance without killing quality
For an LLM to be usable in production, you almost always need some form of quantizationIt is the process of converting from floating-point weights (FP16, FP32) to integer representations with fewer bits (4, 8, etc.), reducing the size of the model and the memory it consumes, at the cost of a slight loss of precision.
A rule of thumb often repeated in the community is that Q4_K_M is the reasonable standard for localIt reduces the size to roughly half that of FP16, the loss of quality is around 1-2% in metrics like perplexity, and the inference speed increases considerably. If you need even more compression, you can go down to Q3 or Q2, but at the cost of more hallucinations and worse reasoning.
To use models with Ollama, the standard format is GGUFwhich packages weights, metadata, and a tokenizer in a way optimized for llama.cpp-type runtimes. Many models in the Ollama library already come in GGUF and are quantized, so a ollama pullIf you bring in external models (for example, from Hugging Face), you can:
- Convert from formats like Safetensors to GGUF using the tools of call.cpp (scripts like convert_hf_to_gguf.py).
- Quantize them with binary quantify from llama.cpp choosing the scheme (Q4_K_M, Q5_K_S, etc.).
- Create a Modelfile in Ollama pointing to the .gguf and defining template, default parameters and system.
This flow of download → convert → quantize → register in Ollama It allows integrating niche models into production, such as legal LLMs (e.g., one like Jurema-7B) or domain-specific ones, while maintaining the same deployment pipeline.
Internal model parameters: num_ctx, temperature, and output control
Once the model is chosen, it's time to tame its behavior. In production, it's not enough for it to "respond nicely"; it has to be predictable, limited and efficientThe key parameters exposed by Ollama (inherited from llama.cpp) are:
On one side is num_ctxThe context window defines how many tokens the model can consider simultaneously: system messages, chat history, and current prompt. Larger windows allow for more tokens. long conversations and analysis of extensive documentsHowever, they significantly increase RAM/VRAM usage and calculation time. Furthermore, if you set a value higher than what the model was trained to handle, you may encounter unusual behavior or performance degradation.
It is also crucial to control generation with num_predict (maximum number of exit tokens), lists of stop and temperature. A low temperature value (0,2-0,5) produces more stable and less creative responses, ideal for RAG, coding or verificationsHigh values are reserved for creative uses, which are rarely serious production scenarios.
In addition, options such as top_p y top_k They help limit randomness. Reducing top_p to moderate values (e.g., 0,8-0,9) restricts the space of possible tokens, which is useful for reducing hallucinations and achieving more repeatable outputs.
All these parameters can be persistently set in the Modelfile through instructions PARAMETER, overwrite specifically with the CLI (command /set in interactive mode) or dynamically pass through REST API in the field options from JSON.
Concurrency, queues and batching at Ollama: squeezing the machine without killing it
The real leap from “local toy” to service in production It arrives when you start receiving multiple requests simultaneously. Ollama incorporates its own system of crowds and queues to manage this without having to set up an additional server.
The central piece is the environment variable OLLAMA_NUM_PARALLELThis defines how many requests a loaded model can process in parallel. The default is usually 4 (or 1 if memory is limited). Higher values increase throughput if you have CPU/GPU and VRAM headroom, but they also increase the load on memory and can worsen the latency of each individual request.
When multiple requests come in for the same model, Ollama tries to make batchingIt groups requests and processes them together, making better use of GPU array operations. From the outside, users see responses begin to be transmitted simultaneously. If more requests arrive than OLLAMA_NUM_PARALLEL allows, they enter a FIFO queue regulated by MAX_QUEUE OVEN, which by default is 512.
If the queue fills up, Ollama returns 503 errors (“Server overload”). And if memory is at its limit, another limit comes into play. OVEN_MAX_LOADED_MODELSThis indicates how many models can be loaded simultaneously. When a new model needs to be loaded and there is not enough memory, inactive models are unloaded and the request waits until the new model is ready.
In real-world deployments, a common approach is to start with OLLAMA_NUM_PARALLEL=1 or 2 To prioritize stability, monitor CPU usage, VRAM and p95 latency, and gradually increase settings as long as no memory shortage errors or queue spikes appear.
Memory management strategies in Ollama
Memory (RAM and VRAM) is the critical resource in any local LLM service. Ollama combines several strategies for avoid crashing the system when more requests arrive than can fit in memory or when you intend to use models that are too large for your machine.
On one hand, it uses a FIFO queue It is controlled by OLLAMA_MAX_QUEUE to avoid rejecting all requests at once when no immediate memory is available. If the queue becomes saturated, it explicitly returns 503 instead of simply letting the process terminate.
On the other hand, it maintains a limited number of models loaded into memory using OLLAMA_MAX_LOADED_MODELS (by default, 3 per GPU or 3 per CPU). Models that have been inactive for a certain period of time can be automatically unloaded, freeing up VRAM and RAM for subsequent requests.
The parameter also comes into play OLLAMA_KEEP_ALIVEThis defines how long a model remains in memory after the last request. Values like 5 minutes prevent the model from being reloaded on each request, but don't keep it in RAM indefinitely. With 0, the model is downloaded immediately upon completion, saving memory but increasing startup time; with -1, it remains indefinitely as long as the server is active.
In extreme memory pressure situations, the operating system may resort to swap on diskThis significantly degrades performance, and can even cause "out of memory" errors and instance crashes. Therefore, it's vital to adjust: model size, quantization, context length, number of parallel requests, and the number of concurrent models loaded.
CPU vs GPU in production environments with Ollama
Not everyone has access to powerful GPUs in production, especially if deployed in budget servers or laptops. Hence the growing interest in Optimize inference using only CPU, choosing lightweight and well-quantized models that allow for reasonable latencies.
For pure CPU use, best practices involve choosing models from 2B, 3B or 7B For quantized contexts (Q4_K_M, Q5, etc.), use moderate context sizes and limit the number of parallel requests. Adjust OLLAMA_NUM_THREADS The number of physical cores (or slightly less) helps to balance performance and resource usage, preventing system overload.
If you have a GPU, it is essential to check, using ollama psthat the model is actually using the graphics card (ideally “100% GPU” in the PROCESSOR field). Configurations that mix too many CPU and GPU layers tend to produce poor yieldsIn environments with sufficient VRAM, it is desirable to offload as many layers as possible to the GPU.
To explicitly activate acceleration, in some environments you need to export variables such as CUDA_OVEN=1 or properly configure the NVIDIA/AMD drivers. If you see errors like “CUDA error” or “ROCm error” in the logs, there is probably a driver or hardware compatibility issue.
Advanced configuration: environment variables and stable deployment
Beyond the inference parameters, Ollama can be fine-tuned with a good handful of Environment Variables that define the network, routes, CORS, logging, and model loading behavior. In production, it's common to modify at least the following:
On one hand, OLLAMA_HOST Defines which interface and port the API listens on. By default, it's 127.0.0.1:11434, meaning it's only accessible locally. If you want to expose it to the internal network, you can change it to 0.0.0.0:11434 or a specific IP address, always protected with a firewall or reverse proxy.
Another very useful setting is OVEN_MODELSThis allows you to move the folder where the models are stored to another disk or volume (for example, a large SSD). This gives you flexibility to manage space and backups, provided the user running the service has read and write permissions to that path.
To integrate web frontends such as Open WebUI or other GUIs, you need to adjust OLLAMA_ORIGINSThis controls the allowed origins in CORS. You can specify specific domains (http://localhost:3000, etc.) or use "*" to allow all domains, which only makes sense if the service is not exposed beyond a tightly controlled network.
During deployment and troubleshooting, enabling OLLAMA_DEBUG=1 To view detailed logs: GPU detection, model blob loading, response times, specific errors, etc. On Linux, these logs can be easily accessed with journalctl -u ollama, being able to redirect them to files or filter them by date.
The specific way to set these variables depends on the environment: in Linux it is typically done with a systemd override for the service ollama.serviceon macOS via launchctlIn Windows, this is done using system environment variables, and in Docker, it's done through the option -e en dockerrun.
Model management, Modelfiles and workflows with multiple LLMs
After the system part, it's time to think about how organize the modelsOllama offers its own catalog that is operated with commands such as ollama pull, ollama list, ollama rm y ollama pushThis last feature allows you to upload custom templates to your registry, facilitating distribution and versioning within teams or companies.
To customize the behavior of a model (tone, prompt template, default parameters), the system is used to ModelfilesThese files define, for example, the instruction FROM with the route to .gguf, the lines PARAMETER (temperature, num_ctx, num_predict, top_p, etc.) and a template which specifies how the prompt is structured (system messages, user messages, wizard messages, delimiters).
With ollama create You can compile a Modelfile and register a new logical model in the system without physically duplicating the disk space. This is very useful for generating multiple variations of the same base model (for example, one for general use, another optimized for code, another for formal style, etc.).
In architectures with agents or multi-stage flows (RAG + guardrail + document evaluation + query expansion + hallucination verification), it is common combine several modelsOne general-purpose model, one small and fast model for classification/guardrails, and perhaps one specialized for terrain. Here again, it's crucial to properly adjust OLLAMA_MAX_LOADED_MODELS and KEEP_ALIVE to avoid constantly loading and unloading models.
Finally, Ollama's REST API exposes endpoints for chat, generation, embeddings and model managementThis allows for easy integration with agent orchestrators like LangGraph, as well as backend applications in Python, JavaScript, PHP, etc., adding jitter retries and keep-alive connections on the client side to avoid occasional queue spikes.
With all this, it's possible to go from a simple local AI "toy" to a robust platform of LLMs in productionWith fine control over performance, memory, and behavior, keeping data under your own infrastructure and without completely depending on external cloud providers, something especially valuable in scenarios with privacy requirements, tight costs, or the need for deep customization.