La local artificial intelligence It's no longer something exclusive to large companies with enormous data centers. Today you can run advanced AI models directly on your PC, laptop, or even embedded devices, leveraging your CPU, GPU, or NPU, without constantly relying on the cloud, and ensuring you meet the requirements. minimum and recommended requirements for optimal performance.
In this article we will calmly examine how ONNX Runtime lets you run AI models on your own hardwareWhether integrating it with .NET and VB.NET, using it from Python, C++, or C#, or leveraging native integration with Windows ML, you'll see what the ONNX format is, how models are exported and optimized, the role of Execution Providers, and why all of this is key to having fast, private, and inexpensive AI in your daily work. We'll also review how... Take advantage of local AI with Ryzen on compatible devices.
What is ONNX and what problem does it solve?
ONNX, acronym for Open Neural Network ExchangeIt is an open standard for representing machine learning and deep learning models. Its goal is to allow you to train a model in a framework such as PyTorch, TensorFlow, Keras, TFLite or scikit-learn, and then run it in a different environment without going crazy with specific conversions.
This format defines a graph-based serialization which describes the model's architecture (layers, connections, operations, data types, etc.) and a set of standard operators (convolutions, activations, normalizations, etc.). Because it is based on graphs and not on closed binary formats like Pickle or Joblib, facilitates interoperability between languages ​​and frameworks, something especially useful in computer vision and low-latency applications.
For AI teams, ONNX has become a key component because It reduces dependence on a single supplier.You train wherever you want, convert to ONNX, and then choose the inference engine that best suits you (ONNX Runtime, TensorRT, OpenVINO, etc.) on the hardware you have available, whether it's a cloud server, a desktop PC, an FPGA, or an edge device.
The standard is supported by major industry players such as Microsoft, NVIDIA, Intel and other manufacturersThis has fostered a very broad ecosystem of tools: official converters, libraries for conversion and optimization, model visualization with Netron, and the famous ONNX Model Zoo with pre-trained models to get started without having to train from scratch.
How does ONNX Runtime fit into Windows ML and the Windows ecosystem?
On Windows, ONNX Runtime is tightly integrated through Windows Machine Learning (Windows ML)This layer acts as middleware that groups and manages your PC's hardware resources (CPU, GPU, and NPU) to run ONNX models with the best possible performance without you having to manage the low-level details of each chip.
Windows ML provides a shared copy of the ONNX runtime At the system level, this allows applications to use this common environment instead of packaging their own version of ONNX Runtime within the installer. This reduces app size, simplifies updates, and ensures that the most appropriate version for the system is used.
Furthermore, the operating system is capable of dynamically download specific execution providers for the available hardware. That is, it detects if your computer has an NVIDIA GPU, a Qualcomm NPU, or an integrated Intel or AMD GPU and downloads the corresponding EP, keeping it updated without the developer having to worry about it.
For C#, C++, or Python programmers, Windows ML offers an API that reuses the concepts and patterns of ONNX Runtime, so that You can migrate existing ONNX Runtime-based workloads to the Windows ML environment with few changesAnd if you're working with the Windows App SDK (version 1.8.1 or later on Windows 11 24H2 or later), this integration is already ready to use; this is especially useful on modern devices such as Windows laptops with Snapdragon X2 Elite.
ONNX models: structure, types and practical examples
An ONNX format model is, essentially, a mathematical representation of the learned system after training. This representation includes everything necessary to perform inference in another environment, without having to preserve the original code of the framework where it was trained.
Within an ONNX file you will find the model architecture (the layers, the computation nodes, the connections between them and the type of operations), the trained weights (the parameters that the model has adjusted during training) and the formal definition of inputs and outputs, with their dimensions, data types, and names.
For example, an image classification model trained in PyTorch can be exported to ONNX to accept a tensor with a normalized image as input and return a vector with the probabilities of belonging to each class as output. That same file can then be run with ONNX Runtime on a Windows server with a GPU or on an IoT device without changing the model; furthermore, if you work with advanced vision, you can review guides for detect objects in images with AI.
The ONNX ecosystem includes the ONNX Model ZooA repository of ready-to-use, pre-trained models: image classification networks (ResNet, MobileNet, and similar, with sizes on the order of 5–10 MB), language models like GPT-2 (~500 MB), or very heavyweight architectures for object detection and segmentation (1–2 GB or more). This allows you to test high-quality models without having to train them from scratch.
Advantages of running AI on your own hardware
One of the reasons why ONNX and ONNX Runtime have gained so much traction is because They facilitate the execution of AI locally.That is, on the user's own device or on a company's on-premise infrastructure, reducing dependence on cloud services.
The first advantage is the data privacyBy not sending information to third-party servers, the risk of leaks or misuse is minimized. This is critical in sectors that handle sensitive data (healthcare, finance, public administration, or corporate legal departments) and do not want their data to travel outside their internal network.
It also significantly reduces recurring costThis is because you don't need to be constantly paying for cloud computing instances to run inference models. You invest in your own hardware (or leverage what you already have) and avoid surprises on your bill at the end of the month when API usage skyrockets.
Another very clear advantage is the low latencyBy running the model on the device itself, you eliminate network round-trip time, which is especially noticeable in real-time applications (industrial machine vision, interactive assistants, live video processing, or video games with integrated AI).
Finally, running models locally gives you full control over deploymentYou can customize the pipeline, apply quantization or pruning, restrict access to certain resources, decide when to update the model, and adapt the inference to your company's internal regulations or applicable regulations.
Basic requirements to get started with ONNX Runtime in .NET and other environments
If you want to start using ONNX Runtime with .NET and VB.NET On your computer, the initial requirements are quite reasonable. You'll need a supported operating system (Windows, macOS, or a popular Linux distribution) and either .NET Framework or .NET Core installed, depending on your project.
In the specific case of Visual Studio, simply open your solution and Add the Microsoft.ML.OnnxRuntime NuGet packageThis package includes the libraries needed to load an ONNX model, create inference sessions, and invoke the engine with the appropriate input data.
In addition to runtime, you will need at least a model in ONNX formatYou can train it yourself with PyTorch or TensorFlow and export the results, or download one from the ONNX Model Zoo for testing. From there, you'll focus on preprocessing the input data and interpreting the model's outputs; if you need help choosing hardware, see our guide to choosing well.
On machines with more modest hardware, ONNX Runtime is designed to be efficient even without a powerful GPUHowever, if you're going to handle large, complex language, generative, or vision models, having a modern GPU or, even better, a dedicated NPU will make a difference in response times.
How to use ONNX Runtime step by step: sessions, inference, and compilation
The typical workflow with ONNX Runtime consists of load a model, create an inference session, and run predictionsAlthough the specific code varies depending on the language, the philosophy is very similar in all of them.
First, you specify the path to the .onnx file and You create an inference session with the desired options (for example, configuring which execution providers you want to allow or adjusting optimization parameters). In .NET, this is done through the classes exposed by Microsoft.ML.OnnxRuntime; in Python, you'll use the onnxruntime API directly.
Next, you construct the set of inputs for the modelTypically, one or more tensors with the form and data type that the model expects (preprocessed images, tokenized text, numeric vectors, etc.). These inputs are packaged into a dictionary or map structure that associates the input name with its value.
Once the input is prepared, you call the method of session executionwhich returns the model's outputs also in the form of tensors. All that remains is to post-process those results to transform them into something understandable (labels, scores, generated text, bounding boxes, etc.), and use them in your application.
Since version 1.22 of ONNX Runtime, there is a more advanced approach: compile models into an optimized representation before releasing them to production. New APIs, such as those associated with the OrtCompileApi framework, better encapsulate the compilation steps so that the runtime generates artifacts specific to the target hardware, further improving performance.
Windows ML, hardware execution and automatic management providers
In the Windows ecosystem, Windows ML serves as a intelligent orchestration layer about ONNX Runtime. It not only reuses its inference engine, but also automatically manages which execution provider is best suited for each machine and model.
Un Execution Provider (EP) It is, basically, a component that knows how to optimize certain AI operations for a specific backend (CPU, GPU, NPU or specialized accelerators), registering the necessary kernels and participating in the partitioning of the operation graph so that each part runs on the most efficient device.
Windows ML offers several notable advantages: Automatic download and management of EPs hardware-specific features, use of a shared system-level ONNX runtime (which drastically reduces application size), and compatibility with a wide variety of configurations (x64 desktop PCs, ARM64 systems, Windows servers, etc.).
The automated deployment flow typically follows these steps: during the installation of a Windows App SDK-based app, Windows ML is initialized; the runtime detects the available hardware; the optimal EPs are downloaded (for example, TensorRT for an RTX GPU, a specific EP for the Snapdragon X NPU, or the OpenVINO stack for Intel); and from there The application can start running AI models immediately..
Thanks to this architecture, developers don't have to package specific drivers or libraries for each manufacturer, nor maintain separate versions of the application for each hardware. Windows ML takes care of that. abstract the backend details and ensure that inference is performed with the best available performance.
Model conversion to ONNX and typical development workflow
To benefit from ONNX Runtime, the first step is convert your models to ONNX formatThe most common workflow in computer vision and general ML projects combines these phases: training, export, optimization, and deployment.
You typically train the model in your preferred framework (for example, an image classification network in PyTorch or an object detection network in TensorFlow) and, once you have achieved the desired performance, export it to ONNXIn PyTorch the torch.onnx.export function is used, while in TensorFlow the tf2onnx tool or others integrated into high-level libraries are usually used.
In that export it is advisable to set a compatible opset version (for example, version 15, which is widely supported) to avoid problems with older inference engines. After generating the ONNX file, you can inspect it with tools like Netron to verify that the graph looks as expected and that there are no exotic, non-standard operators.
Before deployment to production, many teams apply a phase of model optimization: quantization to reduce size and improve speed, weight and filter pruning, parameter grouping, or even knowledge distillation, passing information from a large model to a smaller one that is more manageable on devices with limited resources.
Once the model is in ONNX and optimized, it is integrated with ONNX Runtime or with platforms such as Windows ML, TensorRT or OpenVINO, and deployed in the target environment: cloud servers, industrial PCs, edge devices or mobile and desktop applications.
Hardware acceleration and inference performance with ONNX Runtime
El inference performance This is one of the main reasons to choose ONNX Runtime. In many practical comparisons, significant improvements have been observed when running models converted from PyTorch or TensorFlow. response times and resource consumption.
In use cases with a batch size of 1 (very typical in interactive interfaces or real-time vision), ONNX Runtime usually offers initial inference speeds 20% higher compared to PyTorch, and in prolonged executions accelerations of up to 5 times have been reported, with notable reductions in CPU usage without increasing latency.
These improvements are due to optimizations such as the Kernel fusion, graph simplification, and quantizationThis allows for fewer effective operations to be performed to achieve the same result. At the same time, the ability to segment the graph and assign specific parts to GPUs or NPUs maximizes hardware utilization.
ONNX Runtime supports a wide variety of accelerators: NVIDIA CUDA and TensorRT for dedicated GPUs, Intel OpenVINO and oneDNN for CPUs and VPUs, AMD ROCm, Qualcomm QNN, Apple CoreML and Android NNAPI for mobile devices, and DirectML in Windows environments. This broad compatibility with EPs This allows the same ONNX model to run efficiently on many different platforms.
All of this makes ONNX Runtime an ideal choice for applications of machine vision, embedded systems and edge devices that require real-time responses and very low consumption of both RAM and CPU.
In this context, ONNX and ONNX Runtime have established themselves as a solid bridge between training in popular frameworks and optimized execution on any type of hardware, offering the right mix of performance, flexibility, and privacy to bring artificial intelligence models to your own hardware without unnecessary complications.
