In very simple terms, an agent is software that can perceive its environment in some way, plan accordingly, take some sort of action, and perhaps even learn. One vendor’s “agent” is a Python process that reads JSON. Another is a hosted proprietary service. A third is a LangChain agent running tools locally. Each expects different inputs, different formats, different context handling, and different connection methods. Getting them to talk to each other requires duct tape, wrappers, and a bunch of custom glue code.

That’s the problem the Agent2Agent (A2A) protocol was created to solve. Proposed by Google in the spring of 2025, A2A provides a standard, open way for agents to discover each other, communicate, collaborate, exchange context, and complete tasks together, regardless of framework, vendor, or environment. It’s vendor-neutral, built on open web standards, and designed for real-world enterprise use.

And for network engineering, where we’re starting to see new automation workflows depend on AI-driven troubleshooting, multi-step automations, and cross-domain reasoning, A2A helps get us to an entirely new level of automation.

Why A2A Was Created

Before A2A, multi-agent systems had several problems. 

First, there was no standard communication model. Each agent exposed its capabilities in its own way, which meant developers had to write point-to-point integrations. And as the actual A2A spec notes, this made systems brittle, slower to build, and difficult to scale.

Second, agents needed to be wrapped as “tools.” Because there was no unified protocol, developers often wrapped agents inside other agent frameworks which reduced their autonomy and limited their functionality. A2A removes this by letting agents communicate natively as peers.

Third, there was a lack of discoverability. A “consuming” agent had no standardized way to learn what another agent could do. A2A introduces a concept called the Agent Card, which is a machine-readable metadata document describing skills, capabilities, authentication method, and endpoint URL.

Fourth, there was no consistent task handling. Some agents returned raw text, while others returned JSON. Some provided task IDs for long-running jobs, but others didn’t. A2A provides a standardized structure by introducing a model for:

• Messages
• Tasks
• Artifacts
• Status updates
• Streaming outputs
• Asynchronous completions (push notifications)

Last, security was ad hoc. Because agents were stitched together manually, there was no consistent approach to authentication, authorization, or TLS. A2A enables enterprise-grade security patterns using standard web practices (TLS, OAuth2, bearer tokens) and moves identity to the HTTP layer.

All of this makes A2A a protocol designed not just for tech enthusiasts or PoCs (or should I say neverending PoCs ), but for real enterprise environments.

How A2A Works

A2A consists of three main components:

1. The Agent Card
2. Messages and Tasks
3. Transport and Connectivity  

The Agent Card

Every A2A-enabled agent publishes a JSON description of itself used for capability discovery and usually includes:

• Identity
• Endpoint
• Supported input/output types
• Skills (the tasks it knows how to perform)
• Authentication requirements
• Supported extensions
• Streaming capability
• Push-notification capability

Below is a snippet of code for the A2A agent card. Notice that for capabilities, this agent card defines traceroute and ping. 

The Agent Card is used for discovery, either via a well-known URI, a central registry, or direct configuration (which seems to be common in the enterprises I’ve worked with experimenting and running agent systems). The result is that agents can find each other and learn each other’s capabilities automatically. There are several components:

• Client agent: formulates and communicates tasks (actions, requests for info, etc)
• Remote agent: acts on the tasks from Clients
• Capability discovery: Agents advertise capabilities using an “Agent Card” in JSON format, allowing the client agent to identify the best agent that can perform a task
• Task management: Communication between a client and remote agent is oriented towards task completion, in which agents work to fulfill end-user requests within a lifecycle. The output of a task is known as an “artifact.”

Capabilities are described in the agent card which means descriptions are essential for advertising capabilities correctly. Notice in the image below that we include a simple description along with an input and output schema. The Description can be more explicit to make sure it’s clear what the agent is capable of. 

The actual flow is between the client and the remote agent. If you squint hard enough, A2A can feel DNS-like in that both help distributed systems discover and interact with each other without hardcoding relationships. The similarity stops pretty quickly, but the analogy works from a very high level. Below is a graphic of the A2A communication flow.

Messages and Tasks

A2A defines two fundamental communication objects, messages and tasks. 

Messages are used for short, stateless interactions, and tasks are used for long-running or complex operations that need state management, multi-step processing, intermediate updates, and human-in-the-loop requirements. Think of it as a structured payload that’s sent from one agent to another containing data, instructions, context, or results. 

A message typically contains a sender ID, recipient ID, timestamp, content (structured JSON), and a correlation ID. In networking terms, a message is like an RPC call message that’s atomic and stateless on its own. 

A task has a well-defined lifecycle (submitted, working, input-required, completed/failed) and the artifacts (completed task outputs) they produce can’t be modified, which ensures traceability. Think of a task as “do this thing”, which in NetOps might be “analyze why site A can’t reach application B”, or “validate prechecks for BGP peer turn-up.”

A task has intent, may span multiple messages, and has a lifecycle state. And in terms of network operations, this model is extremely important because many workflows we use in typical NetOps are long-running. Think of complex root cause analysis and remediation steps that call on multiple network data sources, multiple tools like a RAG function, SQL query generator, ML model, traceroute function, and so on, and require very precise tracking.

Transport and Connectivity

A2A typically operates over HTTPS and supports RPC-style interactions such as:

• HTTPS for transport
• JSON-RPC or gRPC for operations
• SSE for streaming
• Webhooks for asynchronous notifications

That makes A2A compatible with typical enterprise networks behind a firewall, existing API gateways, service meshes, and even iIdentity providers. In other words, A2A basically fits the enterprise environment rather than shoehorning it into the enterprise network ecosystem. 

Failure Modes

There are several failure scenarios to consider as well. An agent might be completely unavailable, whether planned or unplanned. Sometimes tasks only partially complete and then stall. Sometimes the agent works fine, but the specific MCP server or downstream tool needed for the task is unavailable. 

In an A2A environment, failure is treated as an expected condition rather than an exception. This is different than a simple request–response systems where a failed step often causes the entire workflow to collapse. Instead, A2A is built around the concept of tasks that have lifecycle states. That means a task isn’t a single interaction but instead is a unit of work that persists across multiple interactions and agents. This persistence is what allows the system to handle unavailable agents or incomplete steps without losing track of progress.

When an agent becomes unavailable, the task itself doesn’t disappear. Because the task maintains state, such as pending, running, waiting, completed, or failed, the system can recognize that progress has stalled and figure out why. 

For example, if an RCA task retrieves telemetry and configuration data successfully but can’t run an active network test because the testing agent is offline, the task can move into a waiting state rather than failing outright. This preserves the work already completed and keeps the workflow recoverable.

A2A systems also assume that partial completion is normal in distributed environments. Instead of requiring every step to succeed in sequence, they allow intermediate results to accumulate while waiting for dependencies to resolve. This is especially important in network operations, where dependencies such as telemetry sources, testing systems, or inventory services may not always be available at the same time.

And very importantly, A2A systems maintain an audit trail of what was attempted, what succeeded, and what failed. This is extremely valuable for post-incident analysis and operational governance.

Observability for AI Agents

Observability for AI agents, especially in NetOps, is about making their behavior as transparent and diagnosable as possible. There are several mechanisms in place for this. 

Task tracing means you can follow an agent’s work from start to finish. When a task was created, which agents participated, what steps were taken, and how long each stage took. This gives engineers visibility into workflow execution, similar to tracing a service request through a distributed system.

Message lineage tracks how information moved between agents. It shows which agent requested what data, which responses were received, and how context evolved over time. In network terms, it’s similar to understanding packet flow but applied to decision-making inputs.

Decision explainability ensures that when an agent proposes a diagnosis or action, you can see why. What data did it rely on? What changes influenced its conclusion? Which assumptions were made? This is very important for trust, governance, and of course for safe automation.

Lastly, task ownership is important for stall detection and escalation. 

Using A2A in Network Operations

So how does A2A help network engineers?

NetOps in a more complex environment would involve multiple specialized AI agents, each performing a piece of a much larger operational activity. In the diagram below, notice that an orchestrator agent communicates via A2A to multiple other agents each specializing in their own capabilities. This can also be designed as a system of agent peers rather than with an  orchestrator or “super-agent.” 

There’s room to argue over what specific capabilities an RCA agent should have over a telemetry agent, and so on, but the point is that we can create a resilient, functional, distributed AI system this way. 

In my example, the orchestrator agent has access to multiple LLMs which act as the “brain” of the system interpreting user prompts, agent card descriptions, and results. In reality, each additional remote agent would also have access to at least one but likely several LLMs as well. 

Of course this comes down to a design tradeoff, because though it may be advisable to use the same LLMs for all agents to maintain a level of predictability and consistency (and to minimize licensing headaches), model diversity can also improve reasoning robustness.    

I’ve been working in a variety of larger network environments recently, and here are a few examples of what I think would be useful in the real-world. 

In my breakdown below, notice that each agent is focused on a specific task, which in reality could be a collection of different tools and workflows, but nevertheless a single conceptual task. There are many benefits to splitting up tasks among multiple agents like being able to upgrade one workflow while leaving others intact, more granular control over access controls, a modular approach to tracing and visibility, and so on. 

The likelihood is that you’d start with one agent working with multiple tools or multiple MCP servers, and in time deploy additional functionality in the form of more specialized agents. 

Telemetry Agent

First, there’s the telemetry agent. Because of the last few years I’ve spent in observability, this is both top of mind for me but also a great first step in creating your own agent system. 

A network telemetry agent would be focused on and have access to telemetry data like device metrics, flow logs, various metadata, your network SoT, IPAM, etc, and the query tools to access that data.  

A pure telemetry agent is just that – a way to query telemetry data and not necessarily do much else. This is a great way to keep functionality constrained so that the agent performs only read-only, specialized queries that a NetOps team can build over time to include more data types and sources. 

Documentation Agent

A documentation agent would be focused on the workflows involved with searching unstructured text data such as vendor docs, ticketing systems, TAC phone call transcripts, and so on. This would probably be a hybrid RAG system (or systems) which the agent utilizes to answer questions from a KB, correlate information embedded in threat feeds or CVE reports, or perhaps identify trends among all the tickets that came in this year.

Configuration Agent

A configuration agent would specialize in actual device configuration. This kind of agent would need access to syntax knowledge, change management systems, deployment scripts, and programmatic access to network devices. 

A config agent starts to get more complex because we need to consider how to add a human-in-the-loop to approve config pushes, rollback procedures, how to identify and fix config drift, strict access controls, logging, and so on. 

Topology Agent

A topology agent is an interesting one, and I’ve seen this only once so far. Basically, it’s an agent that understands network structure at the dataplane level, possibly built on a network model that looks at LLDP/CDP relationships, path tracing, metadata, and more. 

A topology agent can be a real-time (or near real-time) foundation for a digital twin that other agents can interact with to understand network data that isn’t traffic telemetry. That could be path dependencies for an application, understanding routing adjacencies such as what backup routes exist if the primary becomes unavailable, etc.  The topology agent isn’t itself the digital twin, but it would support twin modeling. 

Root Cause Analysis Agent

Once we have workflows (or more likely multiple agents) in place to gather and query telemetry, search network and vendor documentation, and perform real-time tests like path traces or ad-hoc synthetic tests, a RCA agent can compile and consider many factors to then come to a conclusion as to the root cause of an incident. 

A RCA agent is heavily dependent on multiple underlying workflows, some of which likely very complex on their own, and in reality would be an orchestrator agent handling quite a bit of decision-making grounded in the ability of an LLM to understand the various data, the query itself, and the nature of the incident.     

Ticketing Agent

A ticketing agent is a dedicated agent specializing in interacting with human workflow systems. This is something I’m seeing more of lately as folks are reluctant to have one agent handle gathering telemetry, pushing config, updating tickets, and especially touching sources of truth. 

The ticketing agent has tools available to interact with these systems after another agent completes a task requiring an update. This adds a layer of complexity for sure, but it adds more granular control and visibility, both of which are absolutely necessary in a  production network environment. 

There are other specialized agents I’m starting to see people experiment with, but the above give us a good starting point with what’s starting to move from PoC to production in the industry right now. The idea is that with A2A, these independent agents can work together seamlessly, perhaps with a single (fault tolerant) orchestrator agent. 

Real-World Example

The real-world examples are pretty much limitless. It really depends on how complex a system you want to build, what data is available, what your needs are, and what budgets and staff resources you have to actually build a multi-agent system with A2A.

Here’s one example.  

Diagnosing High Interface Utilization

Imagine an alert fires for high utilization on a core router interface. In an A2A-enabled environment, the workflow might look like this:

Step 1: The RCA Agent receives the alert

It determines it needs additional context and sends a message to the Telemetry Agent: in natural language, “Provide 6 hours of bitrate data for device XYZ interface Gi0/0.”

Because A2A standardizes messages, the RCA Agent doesn’t need to know the various tools the Telemetry Agent uses, and it doesn’t need to know its internal implementation.

Step 2: The Telemetry Agent returns a task

The Telemetry Agent interprets the request from the RCA Agent and access various tools via MCP such as database queries or metric lookups. It completes the task and returns flow breakdowns, a report of packet drops, metrics around latency spikes, or perhaps time series chart data. 

Step 3: The RCA Agent interfaces with the Ticketing Agent

The RCA Agent then interfaces with the Ticketing Agent. The Ticketing Agent generates a trouble ticket in ServiceNow, identifies customer dependencies, sends a Slack message in the appropriate channel, perhaps updates a NOC dashboard, and any other necessary documentation or alerting activities.  

Step 4: RCA Agent requests topology context

Next, the RCA Agent queries the Topology Agent using A2A to understand adjacent devices via LLDP/CDP, path dependencies, and routing adjacency changes. This helps determine the scope and severity of the incident including important information for potential remediations.   

Step 5: RCA Agent aggregates findings

Then, the RCA Agent uses the A2A model for Artifacts to store outputs which in this case are the flow reports, time series graphs, log data, topology information, and so on. 

Step 6: Configuration Agent produces remediation

The RCA Agent passes its findings to a Configuration Agent, which generates the actual traffic engineering change which could be BGP policy adjustments, QoS modifications, or maybe even a recommendation to do nothing.

Step 7: Change Control Agent handles workflow

Before pushing any changes, the Configuration Agent uses A2A to talk to the Ticketing Agent to update the ticket, make changes to severity, and explicitly list the recommended remediation and get human approval before proceeding. 

All of this is possible because A2A provides task tracking, agent discoverability, multi-step interaction, and secure, structured communication. Without A2A, this workflow requires custom glue code between every component.

A2A and MCP Together

The A2A/MCP relationship is very important to understand. 

A2A is all about agent-to-agent communication, like its name suggests. MCP, on the other hand, is about agent-to-tool communication. MCP is how agents call tools like NetFlow database queries, CLI command runners, SNMP polling functions, configuration generators, cloud APIs, and so on. A2A is how agents delegate tasks to other agents, which in practice means A2A is horizontal because agents collaborate across domains and MCP is usually more vertical in that agents access their internal tools. 

Why A2A Matters for the Future of NetOps

Network operations is becoming more distributed, more automated, more cross-domain, and in recent days more AI-driven. We don’t really see that much of the “one monolithic network automation platform,” especially when you consider the sum total of network-adjacent platforms relevant to modern NetOps, namely, network sources of truth, observability platforms, digital twins, and so on. 

Instead, we’re entering a world where small, specialized agents each perform a task well, agents negotiate, plan, and collaborate, AI becomes the orchestrator, and humans step in only when needed. A2A is the protocol layer that makes this ecosystem possible because it provides a shared language, a predictable communication model, strong task semantics, and the ability to scale automation without brittle integrations. And the idea is that this translates to faster troubleshooting, safer changes, fewer outages, and more intelligent automation.

A2A is the connective tissue that lets agentic systems work together in a way that’s both structured and flexible. For network engineering and operations, it creates the foundation for next-generation automation that’s distributed, collaborative, AI-driven, and fundamentally interoperable.

Thanks,

Phil

Recently I’ve been having conversations with IT (mostly networking) teams about incorporating AI solutions, and the first questions I have are:

1) What problem are you actually trying to solve with AI (that AI would solve better than a simpler/cheaper solution)?

2) What’s the nature of the available data? This includes follow-up questions like: is network and network-related data stored in disparate locations? Has it been cleaned and pre-processed for analysis? How and how often is the data updated? Do you have different formats of data? How is data ingested and stored?

3) What’s the makeup of your technical staff? Does your network team contain or have access to developers, data engineers, perhaps even data scientists?

The answer to the first question is critical. For some, AI is simply an initiative handed down to operations teams by leadership with no clear direction or no clear problem to solve. Also, sometimes the direction and problem are clear, but it can be solved more easily and for far less cost than a sophisticated AI solution.

The answers to the second group of questions are just as important. The data we use in network operations is both structured (metrics, flows, etc), and unstructured (tickets, email, TAC call transcripts, certain types of logs, etc). The workflow to deal with this in a unified manner is more complex than many realize and requires serious planning, software development, data engineering, and perhaps data scientists.

Much of the data we deal with in NetOps is real-time telemetry requiring significant effort to clean, process, store, and analyze quickly enough to be useful for a production environment. This is no small feat, and addressing these data issues can end up being 80% or more of an AI implementation project.

The answers to third question will shape how we approach the build vs. buy discussion and how feasible an AI initiative is in the first place. There’s real ROI to consider, whether that’s ROI from decrease in current network operations cost, or a perhaps a direct increase to revenue from some new product or feature. Getting there will dedicated require staff – expensive staff – and a significant amount of time and effort. PoCs will require careful planning, timelines, and measureable results.

Kavita explains that successful AI solutions are by nature iterative in that they require constant tweaking. I would argue that any good software solution is iterative to one extent or another, but certainly I agree that an AI solution is much more so. Her perspective is practical and based in real-world scenarios, something I appreciate very much.

I’m excited about the near and long term potential of incorporating AI solutions into IT operations to solve real problems and to provide real value, and I’m spending a significant time and effort researching and experimenting with applying AI to what we do in NetOps. However, I’ve found it’s essential to temper curiosity and experimentation with an understanding of the use-case, cost, and tradeoffs of implementing AI in our beloved niche of tech.

You can check out and purchase Kavita’s book here.

Thanks,

Phil

I believe there are two main use-cases right now, with a third right around the corner. First, LLMs can query large and diverse datasets of network telemetry faster than any engineer, which in turn reduces resolution time and increases network performance and reliability. Second, as part of a larger workflow, LLMs can help us perform advanced analysis on network telemetry thereby providing greater insight and understanding of what is otherwise a complex web of varied telemetry data. And in the near future, agentic AI, which is really an extension of my second use case, will help us automate tasks and more autonomously manage network operations.

In this post we’ll discuss the first two use-cases and touch on the third. We’ll also unpack some of the limitations we face implementing LLMs in NetOps, and we’ll discuss several quick and easy ways to get started right now.

Probabilistic Models

First and foremost, keep in mind that LLMs are based on probabilistic models. Probabilistic models are a type of mathematical model used to predict the next item in a sequence, such as the next word in a sentence. This is based on the idea that the next word in a sequence can be approximated (predicted) by all the preceding words.

However, though LLMs are technically probabilistic, they use the Transformer model which is much more sophisticated than a strict probabilistic model like an n-gram. Transformers, which do use probabilistic techniques, rely on a sophisticated architecture of encoding and decoding as well as an attention mechanism to understand context.

Nevertheless, LLMs really have no “intelligence” as we understand it (some experts disagree), which means that though their outputs can be impressive, there’s room for error. Therefore, when we use an LLM in our NetOps workflow, we need to employ methods to make responses relevant to our specific domain as well as reduce errors.

Network Operations Use Cases

The first use-case I mentioned in the introduction is both practical and an easier way to get started compared to advanced analysis and building autonomous systems.

The first practical use for LLMs in NetOps is to make the process of interrogating very large, diverse, and divergent network datasets much, much easier and faster.

In modern network operations, we’re collecting logs, metrics, flows, and quite a bit of metadata about each of these devices individually and the traffic traversing them. Querying this data easily, quickly, in real-time, and at scale, is a huge undertaking.

Now add to this data all of the configuration files, open tickets, closed tickets, internal KB articles, network diagrams, and all the vendor documents for the devices in your racks and closets. That’s a huge amount of additional data, both in volume and type. And though it’s a different type of data (mostly text) from device and traffic information, it’s nevertheless crucial for NetOps.

This is a big problem. Unfortunately, and in spite of advances with network automation, mining through all of that data is still mostly a manual process based on our intuition and the domain-specific knowledge tucked away in our brains. I mean, as engineers we still buckle down and do it, but it takes hours, days, or weeks of clue-chaining, pouring over logs, and countless Zoom calls. LLMs can help. 

One example is prompting an LLM in natural language to run a query for you. For instance, you could prompt the system to show you all network traffic egressing US-EAST-1 in the last 24 hours heading to embargoed countries. The output could be in human readable text, or if you structure your workflow properly, it could respond with a visualization.

Another example is a level 1 NOC engineer working on a ticket for a slow network. The engineer can ask the LLM in natural language if there are any devices with network problems. This is a broad question, and it assumes the LLM even knows what “network problems” means. Properly designed, an LLM in a NetOps agentic AI workflow can generate a response that the interface on an internal switch connecting to the SD-WAN edge device is dropping packets, but not quite enough to trigger an alert.

In my hypothetical scenario, I envision the network engineer as a level 1 NOC engineer who may not know the commands to access these devices, or they may not have enough experience to even think of checking interface stats in the first place. Because the LLM allows us to use natural language to query a large dataset, we decrease the process of intuitive and manual clue chaining and reduce the mean time to resolution for this incident. LLMs make accessing, filtering, and querying data dramatically easier and faster, as well as democratize information among anyone who can input a prompt into the LLM.

Using natural language to easily and quickly query huge datasets is great, but what about more advanced queries that involve analysis or inference?

A second practical use case for LLMs in NetOps is advanced data analysis.

Well, this isn’t exactly true. LLMs aren’t designed and optimized for complex statistical analysis on new data (regression, correlation calculation, time-series forecasting, etc). They aren’t meant for real-time telemetry processing, either.

However, if we use an LLM as the interface to interact with a RAG system where data processing is already done, or perhaps a workflow of AI agents designed to do that sort of analysis, then we just made advanced network telemetry data analysis dramatically easier to do – using the LLM as our user interface.

For example, we can prompt the LLM with a question that requires some sort of inference. Because the LLM understands language and therefore the nature of what we’re asking, it can create the relevant python script or SQL query, or perhaps call the appropriate tool (agent) to orchestrate a data analysis workflow.

The LLM doesn’t do the analysis, but it’s an incredible tool to facilitate the analysis workflow and report back the results in a human readable format. This use-case requires careful planning and design in terms of LLM routing, data processing and storage, how to handle real-time telemetry, and so on, but it’s another very practical use-case for augmenting an engineer operating production networks.

The third use-case for LLMs in NetOps is as the orchestrator for an agentic AI system that autonomously manages network operations.

2024 is often called the year of AI agents, and I believe that’s for good reason. There’s so much potential to use an LLM in this way. For NetOps, the LLM becomes our user interface to a complex system of agents that search, apply ML models, decide which tools to use to achieve a goal, and so on. I’ve seen network orchestration platforms come and go over the years, but I wonder if agentic AI will finally get us closer to the dream of a self-operating network.

In any case, let’s now focus on how to get started implementing an LLM for the first two use-cases (I expand upon agents a little more later and will go much deeper in a different blog post).

Implementing LLMs in a Network Operations Workflow

Implementing an LLM in a NetOps workflow is more complex than simply asking ChatGPT to generate a config snippet (although that’s definitely useful). There are many factors to consider such as cost, data management, mitigating hallucinations, and handling real-time telemetry.

Hallucinations, for example, are often the deal-breaker for engineers. From a high level, hallucinations are factually incorrect responses caused by imperfect training data and a lack of real-world grounding. This means we need to employ various methods to ensure the responses we get are relevant and accurate.

Also, LLMs aren’t well-suited for real-time telemetry processing. There’s latency in generating responses, a lack of continuous learning capabilities, and so on. So another consideration is how we ingest, process, and make available real-time data from our network.

Clearly there are some things to consider, but don’t let that stop you from trying. There are some quick and relatively easy ways we can get started right now. I’ll start with the easiest and cheapest method and make my way progressively to the most difficult, expensive, and arguably the most effective.

1. Include the Relevant Data in Your Prompt

First, the LLM needs to know about the data we care about, which is probably in an internal database the model wasn’t trained on. An easy way to do this is to include the data you want to query in your actual prompt. That could be literally copying and pasting a body of raw flow logs, SNMP data, syslog messages, device configuration files, and so on right into your prompt, then asking the LLM to summarize them or find some pattern, etc.

This is effective, but you’re limited by the number of available tokens in the model’s context window, or in other words, how many characters you can input. For many models this will be somewhere around 32,000 tokens, though some models are releasing versions with very large context windows. For example, GPT-4o has a 128k context window and Claude 2.1 has a 200k token context window. According to recent research, however, there’s a clear difference between maximum context length and effective context length, not to mention that even a 200k context window amounts to only around 500 pages of text.

So for most models, this means you can’t include all the data about your network in the prompt. Definitely some tables and smaller databases, but not the sum total of all your logs, telemetry, documentation, tickets, and so on. Strategies like summarization and filtering can help, though you should still be careful with filling up a 200k token context window because of long context issues like “lost in the middle.”

For many cases, though, this is an easy, cheap, and effective way to use an LLM in your NetOps workflow right now. Just remember that this method won’t work for doing anything sophisticated like predictive analysis.

Some practical uses for this method are summarizing syslog messages, identifying related events in a log file, or reviewing code for you that you include in your prompt. Effective, but limited to only the data in the prompt. Now let’s move on to methods to query your network telemetry databases.

2. Use an LLM to Generate a Query over Tabular Data

We can use the LLM to translate natural language to a query over tabular data you likely already have or at least can prepare relatively easily.

For example, you can use an LLM to generate a human readable prompt to a python script that runs a pandas operation, executes against a dataframe, the result of which the LLM can interpret and return in a human-readable format.

The simplified workflow looks like this:

You can also use the LLM to translate an engineer’s prompt to SQL. For example, you can input a natural language query such as “show me all network traffic from the last week where the bandwidth exceeded 1 Gbps.” The LLM translates this into an SQL query and routes the query to the relevant database for you.

This workflow looks pretty much like this:

To do this programmatically (instead of just using the LLM as a coding assistant), we can use a simple web front-end that takes your natural language prompt, augments the input behind the scenes with a template of instructions for the LLM to create a certain type of query, like in SQL as the example above, and return the results in a certain format, such as JSON.

With that returned JSON output, you can get back human-readable text or generate a visualization. The power here is making running queries extremely fast and easy using natural language. Your team can create very complex queries without needing to know how to code, you can limit the access to the data based on role, and you can present the results to the engineer in a way that makes the most sense.

Of course this is a very high level overview and assumes the type of data / database you have lends itself to this method. In production, this can be more complex and should include a caching component that allows you to append your previous prompts. This is a great way to utilize an LLM very quickly in the real-world and with real results without having to build a RAG system with a vector database, which we’ll talk about next.

3. Retrieval-Augmented Generation (RAG) with a Vector Database

Another method is to use a RAG system with a vector database. RAG (retrieval-augmented generation) is a workflow in which we point an LLM to an external database so that the LLM is limited to only the data we care about and insert relevant context data into the prompt to tighten the LLMs ability to respond. This doesn’t always have to be a vector database in that we can retrieve data and enrich a response without a vector database. But in this section I’d like to focus on this very popular and effective way to make using an LLM relevant to your specific knowledge domain or organization and reduce hallucinations.

It’s important to understand that RAG is the mechanism the LLM uses to respond to your prompt, not perform data analysis. Neither does it prepare your external database in any way. So as it turns out, using RAG is almost as much an exercise in data management and processing as it is augmenting an LLM.

Based on what kind and how much data we’re dealing with, we select the appropriate embedding model to transform our data (in our case network telemetry, logs, configurations, etc) into embeddings, and then select the most appropriate vector database (like Pinecone, Milvus, or Chroma) to store those embeddings.

The LLM is then programmed to answer prompts only in the context of this new database, sort of like an open book test. In other words, when we input a prompt, the LLM augments its response by retrieving the data for the answer from only the data we tell it to use, thus generating a more accurate response.

This is an excellent way to use LLMs in many different industries, especially if the external database is mainly text-based. If we’re talking about diverse data types, as is the case with the networking industry, this is more complex than just pointing an LLM to an external database. And there are some limitations to RAG, especially for network operations, including retrieval not always being contextually accurate.

The high level workflow is very straightforward:

The enriched response we see on the right is the LLM responding to the prompt using the data in the RAG database. This means we can limit the LLM to respond to the data we care about like our database of flow logs, metrics, configurations, tickets, and so on. There are various ways to accomplish this, too. For example, the vector database may be searched for relevant documents prior to the prompt going to the LLM, in which case the LLM receives the prompt along with the contextually relevant information to synthesize its response.

A great way to start with RAG in NetOps is to use it for asking natural language questions against a knowledgebase, log collection, or ticketing system. Start with text data in just one database and experiment with how to get the results you want. It gets more complicated when dealing with multiple types of network telemetry, especially real-time data, so consider starting with a more simple chatbot for your internal kb.

For example, a straightforward implementation for an AI chatbot without having to get into dealing with real-time telemetry data could be a chat interface for your ticketing system. You can use natural language to open, update, create, and search for tickets easily and quickly. 

4. Agentic AI

This is a more complex method for using an LLM in NetOps, but one that I believe will change the way we run networks. In an AI workflow of independent AI agents, an LLM would act as an orchestrator that translates prompts and results, routes tasks to agents, and possibly assists agents in tool calling to interact with external systems, enabling them to take actions and retrieve data. In this workflow, the LLM is an important part of an overall system.

In the graphic below you can see that the LLM interacts with other AI agents which could be web searches, other databases, ML models, and so on. In practice, the LLM would likely decide based on a semantic understanding of the prompt which agent it would forward to next.

I believe we’ll see a lot of value to our industry as we develop agentic AI systems with LLM orchestrators. I know we’ve had various network orchestrators and promises of automated remediation for years, but I believe applying agentic AI to NetOps has a very real potential to revolutionize the IT industry.

5. Fine-tuning and Retrieval-Augmented Fine Tuning

Fine-tuning a model is generally expensive, time-consuming, and often requires your own GPUs. Fine-tuning adds additional data to the pre-trained model (like network domain data) so that the model’s parameters can be adjusted and it can respond accurately about what you care about.

Though this is totally doable with smaller models, techniques like LoRA, and rented GPUs like Azure AI Studio, this is much more difficult and expensive to do with foundational models like GPT and with real-time telemetry.

Retrieval-Augmented Fine Tuning, or RAFT was developed very recently (June 2024) to combine the benefits of fine-tuning with RAG to make fine-tuning easier and cheaper and RAG more effective. RAFT is something we can use today to make LLMs more accurate, and therefore more effective in a NetOps workflow. But this method is more expensive and complex to implement than RAG alone or using an LLM to generate your queries, and I still foresee a problem handling real-time data.

Nevertheless, fine-tuning and RAFT is another way to improve the relevance and accuracy of responses of LLMs such that they are more useful for query and inference of diverse network data.

Getting Started

All of that being said, I do think that many of us in network operations are still trying to really nail down the long term value and how we can actually implement LLMs into our workflow. Today, toward the end of 2024, I think we’re in a good place to start experimenting with technology and adopting it into our NetOps workflows. I think a major challenge, though, will be to figure out how to do it with the very unique, diverse, and real-time data we have in networking.

In spite of these challenges, I believe there is value in incorporating an LLM in your NetOps workflow right now, and these methods can help you get there. This isn’t an all-or-nothing proposition, either. Experiment with a pilot project to query only one type of static internal data (even if it’s fake), like a week of device metrics from some switches. 

Then try different methods to use an open small scale LLMs like Llama to query it. Build a RAG system for text data only, and use free tools like Ollama, Langchain, Llama, Milvus, Pinecone, Chroma, and maybe a free embedding model off the Huggingface website. There are a ton of free tools out there and just as many blog posts from folks building this out for their own domains.

Here’s a quick-start process I’m working on (I’m on step 2):

1. Experiment with querying a small, static dataset of logs or tickets using an open-source LLM like Llama
2. Build a basic RAG system using text-based internal KBs and free vector databases like Pinecone or Chroma.
3. Experiment with agentic AI by setting up a simple orchestrator that triggers alerts based on LLM output.
4. Experiment with fine-tuning a small model like Mistral 7B (or anything very small) using a relatively small dataset. You can do that right on your laptop if it’s decent, or you can use a service like Azure AI Studio.

Thanks,

Phil