Alright, my friends, I’m back with another post based on my learnings and exploration of AI and how it’ll fit into our work as network engineers. In today’s post, I want to share the first (of what will likely be many) “nerd knobs” that I think we all should be aware of and how they will impact our use of AI and AI tools. I can already sense the excitement in the room. After all, there’s not much a network engineer likes more than tweaking a nerd knob in the network to fine-tune performance. And that’s exactly what we’ll be doing here. Fine-tuning our AI tools to help us be more effective.
First up, the requisite disclaimer or two.
- There are SO MANY nerd knobs in AI. (Shocker, I know.) So, if you all like this kind of blog post, I’d be happy to return in other posts where we look at other “knobs” and settings in AI and how they work. Well, I’d be happy to return once I understand them, at least. 🙂
- Changing any of the settings on your AI tools can have dramatic effects on results. This includes increasing the resource consumption of the AI model, as well as increasing hallucinations and decreasing the accuracy of the information that comes back from your prompts. Consider yourselves warned. As with all things AI, go forth and explore and experiment. But do so in a safe, lab environment.
For today’s experiment, I’m once again using LMStudio running locally on my laptop rather than a public or cloud-hosted AI model. For more details on why I like LMStudio, check out my last blog, Creating a NetAI Playground for Agentic AI Experimentation.
Enough of the setup, let’s get into it!
The impact of working memory size, a.k.a. “context”
Let me set a scene for you.
You’re in the middle of troubleshooting a network issue. Someone reported, or noticed, instability at a point in your network, and you’ve been assigned the joyful task of getting to the bottom of it. You captured some logs and relevant debug information, and the time has come to go through it all to figure out what it means. But you’ve also been using AI tools to be more productive, 10x your work, impress your boss, you know all the things that are going on right now.
So, you decide to see if AI can help you work through the data faster and get to the root of the issue.
You fire up your local AI assistant. (Yes, local—because who knows what’s in the debug messages? Best to keep it all safe on your laptop.)
You tell it what you’re up to, and paste in the log messages.
After getting 120 or so lines of logs into the chat, you hit enter, kick up your feet, reach for your Arnold Palmer for a refreshing drink, and wait for the AI magic to happen. But before you can take a sip of that iced tea and lemonade goodness, you see this has immediately popped up on the screen:
Oh my.
“The AI has nothing to say.”!?! How could that be?
Did you find a question so difficult that AI can’t handle it?
No, that’s not the problem. Check out the helpful error message that LMStudio has kicked back:
“Trying to keep the first 4994 tokens when context the overflows. However, the model is loaded with context length of only 4096 tokens, which is not enough. Try to load the model with a larger context length, or provide shorter input.”
And we’ve gotten to the root of this perfectly scripted storyline and demonstration. Every AI tool out there has a limit to how much “working memory” it has. The technical term for this working memory is “context length.” If you try to send more data to an AI tool than can fit into the context length, you’ll hit this error, or something like it.
The error message indicates that the model was “loaded with context length of only 4096 tokens.” What is a “token,” you wonder? Answering that could be a topic of an entirely different blog post, but for now, just know that “tokens” are the unit of size for the context length. And the first thing that is done when you send a prompt to an AI tool is that the prompt is converted into “tokens”.
So what do we do? Well, the message gives us two possible options: we can increase the context length of the model, or we can provide shorter input. Sometimes it isn’t a big deal to provide shorter input. But other times, like when we are dealing with large log files, that option isn’t practical—all of the data is important.
Time to turn the knob!
It is that first option, to load the model with a larger context length, that is our nerd knob. Let’s turn it.
From within LMStudio, head over to “My Models” and click to open up the configuration settings interface for the model.
You’ll get a chance to view all the knobs that AI models have. And as I mentioned, there are a lot of them.
But the one we care about right now is the Context Length. We can see that the default length for this model is 4096 tokens. But it supports up to 8192 tokens. Let’s max it out!
LMStudio provides a helpful warning and probable reason for why the model doesn’t default to the max. The context length takes memory and resources. And raising it to “a high value” can impact performance and usage. So if this model had a max length of 40,960 tokens (the Qwen3 model I use sometimes has that high of a max), you might not want to just max it out right away. Instead, increase it by a little at a time to find the sweet spot: a context length big enough for the job, but not oversized.
As network engineers, we are used to fine-tuning knobs for timers, frame sizes, and so many other things. This is right up our alley!
Once you’ve updated your context length, you’ll need to “Eject” and “Reload” the model for the setting to take effect. But once that’s done, it’s time to take advantage of the change we’ve made!
And look at that, with the larger context window, the AI assistant was able to go through the logs and give us a nice write-up about what they show.
I particularly like the shade it threw my way: “…consider seeking assistance from … a qualified network engineer.” Well played, AI. Well played.
But bruised ego aside, we can continue the AI assisted troubleshooting with something like this.
And we’re off to the races. We’ve been able to leverage our AI assistant to:
- Process a significant amount of log and debug data to identify possible issues
- Develop a timeline of the problem (that will be super useful in the help desk ticket and root cause analysis documents)
- Identify some next steps we can do in our troubleshooting efforts.
All stories must end…
And so you have it, our first AI Nerd Knob—Context Length. Let’s review what we learned:
- AI models have a “working memory” that is referred to as “context length.”
- Context Length is measured in “tokens.”
- Oftentimes times an AI model will support a higher context length than the default setting.
- Increasing the context length will require more resources, so make changes slowly, don’t just max it out completely.
Now, depending on what AI tool you’re using, you may NOT be able to adjust the context length. If you’re using a public AI like ChatGPT, Gemini, or Claude, the context length will depend on the subscription and models you have access to. However, there most definitely IS a context length that will factor into how much “working memory” the AI tool has. And being aware of that fact, and its impact on how you can use AI, is important. Even if the knob in question is behind a lock and key. 🙂
If you enjoyed this look under the hood of AI and would like to learn about more options, please let me know in the comments: Do you have a favorite “knob” you like to turn? Share it with all of us. Until next time!
PS… If you’d like to learn more about using LMStudio, my buddy Jason Belk put a free tutorial together called Run Your Own LLM Locally For Free and with Ease that can get you started very quickly. Check it out!
Sign up for Cisco U. | Join the Cisco Learning Network today for free.
Learn with Cisco
X | Threads | Facebook | LinkedIn | Instagram | YouTube
Use #CiscoU and #CiscoCert to join the conversation.
Read next:
Share:
