Introduction
Large Language Models (LLMs) are powerful tools, but they face significant challenges when working on long, complex tasks. Two major issues are context overflow and goal loss. Context overflow happens when an LLM's input context window is exceeded, causing it to lose information from earlier parts of the conversation. Goal loss occurs when the model loses track of the original task objective over time. This tutorial will show you how to build a simple context management system that addresses these problems using four key mechanisms: context summarization, task state tracking, dynamic context trimming, and goal reinforcement.
By the end of this tutorial, you'll understand how to create a basic system that helps LLMs maintain context and task focus during long conversations, similar to what's used in advanced agents like LangChain Deep Agents and Claude Code.
Prerequisites
To follow this tutorial, you'll need:
- A basic understanding of Python programming
- Python 3.7 or higher installed on your system
- Access to an LLM API (we'll use OpenAI's API for this example)
- Basic knowledge of how LLMs work (prompting, context windows, etc.)
Step-by-Step Instructions
1. Set Up Your Environment
First, we need to install the required Python packages. Open your terminal and run:
pip install openai python-dotenvThis installs the OpenAI Python library and dotenv for managing API keys securely.
2. Create Your API Key File
Create a file named .env in your project directory and add your OpenAI API key:
OPENAI_API_KEY=your_api_key_hereReplace your_api_key_here with your actual API key from OpenAI.
3. Initialize Your Context Manager
Let's create a basic context manager class that will track conversation history and apply our four mechanisms:
import os
import openai
from dotenv import load_dotenv
# Load environment variables
load_dotenv()
# Initialize OpenAI client
openai.api_key = os.getenv('OPENAI_API_KEY')
class ContextManager:
def __init__(self, max_context_tokens=2000, summary_threshold=1000):
self.conversation_history = []
self.max_context_tokens = max_context_tokens
self.summary_threshold = summary_threshold
self.task_goal = ""
def add_message(self, role, content):
"""Add a message to the conversation history"""
self.conversation_history.append({
'role': role,
'content': content
})
def set_task_goal(self, goal):
"""Set the main task goal for the conversation"""
self.task_goal = goal
def get_context(self):
"""Get current context with applied mechanisms"""
# Apply dynamic context trimming
if len(self.conversation_history) > 0:
# Summarize old context if needed
if self._should_summarize():
self._summarize_context()
# Reinforce goal
self._reinforce_goal()
return self.conversation_history
def _should_summarize(self):
"""Check if context needs summarization"""
# Simple heuristic: if we're approaching max tokens
current_tokens = self._count_tokens()
return current_tokens > self.summary_threshold
def _count_tokens(self):
"""Simple token counter (in practice, use a proper tokenizer)"""
# This is a simplified token count
total = 0
for msg in self.conversation_history:
total += len(msg['content']) // 4 # Approximate tokens
return total
def _summarize_context(self):
"""Summarize the conversation history to reduce context length"""
# Get the conversation history to summarize
messages_to_summarize = self.conversation_history[:-2] # Leave last 2 messages
if len(messages_to_summarize) > 0:
# Create a summary prompt
summary_prompt = f"Summarize the following conversation in 2 sentences:\n\n"
for msg in messages_to_summarize:
summary_prompt += f"{msg['role']}: {msg['content']}\n\n"
# Get summary from LLM
try:
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system", "content": "You are a helpful assistant that summarizes conversations."},
{"role": "user", "content": summary_prompt}
],
max_tokens=100
)
# Replace old messages with summary
summary_content = response.choices[0].message.content
self.conversation_history = [self.conversation_history[-2], self.conversation_history[-1]]
self.conversation_history.insert(0, {"role": "assistant", "content": f"Summary of previous conversation: {summary_content}"})
except Exception as e:
print(f"Error during summarization: {e}")
def _reinforce_goal(self):
"""Reinforce the task goal in the context"""
if self.task_goal and len(self.conversation_history) > 0:
# Add goal reminder at the beginning
goal_prompt = f"Remember: Your task is to {self.task_goal}."
# Check if goal is already present
goal_present = any('Remember:' in msg['content'] and self.task_goal in msg['content']
for msg in self.conversation_history)
if not goal_present:
self.conversation_history.insert(0, {"role": "system", "content": goal_prompt})
# Initialize the context manager
context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)4. Test Your Context Manager
Now let's create a simple test to see how the context manager works:
# Set up the context manager
context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)
# Set task goal
context_manager.set_task_goal("analyze customer feedback and generate a summary report")
# Add some messages
context_manager.add_message("user", "I want to analyze customer feedback about our new product.")
context_manager.add_message("assistant", "I can help with that. What specific aspects would you like to focus on?")
context_manager.add_message("user", "I want to focus on product features, pricing, and user experience.")
# Get current context
context = context_manager.get_context()
print("Current context:")
for msg in context:
print(f"{msg['role']}: {msg['content']}")5. Simulate Long Conversations
Let's simulate a long conversation to see how our context manager handles it:
def simulate_long_conversation():
# Set up
context_manager = ContextManager(max_context_tokens=2000, summary_threshold=1000)
context_manager.set_task_goal("research and summarize the impact of AI on healthcare")
# Simulate adding many messages
for i in range(50):
user_input = f"This is message number {i} about AI in healthcare. It contains some information about how artificial intelligence is changing medical diagnosis."
context_manager.add_message("user", user_input)
# Simulate assistant response
assistant_response = f"I understand message {i}. This is a response to your input about AI in healthcare."
context_manager.add_message("assistant", assistant_response)
# Print context status every 10 messages
if i % 10 == 0:
print(f"After {i} messages:")
context = context_manager.get_context()
print(f"Total messages in context: {len(context)}")
print(f"Context tokens: {context_manager._count_tokens()}")
print("---")
# Run simulation
simulate_long_conversation()6. Add Task State Tracking
Let's enhance our context manager with task state tracking to prevent goal loss:
class EnhancedContextManager(ContextManager):
def __init__(self, max_context_tokens=2000, summary_threshold=1000):
super().__init__(max_context_tokens, summary_threshold)
self.task_state = {}
def update_task_state(self, key, value):
"""Update the task state"""
self.task_state[key] = value
def get_task_state(self):
"""Get current task state"""
return self.task_state
def _reinforce_goal(self):
"""Enhanced goal reinforcement with task state"""
if self.task_goal and len(self.conversation_history) > 0:
# Add goal reminder at the beginning
goal_prompt = f"Remember: Your task is to {self.task_goal}."
# Add task state reminder
if self.task_state:
state_str = ", ".join([f'{k}: {v}' for k, v in self.task_state.items()])
goal_prompt += f"\nCurrent task state: {state_str}"
# Check if goal is already present
goal_present = any('Remember:' in msg['content'] and self.task_goal in msg['content']
for msg in self.conversation_history)
if not goal_present:
self.conversation_history.insert(0, {"role": "system", "content": goal_prompt})Summary
In this tutorial, you've learned how to build a basic context management system that addresses two major challenges in long-horizon LLM tasks: context overflow and goal loss. We've implemented four key mechanisms:
- Context summarization: Automatically summarize old conversation history when it approaches token limits
- Task state tracking: Maintain and update the current state of the task
- Dynamic context trimming: Remove unnecessary information while preserving key details
- Goal reinforcement: Continuously remind the model of the original task objective
These mechanisms work together to help LLMs maintain context and task focus, similar to what's used in advanced agent systems. While this is a simplified implementation, it demonstrates the core concepts that power systems like LangChain Deep Agents and Claude Code. In practice, you'd want to use more sophisticated token counting, better summarization models, and more robust state management.
