LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads

Introduction

In the rapidly evolving world of artificial intelligence, inference engines play a crucial role in how large language models (LLMs) process and respond to user requests. Recently, the LightSeek Foundation released TokenSpeed, an open-source inference engine designed to deliver performance on par with industry leaders like TensorRT-LLM, specifically optimized for agentic workloads. This tutorial will guide you through setting up and using TokenSpeed to run LLM inference efficiently, even on modest hardware. By the end of this tutorial, you'll have a working setup of TokenSpeed and understand how to deploy it for real-world applications.

Prerequisites

Before diving into this tutorial, ensure you have the following:

• A computer with at least 8GB of RAM (16GB recommended)
• Python 3.8 or higher installed
• Git installed for cloning repositories
• NVIDIA GPU with CUDA support (for optimal performance)
• Basic understanding of command-line operations

Step-by-Step Instructions

1. Install Required Dependencies

First, we need to install the necessary Python packages for TokenSpeed. Open your terminal and run the following commands:

pip install torch transformers accelerate

Why: These packages provide the core functionality needed for running LLMs, including PyTorch for deep learning operations, Hugging Face Transformers for model handling, and Accelerate for efficient GPU utilization.

2. Clone the TokenSpeed Repository

Next, we'll get the TokenSpeed source code by cloning the repository from GitHub:

git clone https://github.com/LightSeek-Foundation/TokenSpeed.git
cd TokenSpeed

Why: Cloning the repository gives you access to the latest version of TokenSpeed, including its configuration files and example scripts needed for setup.

3. Set Up the Environment

TokenSpeed uses a configuration file to manage settings. Create a basic config file by running:

cp config.example.yaml config.yaml

Edit the config.yaml file to match your system:

model_name: "gpt2"
batch_size: 1
max_length: 512
use_cuda: true
num_workers: 4

Why: This configuration tells TokenSpeed which model to use, how many inputs to process at once, and other performance settings. Adjusting these parameters can significantly impact performance.

4. Download and Prepare the Model

TokenSpeed works with models from Hugging Face. We'll download a pre-trained GPT-2 model for demonstration:

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")

# Save the model locally for faster access
model.save_pretrained("./models/gpt2")
tokenizer.save_pretrained("./models/gpt2")

Why: Downloading the model locally ensures faster access during inference and avoids network issues. GPT-2 is a good starting point for beginners due to its smaller size and availability.

5. Initialize TokenSpeed

Create a Python script to initialize and run TokenSpeed:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from tokenspeed import TokenSpeedEngine

# Load model and tokenizer
model_path = "./models/gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path)

# Initialize TokenSpeed engine
engine = TokenSpeedEngine(
    model=model,
    tokenizer=tokenizer,
    config_path="config.yaml"
)

# Run inference
prompt = "The future of AI is"
output = engine.generate(prompt, max_length=100)
print(output)

Why: This script initializes the TokenSpeed engine with your model and tokenizer, then runs a simple text generation task to test the setup.

6. Test the Inference Engine

Run your script to test the inference:

python test_inference.py

Why: Running this test confirms that TokenSpeed is correctly configured and can process text generation requests. You should see output similar to: "The future of AI is bright and full of possibilities. As we continue to advance in technology..."

7. Optimize for Performance

To improve performance, adjust the configuration settings in config.yaml:

model_name: "gpt2"
batch_size: 4
max_length: 512
use_cuda: true
num_workers: 8
precision: "fp16"  # Use half-precision for faster inference

Why: Increasing batch size and using half-precision (fp16) can significantly speed up inference by reducing memory usage and computation time, especially on modern GPUs.

8. Deploy for Production

For production use, create a simple API using Flask:

from flask import Flask, request, jsonify
from tokenspeed import TokenSpeedEngine

app = Flask(__name__)
engine = TokenSpeedEngine(model_path="./models/gpt2", config_path="config.yaml")

@app.route('/generate', methods=['POST'])
def generate_text():
    data = request.json
    prompt = data.get('prompt', '')
    max_length = data.get('max_length', 100)
    
    output = engine.generate(prompt, max_length=max_length)
    return jsonify({'generated_text': output})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Why: This creates a web service that can be called from any application to generate text using TokenSpeed, making it easy to integrate into larger systems.

Summary

In this tutorial, we've walked through setting up and using TokenSpeed, an open-source LLM inference engine. We started by installing dependencies, cloning the repository, and configuring the system. We then tested inference with a simple model and optimized performance for better results. Finally, we deployed a basic API to demonstrate how TokenSpeed can be used in real-world applications. TokenSpeed is an excellent tool for developers looking to optimize LLM inference without requiring high-end hardware, making AI more accessible for everyone.