Tag

#LLM

26 articles

Poetiq’s Meta-System Automatically Builds a Model-Agnostic Harness That Improved Every LLM Tested on LiveCodeBench Pro Without Fine-Tuning

Poetiq's new meta-system automatically builds a model-agnostic inference harness that improves performance across multiple LLMs without fine-tuning.

May 149

tools

How to Build a Cost-Aware LLM Routing System with NadirClaw Using Local Prompt Classification and Gemini Model Switching

A new tutorial explores how to build a cost-aware LLM routing system using NadirClaw, which classifies prompts locally and switches between models like Gemini for optimal performance and cost efficiency.

May 1118

tech

A Coding Implementation to Build Agent-Native Memory Infrastructure with Memori for Persistent Multi-User and Multi-Session LLM Applications

A new tutorial demonstrates how Memori, an agent-native memory infrastructure, can be implemented to build persistent and context-aware LLM applications in multi-user and multi-session environments.

May 1010

Sakana AI and NVIDIA Introduce TwELL with CUDA Kernels for 20.5% Inference and 21.9% Training Speedup in LLMs

Learn to implement sparse matrix operations using CUDA kernels to achieve 20.5% inference and 21.9% training speedup in LLMs, following the TwELL approach by Sakana AI and NVIDIA.

May 1015

LightSeek Foundation Releases TokenSpeed, an Open-Source LLM Inference Engine Targeting TensorRT-LLM-Level Performance for Agentic Workloads

Learn how to set up and use TokenSpeed, an open-source LLM inference engine optimized for agentic workloads, with step-by-step instructions for beginners.

May 96

Top 10 KV Cache Compression Techniques for LLM Inference: Reducing Memory Overhead Across Eviction, Quantization, and Low-Rank Methods

A new analysis explores top 10 KV cache compression techniques for LLM inference, focusing on eviction, quantization, and low-rank methods to reduce memory overhead.

Apr 299

How to Build Traceable and Evaluated LLM Workflows Using Promptflow, Prompty, and OpenAI

Learn how to build and manage LLM workflows using tools like Promptflow, Prompty, and OpenAI to make your AI projects more reliable and traceable.

Apr 2816

A Coding Implementation on kvcached for Elastic KV Cache Memory, Bursty LLM Serving, and Multi-Model GPU Sharing

Learn how to implement kvcached for dynamic KV-cache management in LLM serving, including setting up Qwen2.5 models with an OpenAI-compatible API and simulating bursty inference workloads.

Apr 2530

Moonshot AI and Tsinghua Researchers Propose PrfaaS: A Cross-Datacenter KVCache Architecture that Rethinks How LLMs are Served at Scale

Learn how PrfaaS (Pre-fill and Decode as a Service) rethinks how large language models are served across datacenters to make AI faster and more efficient.

Apr 1944

tools

Google AI Releases Auto-Diagnose: An Large Language Model LLM-Based System to Diagnose Integration Test Failures at Scale

Google AI introduces Auto-Diagnose, an LLM-powered tool that automates the diagnosis of integration test failures, significantly reducing debugging time for developers.

Apr 1755

Physical Intelligence shows robot model with LLM-like generalization, flaws included

This article explains the concept of compositional generalization in robotics, as demonstrated by the π0.7 robot model from Physical Intelligence. It explores how robots can recombine learned skills to tackle novel tasks, similar to how large language models generate new text.

Apr 1739

tech

An End-to-End Coding Guide to NVIDIA KVPress for Long-Context LLM Inference, KV Cache Compression, and Memory-Efficient Generation

NVIDIA's KVPress offers a memory-efficient solution for long-context language model inference through advanced KV cache compression, enabling more scalable AI applications.

Apr 948