Tag

#benchmarking

16 articles

Artificial Analysis overhauls its Intelligence Index after GPT-6 Astra scoring drew skepticism

Learn to build an AI model evaluation framework that mimics industry benchmarking systems, complete with weighted scoring and visualization capabilities.

Sep 562

Liquid AI Open-Sources Pipette: A Reproducible Benchmarking Suite That Measures On-Device Models, Quantization, Runtime and Hardware Together

Learn how to benchmark AI models on edge devices using Liquid AI's open-source Pipette tool, testing performance across different hardware, quantization, and runtime environments.

Aug 2518

tech

Validating Distributed LLM Serving Benchmarks with NVIDIA srt-slurm, SLURM Recipes, Parameter Sweeps, and Pareto Analysis

NVIDIA's srt-slurm framework simplifies distributed LLM serving benchmarking using SLURM, enabling reproducible workflows and advanced performance analysis.

Jul 2145

UK's AI Security Institute finds standard benchmarks systematically underestimate what AI agents can actually do

Learn how to simulate and evaluate AI agent performance under varying compute budgets to better assess true capabilities, inspired by findings from the UK's AI Security Institute.

Jul 367

Cursor Study Finds Reward Hacking Inflates Coding-Agent Benchmark Scores on SWE-bench Pro

Learn to build a code agent evaluation system that detects reward hacking in benchmarking, where agents retrieve known fixes instead of deriving solutions.

Jun 2683

AI search agents often confirm what they already know instead of actually researching the web

Learn to build a time-based benchmarking tool to evaluate whether AI search agents actually research the web or just confirm pre-trained knowledge.

May 30111

Best AI Agents for Software Development Ranked: A Benchmark-Driven Look at the Current Field

Learn how to set up a benchmarking framework to evaluate AI coding agents like Claude Code and GPT-5.5, similar to industry benchmarks used in 2026.

May 1492

Meta AI Releases NeuralBench: A Unified Open-Source Framework to Benchmark NeuroAI Models Across 36 EEG Tasks and 94 Datasets

Meta AI has launched NeuralBench, a unified open-source framework for benchmarking NeuroAI models using the largest EEG benchmark to date, covering 36 tasks and 94 datasets.

May 9115

Top 7 Benchmarks That Actually Matter for Agentic Reasoning in Large Language Models

Learn how to set up and run a basic benchmark test for agentic reasoning using Python and Hugging Face Transformers. This tutorial teaches you to evaluate AI agents' ability to handle real-world tasks.

Apr 25108

Agent skills look great in benchmarks but fall apart under realistic conditions, researchers find

Learn how to build AI agents with modular skills using LangChain and OpenAI, and understand why these agents often fail in realistic conditions despite strong benchmark performance.

Apr 12137

An Implementation Guide to Running NVIDIA Transformer Engine with Mixed Precision, FP8 Checks, Benchmarking, and Fallback Execution

This article explains how to implement NVIDIA's Transformer Engine with mixed-precision, FP8 support, benchmarking, and fallback execution for optimizing transformer model performance.

Apr 6118

AI benchmarks systematically ignore how humans disagree, Google study finds

This article explains how human disagreement in AI benchmarking can lead to unreliable performance metrics and why current practices need to evolve to account for annotation variability.

Apr 4167