Agent Evaluation · Shopping Agents · Agent Observability
A blog and newsletter on AI agent evaluation methodology and benchmarking and E-commerce and shopping agent capabilities.
Latest research
Agent Observability
Latency Profiling in LLM-Backed Agent Pipelines
Decomposed traces reveal where agent pipelines actually bottleneck.
- Preference Alignment in Personalized Shopping AgentsShopping agents must infer what customers actually want from incomplete signals and past behavior.
- Distributed Tracing for Multi-Agent LLM SystemsHierarchical spans and explicit context propagation enable accountability across agent boundaries.
- Natural Language Constraint Parsing in Shopping Agent InputsShopping agents must parse casual requests into structured constraints they can actually enforce.
- Price Comparison Agent Accuracy Across Retailer SchemasRetailers' inconsistent product schemas break price agents before models even process the data.
Agent Evaluation
View all →- Human Evaluation Protocols for Open-Ended Agent TasksEvaluating agent systems requires inspecting their decision paths, not just their final answers.
- Counterfactual Testing for Agent Decision PointsIdentifying which agent decisions actually caused failures, not just where they appeared.
- Regression Testing Pipelines for Iterative Agent DevelopmentAutomated testing catches AI agents answering the wrong question while appearing to succeed.
- Eval Harness Design for Tool-Using AgentsHarness design choices shape agent performance as much as the underlying model weights do.
Shopping Agents
View all →- Preference Alignment in Personalized Shopping AgentsShopping agents must infer what customers actually want from incomplete signals and past behavior.
- Natural Language Constraint Parsing in Shopping Agent InputsShopping agents must parse casual requests into structured constraints they can actually enforce.
- Price Comparison Agent Accuracy Across Retailer SchemasRetailers' inconsistent product schemas break price agents before models even process the data.
- Product Search and Filtering Accuracy in Shopping AgentsShopping agents eliminate products before ranking them, making data accuracy a survival question.
Agent Observability
View all →- Latency Profiling in LLM-Backed Agent PipelinesDecomposed traces reveal where agent pipelines actually bottleneck.
- Distributed Tracing for Multi-Agent LLM SystemsHierarchical spans and explicit context propagation enable accountability across agent boundaries.
- Observability Tooling Comparison for LLM Agent FrameworksMost LLM monitoring logs outputs without judging them, and that's the gap.