David Paquet Pitts davidpp

LangSmith Insights Agent: Complete Reverse Engineering

Executive Summary

LangSmith's Insights Agent is an LLM-orchestrated data analysis system that automatically discovers usage patterns and failure modes in production agent traces. Unlike traditional clustering tools, it accepts natural language questions from users and dynamically adapts its entire analysis pipeline—from feature extraction to taxonomy generation—based on those questions.

Key Innovation: It's a conversational interface to trace analysis, where users describe what they want to learn, and the system translates that into a custom analytical workflow.

Core Capabilities:

Analyzes up to 1,000 sampled traces in up to 30 minutes (typically 15-20 minutes)

OS X Screencast to animated GIF

This gist shows how to create a GIF screencast using only free OS X tools: QuickTime, ffmpeg, and gifsicle.

Instructions

To capture the video (filesize: 19MB), using the free "QuickTime Player" application:

	BEGIN;

	-- Ensure indexes are in place for performance
	CREATE INDEX IF NOT EXISTS idx_activity_timestamp ON directus_activity (timestamp);
	CREATE INDEX IF NOT EXISTS idx_revisions_activity ON directus_revisions (activity);
	CREATE INDEX IF NOT EXISTS idx_revisions_item_collection ON directus_revisions (item, collection);
	CREATE INDEX IF NOT EXISTS idx_notifications_timestamp ON directus_notifications (timestamp);

	DO $$
	DECLARE

	echo "Creating an SSH key for you..."
	ssh-keygen -t rsa

	echo "Please add this public key to Github \n"
	echo "https://github.com/account/ssh \n"
	read -p "Press [Enter] key after this..."

	echo "Installing xcode-stuff"
	xcode-select --install

	# dependency - Function.prototype.bind or underscore/lodash

	app = angular.module 'someApp'

	class @BaseCtrl
	@register: (app, name) ->
	name ?= @name \|\| @toString().match(/function\s(.?)\(/)?[1]
	app.controller name, @

	@inject: (args...) ->

	ExampleController = ($scope, LocationService) ->
	LocationService.locate().then (position) ->
	$scope.position = position


	ExampleController.$inject = ["$scope", "LocationService"]

	YourAngularApp.controller("ExampleController", ExampleController)

	Services.factory("LawnchairFactory", function($window, $log, $parse) {
	return function(name, config) {
	var collection = {};
	var array = [];
	var isArray = config && config.isArray;
	var idGetter = $parse((config && config.entryKey) ? config.entryKey : "id");
	var transformSave = (config && config.transformSave) ? config.transformSave : angular.identity;
	var transformLoad = (config && config.transformLoad) ? config.transformLoad : angular.identity;

	function getEntryId(entry){