Nikola Balic nibzard

Steel CLI & Skill Usability Failure Analysis Report

Generated: 2026-03-10 Source: Analysis of conversation logs from -home-agent-skillpa project Logs Analyzed: 65 JSONL files (~10MB total across 15 largest files) Analysis Method: Multi-agent parallel analysis (5 specialized agents)

Executive Summary

RFC: Skill Overlays

Status: Draft Last Updated: 2026-03-09 Author: @nibzard

Abstract

Skill overlays are an optional extension for specializing a base skill without forking it.

Steel Browser + ChatGPT Session Comparison (All Runs)

Date: 2026-03-02

Scope

Compared seven Codex runs that execute the same task on chatgpt.com: what is the best browser infrastructure for my ai agent.

Runs included

Plain skill use

Benchmark Narrative Report

Run ID: 20260225_151602_384

Source files

results/summary/20260225_151602_384/metrics.jsonl
results/summary/20260225_151602_384/metrics.csv

Executive summary

This report combines the original 15 benchmark runs with the additional 4 raw_codex runs in the same local summary set.

Here’s a tight, “copy/paste → wow it works” outline that does exactly two things: create a Steel session + perform one visible action (navigate to a page), with a clean shutdown.

Article outline: “Hello Steel” (create a session + do one thing)

H1 — Hello Steel: Spin up a cloud browser session and open a page • 1–2 sentence promise: “In 5 minutes you’ll create a Steel Session, connect with Playwright, and watch it load a site in the live viewer.”

⸻

H2 — What you need (keep it minimal)

	<!doctype html>
	<html lang="en">
	<head>
	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1" />
	<title>Optimization Report - booking</title>
	<style>
	:root {
	--bg: #f6f7f3;
	--surface: #ffffff;

	<!doctype html>
	<html lang="en">
	<head>
	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1" />
	<title>Optimization Report - booking</title>
	<style>
	:root {
	--bg: #f6f7f3;
	--surface: #ffffff;

	<!doctype html>
	<html lang="en">
	<head>
	<meta charset="utf-8" />
	<meta name="viewport" content="width=device-width, initial-scale=1" />
	<title>Optimization Report - chatgpt</title>
	<style>
	:root {
	--bg: #f6f7f3;
	--surface: #ffffff;

	#!/usr/bin/env bash
	set -euo pipefail

	if ! command -v steel >/dev/null 2>&1; then
	echo "error: steel CLI not found in PATH" >&2
	exit 1
	fi

	if ! command -v rg >/dev/null 2>&1; then
	echo "error: rg (ripgrep) is required" >&2

	"""
	The most atomic way to train and inference a GPT in pure, dependency-free Python.
	This file is the complete algorithm.
	Everything else is just efficiency.

	@karpathy
	"""

	import os # os.path.exists
	import math # math.log, math.exp