I generally get this kind of information from open router's (OR) model API. One of the areas where I think there payload is weak is in the seperation of model capabilities. I like the way you are proposing to sperate capabilities into individual text, image and audio. For multi-modal models I would expect that each "mode" is true. You might consider capabilities as being bi-direction. for example text->image and/or image->text. same with the audio. Does it support audio in and text out (transcription) and/or text in and audio out (text to speech.

One of the areas where I think the OR payload is strong is in the model description. Here is an example:

- :id: openai/o3-mini-high
  :name: 'OpenAI: o3 Mini High'
  :created: 1739372611
  :description: "OpenAI o3-mini-high is the same model as [o3-mini](/openai/o3-mini)
    with reasoning_effort set to high. \n\no3-mini is a cost-efficient language model
    optimized for STEM reasoning tasks, particularly excelling in science, mathematics,
    and coding. The model features three adjustable reasoning effort levels and supports
    key developer capabilities including function calling, structured outputs, and
    streaming, though it does not include vision processing capabilities.\n\nThe model
    demonstrates significant improvements over its predecessor, with expert testers
    preferring its responses 56% of the time and noting a 39% reduction in major errors
    on complex questions. With medium reasoning effort settings, o3-mini matches the
    performance of the larger o1 model on challenging reasoning evaluations like AIME
    and GPQA, while maintaining lower latency and cost."
  :context_length: 200000
  :architecture:
    modality: text->text
    tokenizer: Other
    instruct_type:
  :pricing:
    prompt: '0.0000011'
    completion: '0.0000044'
    image: '0'
    request: '0'
  :top_provider:
    context_length: 200000
    max_completion_tokens: 100000
    is_moderated: true
  :per_request_limits:

Thank you so much for the info @MadBomber!

Updated the spec taking inspiration from OpenRouter @raznem @MadBomber

@crmne Any updates on this? That would be a great resource for our LLM client/framework for Smalltalk.

The spec looks very helpful! Specifically for OpenAI but perhaps generalizable, the following addition would be helpful but not desired:

• cached token prices for audio/image input tokens
• availability in endpoints (eg, chat completions, responses, realtime, ...)
• default rate limits per tier (though actual rate limits can vary and are provided by responses from the official API)
• resolved model aliases (e.g., gpt-realtime currently points to gpt-realtime-2025-08-28)
• [very maybe also the performance/speed assessments from the official model pages)

It would be great to hear whether this idea has made any progress or how I could maybe contribute!