Skip to content

Instantly share code, notes, and snippets.

@etrotta

etrotta/geminiextractor-ptbr.ipynb

Created May 9, 2024 00:42
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save etrotta/7c87f9ebeab0554769a518ef6d0bcfd2 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save etrotta/7c87f9ebeab0554769a518ef6d0bcfd2 to your computer and use it in GitHub Desktop.
Download ZIP
GeminiExtractor-PTBR.ipynb
Display the source blob
Display the rendered blob
Raw
geminiextractor-ptbr.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"language_info": {
"name": "python"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/etrotta/7c87f9ebeab0554769a518ef6d0bcfd2/geminiextractor-ptbr.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"source": [
"## Visão Geral - Principais Etapas\n",
"\n",
"- Crie uma copia desse Notebook para o seu próprio Drive (menu acima)\n",
"- Acesse o <a href=\"https://aistudio.google.com/\">Google AI Studio</a> e faça login com sua conta do Google\n",
"- <a href=\"https://aistudio.google.com/app/apikey\">Crie uma chave de API</a>.</li>\n",
"- Salve-a como um Segredo chamado \"`GOOGLE_API_KEY`\" (menu a direita)\n",
"- Defina os **Modelos de Dados**, as **Funções de Modelo**, e o **Template de Prompt** (mais sobre isso depois)\n",
"- Carregue seus arquivos\n",
"\n",
"---\n",
"\n",
"## Como funciona\n",
"\n",
"Este Notebook permite que você defina Modelos de Dados baseados no [Pydantic](https://docs.pydantic.dev/latest/) para formatar como a saída será exibida, e em seguida, carregue seus arquivos de texto para o Gemini de forma que ele possa extrair dados em um formato fixo.\n",
"\n",
"Não se preocupe com a parte mais tecnica do codigo nem do Pydantic - basta ajustar o modelo exemplo abaixo.\n",
"\n",
"É fortemente recomendado usar nomes e descrições em Inglês para **Modelos de Dados e Funções de Modelo**, mas resto (Template de Prompt, a descrição dos campos do modelo de dados e os arquivos enviados) podem ser escritos em Português.\n",
"\n",
"---\n",
"### IMPORTANTE\n",
"\n",
"Você precisa adicionar sua própria Chave de API do Gemini e personalizar os:\n",
"- Modelos de Dados\n",
"- Funções de Modelo\n",
"- Template da Prompt\n",
"\n",
"O programa fará uma solicitação ao Gemini para cada combinação de `Arquivo x Modelo de Dados`, passando o conteúdo inteiro do arquivo a cada vez, logo tenha cuidado com os seus limites de uso da API.\n",
"\n",
"Todas as funções de modelo são passadas para o Gemini e cabe a ele decidir quais usar.\n",
"\n",
"O modelo analisará apenas um arquivo por vez - ele não tem acesso ao historico de arquivos previous nem informações sobre prompts anteriores. O Template da Prompt deve conter todas as informações necessárias, especialmente o `conteudo_do_arquivo`.\n",
"\n",
"---\n",
"\n",
"## Configuração:\n",
"### Funções de Modelo\n",
"\n",
"`FUNÇÕES_DE_MODELO` define as funções que o Gemini tentará chamar para cada elemento que ele identifica nos arquivos enviados.\n",
"\n",
"Conceitualmente, uma função define uma ação que ele tentará iniciar por meio dos dados que ele retorna, por exemplo registrar ou atualizar os dados.\n",
"\n",
"> Formato: (nome da função, descrição da função)\n",
"\n",
"Elas serão combinadas com cada Modelo de Dados registrado. Por exemplo:\n",
"```python\n",
"FUNÇÕES_DE_MODELO = [\n",
" (\"register_{}\", \"Register a new\"),\n",
"]\n",
"\n",
"@registrar_modelo\n",
"class Character(pydantic.BaseModel):\n",
" \"important character present in the book\"\n",
" nome: str = pydantic.Field(default=\"Desconhecido\", description=\"O nome real deste personagem.\")\n",
"```\n",
"\n",
"Isso gerará uma função chamada `\"register_character\"`, cuja descrição enviada ao Gemini é `\"Register a new important character present in the book\"`\n",
"\n",
"---\n",
"\n",
"Observação tecnica: Apesar de chamá-las de Funções, elas não são funções no sentido de funções do python, criadas com `def`.\n",
"As funções de modelo só são tratadas como \"funções\" conceitualmente, uma vez que o Gemini tenta as invocar, porém na pratica isso so é usado para formatar os dados - essas funções não existem de verdade dentro do Python.\n",
"\n",
"\n",
"---\n",
"\n",
"## Modelo de Prompt\n",
"\n",
"O `PROMPT_TEMPLATE` será usado como base para criar o Prompt enviado para o modelo.\n",
"\n",
"Os seguintes _keyword arguments_ serão passados para `PROMPT_TEMPLATE.format()`, o que significa que você deve incluí-los no template usando `{variavel}`:\n",
"\n",
"- `modelo_de_dados`: O nome do Modelo de Dados para o qual está sendo chamado\n",
"- `conteudo_do_arquivo`: O conteúdo do arquivo\n",
"\n",
"Por exemplo,\n",
"```\n",
"PROMPT_TEMPLATE = \"\"\"Dado o seguinte documento, \\\n",
"identifique e registre todos os {modelo_de_dados}s importantes, \\\n",
"incluindo todos os dados relevantes sobre cada um deles.\n",
"---\n",
"{conteudo_do_arquivo}\n",
"\"\"\"\n",
"```\n",
"\n",
"---\n",
"\n",
"### Modelos de Dados\n",
"\n",
"Você precisa criar modelos Pydantic para definir como a saída do modelo deve ser formatada.\n",
"\n",
"Combinados com as Funções de Modelos, eles são convertidos para um formato que o Gemini entende, mas para que isso aconteça, você precisa usar `@registrar_modelo` ao criar o modelo.\n",
"\n",
"---\n",
"\n",
"Atualmente, os tipos de dados suportados são:\n",
"\n",
"- `str` (String; Qualquer Texto)\n",
"- `bool` (Booleano; Verdadeiro ou Falso)\n",
"- `int` (Inteiros; Números sem parte fracionária)\n",
"- `float` (Decimal; Números que podem ter uma parte fracionária)\n",
"\n",
"Também são suportadas listas dos tipos acima,\n",
"> `list[str]`, `list[bool]`, `list[int]`, `list[float]`\n",
"\n",
"Mas ainda não suportadas listas dentro de outras listas nem dicionários.\n",
"\n",
"Você também pode usar `typing.Optional` para permitir \"`None`\" (\"nulo\") como um valor:\n",
"\n",
"> `typing.Optional[str]`, `typing.Optional[list[str]]`\n",
"> `typing.Optional[bool]`, `typing.Optional[list[bool]]`\n",
"> `typing.Optional[int]`, `typing.Optional[list[int]]`\n",
"> `typing.Optional[float]`, `typing.Optional[list[float]]`\n",
"\n",
"Mas você precisa especificar um valor `default=` ou uma função `default_factory=` para que seja o campo tratado como não obrigatório.\n",
"\n",
"Por exemplo:\n",
"```python\n",
"@registrar_modelo\n",
"class Exemplo(pydantic.BaseModel):\n",
" \"code example in the documentation\"\n",
" valor_1: int = pydantic.Field(\"Valor de exemplo 1\", default=0) # Sempre será um numero inteiro, com valor padrão 0 caso o Gemini não encontre informações sobre esse campo no documento\n",
" valor_2: typing.Optional[float] = pydantic.Field(\"Valor de exemplo 2\", default=None) # Pode ser um numero ou o valor especial \"None\", com valor padrão None caso o Gemini não encontre informações sobre esse campo no documento\n",
" valor_3: int = pydantic.Field(\"Valor de exemplo 3\") # O Gemini *sempre* vai ter que adivinhar um valor para esse campo, mesmo caso não esteja presente no documento\n",
" valor_4: typing.Optional[int] = pydantic.Field(\"Valor de exemplo 4\") # O Gemini *sempre* vai ter dar um valor para esse campo, mas ele pode retornar \"None\" caso não encontre.\n",
" \n",
"```\n",
"\n",
"É obrigatorio que todos os campos tenham descrições, pois é baseada nessa descrição que o Gemini vai saber o que extrair.\n",
"\n",
"> Cuidado: É possível que o Gemini retorne dados com errors caso você faça requisições confusas ou muito dificeis. Se isso acontecer, esse programa vai tentar filtrar para só incluir dados sem errors no resultado final."
],
"metadata": {
"id": "xTfQBnZ9xLl0"
}
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"cellView": "form",
"id": "LPLx6Xubt6fW",
"outputId": "33844f59-23c2-46ab-8bb4-e9860bbb81d8",
"collapsed": true
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Requirement already satisfied: google-generativeai in /usr/local/lib/python3.10/dist-packages (0.5.2)\n",
"Requirement already satisfied: google-ai-generativelanguage==0.6.2 in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (0.6.2)\n",
"Requirement already satisfied: google-api-core in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (2.11.1)\n",
"Requirement already satisfied: google-api-python-client in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (2.84.0)\n",
"Requirement already satisfied: google-auth>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (2.27.0)\n",
"Requirement already satisfied: protobuf in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (3.20.3)\n",
"Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (2.7.1)\n",
"Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (4.66.4)\n",
"Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from google-generativeai) (4.11.0)\n",
"Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/dist-packages (from google-ai-generativelanguage==0.6.2->google-generativeai) (1.23.0)\n",
"Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai) (5.3.3)\n",
"Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai) (0.4.0)\n",
"Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth>=2.15.0->google-generativeai) (4.9)\n",
"Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai) (1.63.0)\n",
"Requirement already satisfied: requests<3.0.0.dev0,>=2.18.0 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai) (2.31.0)\n",
"Requirement already satisfied: httplib2<1dev,>=0.15.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai) (0.22.0)\n",
"Requirement already satisfied: google-auth-httplib2>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai) (0.1.1)\n",
"Requirement already satisfied: uritemplate<5,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from google-api-python-client->google-generativeai) (4.1.1)\n",
"Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from pydantic->google-generativeai) (0.6.0)\n",
"Requirement already satisfied: pydantic-core==2.18.2 in /usr/local/lib/python3.10/dist-packages (from pydantic->google-generativeai) (2.18.2)\n",
"Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai) (1.63.0)\n",
"Requirement already satisfied: grpcio-status<2.0.dev0,>=1.33.2 in /usr/local/lib/python3.10/dist-packages (from google-api-core->google-generativeai) (1.48.2)\n",
"Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/local/lib/python3.10/dist-packages (from httplib2<1dev,>=0.15.0->google-api-python-client->google-generativeai) (3.1.2)\n",
"Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth>=2.15.0->google-generativeai) (0.6.0)\n",
"Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core->google-generativeai) (3.3.2)\n",
"Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core->google-generativeai) (3.7)\n",
"Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core->google-generativeai) (2.0.7)\n",
"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0.dev0,>=2.18.0->google-api-core->google-generativeai) (2024.2.2)\n"
]
}
],
"source": [
"#@title Baixar dependencias e importar bibliotecas necessárias\n",
"!pip install -U google-generativeai\n",
"\n",
"import re\n",
"import typing\n",
"import pydantic\n",
"import time\n",
"from pydantic.fields import FieldInfo\n",
"from google.colab import files, userdata\n",
"import google.ai.generativelanguage as glm\n",
"import google.generativeai as genai"
]
},
{
"cell_type": "code",
"source": [
"#@title Variaveis fixas\n",
"\"\"\"Para essas variaveis, os valores padrões devem funcionar, mas você pode mudar o que quiser a partir de configure()\"\"\"\n",
"\n",
"modelos_registrados: dict[str, typing.Type[pydantic.BaseModel]] = {} # Pydantic Model Name => Model\n",
"funções_registradas: dict[str, list[glm.FunctionDeclaration]] = {} # Pydantic Model Name => Function\n",
"\n",
"genai.configure(api_key=userdata.get('GOOGLE_API_KEY'))\n",
"\n",
"gemini = genai.GenerativeModel(\n",
" \"gemini-1.0-pro\",\n",
" generation_config=genai.GenerationConfig(temperature=0.4),\n",
")\n",
"\n",
"TOOL_CONFIG = {\"function_calling_config\": \"ANY\"} # Recomendo fortemente não mexer nessa linha\n",
"FILE_ENCODING = \"UTF-8\" # Pode trocar se souber o que está fazendo\n"
],
"metadata": {
"id": "iQtymD9Pxzp1",
"cellView": "form"
},
"execution_count": 2,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title Variaveis Especificas (Requer Mudanças)\n",
"\"\"\"Você deve ter que ajustar os valores de PROMPT_TEMPLATE e FUNÇÕES_DE_MODELO dependendo dos seus dados.\"\"\"\n",
"\n",
"# Lembre-se de usar nomes em Inglês para as funções de modelo\n",
"FUNÇÕES_DE_MODELO = [\n",
" (\"register_{}\", \"Register a new\"),\n",
"]\n",
"\n",
"PROMPT_TEMPLATE = \"\"\"Dado o seguinte documento, \\\n",
"identifique e registre todos os {modelo_de_dados}s importantes, \\\n",
"incluindo todos os dados relevantes sobre cada um deles.\n",
"---\n",
"{conteudo_do_arquivo}\n",
"\"\"\""
],
"metadata": {
"id": "lrnT4LpPZUN4"
},
"execution_count": 3,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title Codigo responsavel por converter os Modelos de Dados\n",
"\"\"\"Most technical Cell of the entire Notebook - Formats the pydantic models as Gemini Functions.\n",
"Do not worry about understanding it, just run and move on to the next Cell\n",
"\"\"\"\n",
"\n",
"class SchemaConfig(typing.TypedDict):\n",
" description: str\n",
" nullable: bool\n",
"\n",
"\n",
"# def _format_type(annotation: type, **kwargs: typing.Unpack[SchemaConfig]) -> glm.Schema: # Typing for (3.11+? 3.12+?)\n",
"def _format_type(annotation: type, **kwargs) -> glm.Schema:\n",
" mapping = {\n",
" str: glm.Schema(type=glm.Type.STRING, **kwargs),\n",
" bool: glm.Schema(type=glm.Type.BOOLEAN, **kwargs),\n",
" int: glm.Schema(type=glm.Type.INTEGER, **kwargs),\n",
" float: glm.Schema(type=glm.Type.NUMBER, **kwargs),\n",
" list[str]: glm.Schema(type=glm.Type.ARRAY, items=glm.Schema(type=glm.Type.STRING), **kwargs),\n",
" list[bool]: glm.Schema(type=glm.Type.ARRAY, items=glm.Schema(type=glm.Type.BOOLEAN), **kwargs),\n",
" list[int]: glm.Schema(type=glm.Type.ARRAY, items=glm.Schema(type=glm.Type.INTEGER), **kwargs),\n",
" list[float]: glm.Schema(type=glm.Type.ARRAY, items=glm.Schema(type=glm.Type.NUMBER), **kwargs),\n",
" }\n",
" return mapping[annotation]\n",
"\n",
"\n",
"def format_parameter(field: FieldInfo) -> glm.Schema:\n",
" assert field.annotation is not None, f\"Field {field} missing annotation\"\n",
" assert field.description is not None, f\"Field {field} missing description\"\n",
" _optional_mappings = {typing.Optional[ann]: ann for type_ in (str, bool, int, float) for ann in (type_, list[type_])}\n",
" annotation = field.annotation\n",
" nullable = annotation in _optional_mappings\n",
" if nullable:\n",
" annotation = _optional_mappings[annotation]\n",
" config = SchemaConfig(description=field.description, nullable=nullable)\n",
" return _format_type(annotation, **config)\n",
"\n",
"\n",
"# def register_model[ModelType: typing.Type[pydantic.BaseModel]](model: ModelType) -> ModelType: # 3.12+\n",
"def register_model(model):\n",
" \"\"\"Register the Model and declare Tool functions to manage instances of it.\"\"\"\n",
" snake_case_name = '_'.join(re.findall(r\"([A-Z][a-z0-9\\_]+)\", model.__name__)).lower()\n",
"\n",
" modelos_registrados[model.__name__] = model\n",
"\n",
" parameters: dict[str, glm.Schema] = {}\n",
" required: list[str] = []\n",
" for key, field in model.model_fields.items():\n",
" parameters[key] = format_parameter(field)\n",
" if field.is_required():\n",
" required.append(key)\n",
"\n",
" model_schema = glm.Schema(\n",
" type=glm.Type.OBJECT,\n",
" nullable=False,\n",
" properties=parameters,\n",
" required=required,\n",
" )\n",
" functions = {\n",
" model.__name__: [glm.FunctionDeclaration(\n",
" name=function_name.format(snake_case_name),\n",
" description=f\"{function_description} {model.__doc__}\",\n",
" parameters=model_schema,\n",
" )\n",
" for function_name, function_description in FUNÇÕES_DE_MODELO]\n",
" }\n",
" funções_registradas.update(functions)\n",
" return model\n",
"\n",
"def clear_models_and_functions() -> None:\n",
" \"Unregister all models and functions previously declared by `register_model`\"\n",
" modelos_registrados.clear()\n",
" funções_registradas.clear()\n",
"\n",
"def collect_models() -> dict[str, typing.Type[pydantic.BaseModel]]:\n",
" \"Lists all models registered as `@register_model`s\"\n",
" return modelos_registrados.copy()\n",
"\n",
"def collect_tools(model_name: typing.Optional[str] = None) -> glm.Tool:\n",
" \"Collects functions declared by `@register_model`, optionally filtering by the relevant model\"\n",
" if model_name:\n",
" functions = funções_registradas[model_name]\n",
" else:\n",
" functions = [func for sublist in funções_registradas.values() for func in sublist]\n",
" return glm.Tool(function_declarations=functions)\n",
"\n",
"def get_model_from_function_name(function_name: str) -> typing.Type[pydantic.BaseModel]:\n",
" owners = {function.name: owner for owner, functions in funções_registradas.items() for function in functions}\n",
" return modelos_registrados[owners[function_name]]\n",
"\n",
"\n",
"registrar_modelo = register_model"
],
"metadata": {
"cellView": "form",
"id": "1C2IQpCmwb3t"
},
"execution_count": 4,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title Definir os Modelos de Dados (Requer Mudanças)\n",
"\"\"\"Você precisa definir os seus modelos,\n",
"que correspondem aos dados que você quer extrair dos arquivos enviados.\"\"\"\n",
"\n",
"# Lembre-se de usar nomes e descrições em Inglês para os Modelos,\n",
"# mas sinta-se livra para usar Português para as descrições dos campos\n",
"\n",
"@registrar_modelo\n",
"class Character(pydantic.BaseModel):\n",
" \"important character present in the book\"\n",
" nome: str = pydantic.Field(default=\"Desconhecido\", description=\"O nome dessa personagem.\")\n",
" descricao: str = pydantic.Field(description=\"Descricao curta sobre essa personagem.\")\n",
" hobbies: list[str] = pydantic.Field(description=\"Lista de hobbies dessa personagem\", default=[])\n",
" idade: typing.Optional[int] = pydantic.Field(description=\"Idade em anos dessa personagem, caso conhecida\", default=None)\n"
],
"metadata": {
"id": "lR3hkjsOwcn0"
},
"execution_count": 5,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title Codigo para chamar o API do Gemini e formatar a saida\n",
"\"\"\"Define the actual python function that calls Gemini for a given (Model X File) combination\"\"\"\n",
"\n",
"def run_for_file(model, file_name: str, file_content: str, data_model: type[pydantic.BaseModel]):\n",
" assert \"conteudo_do_arquivo\" in PROMPT_TEMPLATE, \"\"\"ERRO: Você parece ter se esquecido de incluir o `{conteudo_do_arquivo}` no PROMPT_TEMPLATE\n",
" O Gemini não teria acesso aos conteudos dessa forma. Se isso for intencional, apague essa linha de assert\"\"\"\n",
"\n",
" prompt = PROMPT_TEMPLATE.format(conteudo_do_arquivo=file_content, modelo_de_dados=data_model.__name__)\n",
"\n",
" assert '{' not in prompt, \"A prompt não deveria conter nenhum '{'\"\n",
"\n",
" model_tools = collect_tools(data_model.__name__)\n",
" # print(model_tools)\n",
" response = model.generate_content(\n",
" prompt,\n",
" tools=model_tools,\n",
" tool_config=TOOL_CONFIG,\n",
" )\n",
"\n",
" # Just logging / for assisting in in debugging\n",
" for i, candidate in enumerate(response.candidates, 1):\n",
" for j, part in enumerate(candidate.content.parts, 1):\n",
" print(f\"Arquivo {file_name!r} - {data_model.__name__} Data\"\n",
" f\" Candidato {i}/{len(response.candidates)} Part {j}/{len(candidate.content.parts)}:\")\n",
" if part.text:\n",
" # *Should* always be empty. In practice it can not be thanks to bugs on their side.\n",
" print(\"- !AVISO: Não deveriam haver saidas de texto. Esses dados não são formatados nem inclusos no arquivo de saida.\"\n",
" f\"\\nSaida de Texto: {part.text!r}\")\n",
" if part.function_call.name:\n",
" _function_args = dict(part.function_call.args.items()) if part.function_call.args else None\n",
" print(f\"- Saida de Função: {part.function_call.name!r}({_function_args!r})\")\n",
"\n",
"\n",
" result = {}\n",
"\n",
" for part in (part for candidate in response.candidates for part in candidate.content.parts):\n",
" call = part.function_call\n",
" if call.name == \"\" or call.args is None:\n",
" continue\n",
" arguments = dict(call.args.items())\n",
"\n",
" try:\n",
" model_type = get_model_from_function_name(call.name)\n",
" except KeyError:\n",
" print(f\"O modelo tentou chamar uma função que não existe: {call.name}. Pulando esse resultado.\")\n",
" continue\n",
" if model_type is not data_model:\n",
" print(f\"Esperava-se que o modelo fizesse uma operação com {data_model}, mas ele está tentado realizar uma operação em {model_type}. Pulando esse resultado\")\n",
" # ^ Extremely unlikely - we only give information about one model at a time,\n",
" # it would have to randomly pick a string correspondent to different existing model by coincidence...\n",
" continue\n",
"\n",
" try:\n",
" created_model = model_type.model_validate(arguments) # Feel free to change this logic a bit, e.g. specifying `strict=...`\n",
" except pydantic.ValidationError:\n",
" print(f\"Erro tentando carregar {model_type} com argumentos {arguments!r}. Pulando esse resultado. Tente simplificar o seu modelo.\")\n",
" continue\n",
" result.setdefault(model_type.__name__, []).append(created_model)\n",
" return result\n"
],
"metadata": {
"cellView": "form",
"id": "sBwOcPUOweVW"
},
"execution_count": 6,
"outputs": []
},
{
"cell_type": "code",
"source": [
"#@title Envie os seus arquivos - CUIDADO! Lembre-se de nunca enviar documentos confidenciais para o Gemini.\n",
"\n",
"\"\"\"\n",
"CUIDADO! Lembre-se de nunca enviar documentos confidenciais para o Gemini.\n",
"\n",
"Os dados enviados para o API Gratuido do Google são usados para treinar o modelo, o que pode os tornar efetivamente publicos.\n",
"\"\"\"\n",
"\n",
"uploaded = files.upload()\n",
"# uploaded = {\"file.txt\": open(\"file.txt\", 'rb').read()}\n",
"\n",
"resultado_completo = {}\n",
"for file_name, file_content in uploaded.items():\n",
" file_content = file_content.decode(FILE_ENCODING)\n",
" print(f\"Previa do Arquivo: {file_name}\\n--------------\\n{file_content[:100]}\\n--------------\")\n",
"\n",
" file_results: dict[str, list] = {}\n",
" for data_model in collect_models().values():\n",
" print(f\"Processando arquivo {file_name!r} para {data_model.__name__}\")\n",
" result = run_for_file(gemini, file_name=file_name, file_content=file_content, data_model=data_model)\n",
" print(result)\n",
" file_results.update(result)\n",
" print(\"Esperando por 4 segundos em consideração dos limites do API\")\n",
" time.sleep(4)\n",
" # Rate Limits em particular - Atualmente o limite é de 15 requisições por minuto\n",
"\n",
" resultado_completo[file_name] = file_results\n"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 291
},
"id": "ii5ihhHjwfkm",
"outputId": "a0bfe1a9-e394-4d82-a254-d6a7a6effa8c",
"cellView": "form"
},
"execution_count": 7,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.HTML object>"
],
"text/html": [
"\n",
" <input type=\"file\" id=\"files-b24c2e5c-d6ef-46d9-a1e5-97f7f1b7a99b\" name=\"files[]\" multiple disabled\n",
" style=\"border:none\" />\n",
" <output id=\"result-b24c2e5c-d6ef-46d9-a1e5-97f7f1b7a99b\">\n",
" Upload widget is only available when the cell has been executed in the\n",
" current browser session. Please rerun this cell to enable.\n",
" </output>\n",
" <script>// Copyright 2017 Google LLC\n",
"//\n",
"// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
"// you may not use this file except in compliance with the License.\n",
"// You may obtain a copy of the License at\n",
"//\n",
"// http://www.apache.org/licenses/LICENSE-2.0\n",
"//\n",
"// Unless required by applicable law or agreed to in writing, software\n",
"// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
"// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
"// See the License for the specific language governing permissions and\n",
"// limitations under the License.\n",
"\n",
"/**\n",
" * @fileoverview Helpers for google.colab Python module.\n",
" */\n",
"(function(scope) {\n",
"function span(text, styleAttributes = {}) {\n",
" const element = document.createElement('span');\n",
" element.textContent = text;\n",
" for (const key of Object.keys(styleAttributes)) {\n",
" element.style[key] = styleAttributes[key];\n",
" }\n",
" return element;\n",
"}\n",
"\n",
"// Max number of bytes which will be uploaded at a time.\n",
"const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
"\n",
"function _uploadFiles(inputId, outputId) {\n",
" const steps = uploadFilesStep(inputId, outputId);\n",
" const outputElement = document.getElementById(outputId);\n",
" // Cache steps on the outputElement to make it available for the next call\n",
" // to uploadFilesContinue from Python.\n",
" outputElement.steps = steps;\n",
"\n",
" return _uploadFilesContinue(outputId);\n",
"}\n",
"\n",
"// This is roughly an async generator (not supported in the browser yet),\n",
"// where there are multiple asynchronous steps and the Python side is going\n",
"// to poll for completion of each step.\n",
"// This uses a Promise to block the python side on completion of each step,\n",
"// then passes the result of the previous step as the input to the next step.\n",
"function _uploadFilesContinue(outputId) {\n",
" const outputElement = document.getElementById(outputId);\n",
" const steps = outputElement.steps;\n",
"\n",
" const next = steps.next(outputElement.lastPromiseValue);\n",
" return Promise.resolve(next.value.promise).then((value) => {\n",
" // Cache the last promise value to make it available to the next\n",
" // step of the generator.\n",
" outputElement.lastPromiseValue = value;\n",
" return next.value.response;\n",
" });\n",
"}\n",
"\n",
"/**\n",
" * Generator function which is called between each async step of the upload\n",
" * process.\n",
" * @param {string} inputId Element ID of the input file picker element.\n",
" * @param {string} outputId Element ID of the output display.\n",
" * @return {!Iterable<!Object>} Iterable of next steps.\n",
" */\n",
"function* uploadFilesStep(inputId, outputId) {\n",
" const inputElement = document.getElementById(inputId);\n",
" inputElement.disabled = false;\n",
"\n",
" const outputElement = document.getElementById(outputId);\n",
" outputElement.innerHTML = '';\n",
"\n",
" const pickedPromise = new Promise((resolve) => {\n",
" inputElement.addEventListener('change', (e) => {\n",
" resolve(e.target.files);\n",
" });\n",
" });\n",
"\n",
" const cancel = document.createElement('button');\n",
" inputElement.parentElement.appendChild(cancel);\n",
" cancel.textContent = 'Cancel upload';\n",
" const cancelPromise = new Promise((resolve) => {\n",
" cancel.onclick = () => {\n",
" resolve(null);\n",
" };\n",
" });\n",
"\n",
" // Wait for the user to pick the files.\n",
" const files = yield {\n",
" promise: Promise.race([pickedPromise, cancelPromise]),\n",
" response: {\n",
" action: 'starting',\n",
" }\n",
" };\n",
"\n",
" cancel.remove();\n",
"\n",
" // Disable the input element since further picks are not allowed.\n",
" inputElement.disabled = true;\n",
"\n",
" if (!files) {\n",
" return {\n",
" response: {\n",
" action: 'complete',\n",
" }\n",
" };\n",
" }\n",
"\n",
" for (const file of files) {\n",
" const li = document.createElement('li');\n",
" li.append(span(file.name, {fontWeight: 'bold'}));\n",
" li.append(span(\n",
" `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
" `last modified: ${\n",
" file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
" 'n/a'} - `));\n",
" const percent = span('0% done');\n",
" li.appendChild(percent);\n",
"\n",
" outputElement.appendChild(li);\n",
"\n",
" const fileDataPromise = new Promise((resolve) => {\n",
" const reader = new FileReader();\n",
" reader.onload = (e) => {\n",
" resolve(e.target.result);\n",
" };\n",
" reader.readAsArrayBuffer(file);\n",
" });\n",
" // Wait for the data to be ready.\n",
" let fileData = yield {\n",
" promise: fileDataPromise,\n",
" response: {\n",
" action: 'continue',\n",
" }\n",
" };\n",
"\n",
" // Use a chunked sending to avoid message size limits. See b/62115660.\n",
" let position = 0;\n",
" do {\n",
" const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
" const chunk = new Uint8Array(fileData, position, length);\n",
" position += length;\n",
"\n",
" const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
" yield {\n",
" response: {\n",
" action: 'append',\n",
" file: file.name,\n",
" data: base64,\n",
" },\n",
" };\n",
"\n",
" let percentDone = fileData.byteLength === 0 ?\n",
" 100 :\n",
" Math.round((position / fileData.byteLength) * 100);\n",
" percent.textContent = `${percentDone}% done`;\n",
"\n",
" } while (position < fileData.byteLength);\n",
" }\n",
"\n",
" // All done.\n",
" yield {\n",
" response: {\n",
" action: 'complete',\n",
" }\n",
" };\n",
"}\n",
"\n",
"scope.google = scope.google || {};\n",
"scope.google.colab = scope.google.colab || {};\n",
"scope.google.colab._files = {\n",
" _uploadFiles,\n",
" _uploadFilesContinue,\n",
"};\n",
"})(self);\n",
"</script> "
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"Saving exemplo.txt to exemplo.txt\n",
"Previa do Arquivo: exemplo.txt\n",
"--------------\n",
"Em um vilarejo tranquilo, aninhado entre montanhas verdejantes, viviam João e Maria, dois irmãos ins\n",
"--------------\n",
"Processando arquivo 'exemplo.txt' para Character\n",
"Arquivo 'exemplo.txt' - Character Data Candidato 1/1 Part 1/2:\n",
"- Saida de Função: 'register_character'({'idade': 12.0, 'nome': 'João', 'descricao': 'Rapaz aventureiro que explora florestas e esculpe em madeira.', 'hobbies': ['Explorar florestas', 'Esculpir em madeira']})\n",
"Arquivo 'exemplo.txt' - Character Data Candidato 1/1 Part 2/2:\n",
"- Saida de Função: 'register_character'({'idade': 10.0, 'descricao': 'Jovem sonhadora que tece histórias em tapetes e ama ler.', 'hobbies': ['Tecer histórias', 'Ler histórias antigas'], 'nome': 'Maria'})\n",
"{'Character': [Character(nome='João', descricao='Rapaz aventureiro que explora florestas e esculpe em madeira.', hobbies=['Explorar florestas', 'Esculpir em madeira'], idade=12), Character(nome='Maria', descricao='Jovem sonhadora que tece histórias em tapetes e ama ler.', hobbies=['Tecer histórias', 'Ler histórias antigas'], idade=10)]}\n",
"Esperando por 4 segundos em consideração dos limites do API\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"#@title Exporta o resultado\n",
"adapter = pydantic.TypeAdapter(dict[str, dict[str, list]])\n",
"\n",
"with open(\"results.json\", \"wb\") as out_file:\n",
" out_file.write(adapter.dump_json(resultado_completo, indent=4))\n",
"\n",
"files.download(\"results.json\")\n",
"\n",
"!cat results.json"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 446
},
"id": "ybmkmPqawgFF",
"outputId": "1495a329-a380-4969-c5e6-f25e7223edf6",
"cellView": "form"
},
"execution_count": 8,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"\n",
" async function download(id, filename, size) {\n",
" if (!google.colab.kernel.accessAllowed) {\n",
" return;\n",
" }\n",
" const div = document.createElement('div');\n",
" const label = document.createElement('label');\n",
" label.textContent = `Downloading \"${filename}\": `;\n",
" div.appendChild(label);\n",
" const progress = document.createElement('progress');\n",
" progress.max = size;\n",
" div.appendChild(progress);\n",
" document.body.appendChild(div);\n",
"\n",
" const buffers = [];\n",
" let downloaded = 0;\n",
"\n",
" const channel = await google.colab.kernel.comms.open(id);\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
"\n",
" for await (const message of channel.messages) {\n",
" // Send a message to notify the kernel that we're ready.\n",
" channel.send({})\n",
" if (message.buffers) {\n",
" for (const buffer of message.buffers) {\n",
" buffers.push(buffer);\n",
" downloaded += buffer.byteLength;\n",
" progress.value = downloaded;\n",
" }\n",
" }\n",
" }\n",
" const blob = new Blob(buffers, {type: 'application/binary'});\n",
" const a = document.createElement('a');\n",
" a.href = window.URL.createObjectURL(blob);\n",
" a.download = filename;\n",
" div.appendChild(a);\n",
" a.click();\n",
" div.remove();\n",
" }\n",
" "
]
},
"metadata": {}
},
{
"output_type": "display_data",
"data": {
"text/plain": [
"<IPython.core.display.Javascript object>"
],
"application/javascript": [
"download(\"download_3659cc87-eeca-4a96-aa47-106e5b2e4f78\", \"results.json\", 691)"
]
},
"metadata": {}
},
{
"output_type": "stream",
"name": "stdout",
"text": [
"{\n",
" \"exemplo.txt\": {\n",
" \"Character\": [\n",
" {\n",
" \"nome\": \"João\",\n",
" \"descricao\": \"Rapaz aventureiro que explora florestas e esculpe em madeira.\",\n",
" \"hobbies\": [\n",
" \"Explorar florestas\",\n",
" \"Esculpir em madeira\"\n",
" ],\n",
" \"idade\": 12\n",
" },\n",
" {\n",
" \"nome\": \"Maria\",\n",
" \"descricao\": \"Jovem sonhadora que tece histórias em tapetes e ama ler.\",\n",
" \"hobbies\": [\n",
" \"Tecer histórias\",\n",
" \"Ler histórias antigas\"\n",
" ],\n",
" \"idade\": 10\n",
" }\n",
" ]\n",
" }\n",
"}"
]
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment