Google Summer of Code 2025 – Final Report

Organization: BeagleBoard.org

Project: BeagleMind – AI Assistant for BeagleBoard Documentation & Contributor Onboarding

Contributor: Fayez Zouari

Mentors: Aryan Nanda, Jason Kridner and Kumar Abhishek

---

Important Links

• Pre-GSoC Discussion Thread: BeagleMind Early Discussion
• GSoC Forum Thread: BeagleMind Conversational AI Assistant
• Discord BeagleMind Thread: BeagleBoard Discord – BeagleMind
• Project Introduction Video: YouTube – BeagleMind Intro
• CLI Repository: BeagleMind-CLI
• Platform Repository: BeagleMind-Platform
• Live Platform: mind.beagleboard.org
• CLI on PyPI: beaglemind-cli

---

Project Overview

The BeagleMind project introduces an AI-powered assistant designed to simplify the BeagleBoard.org contributor experience. Its goal is twofold:

1. Documentation Support – helping users navigate and query BeagleBoard documentation efficiently.
2. Contributor Onboarding & Troubleshooting – assisting new community members with step-by-step guidance and interactive help.

The assistant is accessible both via a CLI tool and a web platform:

• CLI Tool: Available on PyPI as beaglemind-cli for direct use on BeagleBoard hardware or any development environment.
• Web Platform: Hosted at mind.beagleboard.org, providing login with Google, conversation history, and an onboarding Wizard tool for guiding new contributors.

Forum thread for discussions and updates: BeagleMind Conversational AI Assistant – BeagleBoard.org Forum

---

Project Deployment

BeagleMind is composed of multiple components working together:

• API Layer: Powers both the CLI and platform, responsible for managing retrieval augmented generation (RAG).
• Frontend (Platform): Interactive web interface to chat with BeagleMind, manage projects, and use the Wizard onboarding tool.
• CLI Tool: Lightweight terminal client, enabling local or cloud inference and supporting OpenAI / Groq inference with tool calling capability.

Repositories & Links

• Platform Access: mind.beagleboard.org
• CLI (PyPI): pip install beaglemind-cli
• Platform Repo: BeagleMind-Platform
• CLI Repo: BeagleMind-CLI

Development Journey

1. Proof of Concept – Retrieval Augmented Generation (RAG)

The project started with a PoC hosted on Hugging Face Spaces (link to be added). This initial version validated the idea of combining BeagleBoard documentation with a conversational AI assistant through RAG.

2. BeagleMind CLI

The next milestone was building beaglemind-cli, available on PyPI. It allows contributors to interact with BeagleMind directly from their development environment.

Key Features:

• Easy installation via pip install beaglemind-cli.

• Runs directly on BeagleBoard hardware.

• Supports different inference modes:

  • Local inference (lightweight models for testing) using Ollama.
  • OpenAI / Groq inference with tool calling, enabling dynamic capabilities.

Start Chatting

Ask questions about the documentation:

# Simple question
beaglemind chat -p "How do I configure the BeagleY-AI board?"

# With specific model and backend
beaglemind chat -p "Show me GPIO examples" --backend groq --model llama-3.3-70b-versatile

# Interactive chatbot
beaglemind chat

Running Models Locally on BeagleBoard Hardware

One of the major goals of this project was to test the feasibility of running inference directly on BeagleBoard hardware, reducing dependency on cloud services.

What Worked?

• Running small LLMs (quantized versions of models) locally on CPU was possible using Ollama, though performance was limited.
• The CLI provided a consistent interface, so local vs. cloud inference was seamless for the user.

Challenges?

• Running larger LLMs on BeagleBoard’s accelerators (via TIDL) proved difficult.
• The TIDL library lacks full transformer layer support, making it incompatible with many modern LLM architectures.
• As a result, only CPU execution was stable, but this came with significant performance trade-offs (latency, memory usage).

Takeaway

While local inference worked in a limited fashion, cloud-backed inference (OpenAI / Groq) remains the most practical solution today. Future improvements in accelerator compatibility or lightweight LLM architectures could make truly local, real-time assistance feasible on BeagleBoard devices.

Repo: BeagleMind-CLI

Package: pip install beaglemind-cli

3. BeagleMind API & Platform

The CLI foundation evolved into a full platform with:

• Web App – interactive interface to communicate with BeagleMind.
• Context Retrieval – powered by RAG with a Milvus vector store backend.
• Conversation History – stored inside session storage for simplicity and privacy.
• Wizard Tool – step-by-step onboarding helper for new BeagleBoard contributors.

In addition, the platform provides a set of well-defined APIs for integration and automation:

• Retrieval

  • POST /api/retrieve – retrieve contextually relevant information.

• GitHub Ingestion

  • POST /api/ingest-data – ingest a GitHub repository for indexing.
  • GET /api/ingest-data/status – check ingestion status.

• Default

  • GET / – root endpoint.
  • GET /health – health check.

Live Platform: mind.beagleboard.org

Repo: BeagleMind-Platform

---

RAG Evaluations

To assess the quality of the retrieval augmented generation (RAG) pipeline, I ran systematic evaluations using 20 QA pairs derived from BeagleBoard documentation and contributor FAQs.

Methodology

• Used the RAGAS evaluation framework.

• Tested on four key aspects:

  • Answer Relevancy – is the answer on-topic?
  • Answer Correctness – does the response match the ground-truth reference?
  • Faithfulness – does the model stay grounded in retrieved context without hallucination?
  • Context Recall & Precision – does retrieval return the right passages while avoiding irrelevant noise?

Results

The following chart shows the average values of metrics from 20 QA pairs using gpt-4.1-mini as generator:

• Answer Relevancy: Very high (>0.85) – responses were strongly aligned with user queries.
• Answer Correctness: Lower (~0.40) – mainly due to long chunk sizes, where answers existed inside lengthy contexts. This made it harder for the evaluator to match the exact expected reference, even though the information was present.
• Faithfulness: ~0.72 – improved with “source citation” prompting, which grounded answers more consistently in the retrieved text.
• Context Recall: ~0.80 – retrieval effectively pulled in relevant documentation, especially after tuning top_k and adding reranking.
• Context Precision: ~0.58 – some extra irrelevant passages were retrieved, reducing precision.

To assess the quality of our RAG pipeline, I initially generated 120 QA pairs using GPT-4o with the websearch tool. While these pairs were grounded in reliable sources such as BeagleBoard PDF books, TI documentation, etc... Our retrieval corpus currently contains only the official BeagleBoard documentation and the forum. This created a mismatch, since many of the generated questions could not be answered from the available documents, leading to an evaluation that would unfairly penalize the system for limited corpus coverage rather than pipeline performance. To address this, I manually curated a subset of 20 QA pairs that can be supported by the existing documentation.

---

Next Steps & Future Work

• Improve documentation coverage and expand datasets for RAG.
• Add multi-modal support (e.g., images for troubleshooting hardware).
• Optimize inference for resource-constrained BeagleBoard devices.
• Integrate community feedback from the forum thread.

---

Acknowledgments

I would like to thank BeagleBoard.org, my mentors, and the wider community for their continuous support and feedback throughout GSoC. This project has been an exciting journey in combining AI with open hardware documentation and community engagement.