FitDesk AI – Agentic Fitness Coach
An agentic AI fitness coach built in Flutter, where the AI is the product rather than a feature. The agent has structured access to the user's training data, answers from real data, takes real actions, and streams its tool calls in real time.
FitDesk AI is a Flutter fitness coaching app where an AI agent sits at the center of the experience. The agent has structured access to the user's complete training data: workout history, nutrition logs, goals, and schedule. It answers questions from real data, takes real actions on the user's behalf (logging meals, rescheduling workouts), and surfaces proactive insights — all with the architecture built around the agent rather than bolted onto an existing screen.
Most "AI-powered" apps ship a ChatGPT button in the corner and call it AI integration. FitDesk AI was built to prove what proper AI integration in a mobile app actually looks like, combining production Flutter engineering with hands-on agentic AI work in a single 14-day solo build.
The problem it solves is that fitness apps collect enormous amounts of user data, but most of it sits in charts and logs that users never dig through. The insight a user actually wants ("am I making progress?", "should I rest today?", "did I eat enough protein this week?") is buried under taps and tabs. A generic chatbot can't solve this because it has no access to the user's real data — it either gives generic advice or invents numbers. The challenge was to build an AI that answers from the user's actual training history, takes meaningful actions, and never hallucinates data.
The agent supports conversational coaching grounded in real data — ask "which day did I lift the most?" and it calls a tool, queries the database, and answers with the exact session and total weight, no guessing. It is action-taking, not just answering — say "I had pizza for lunch" and it logs the meal and comments on how it fits the recent nutrition pattern; say "move Thursday's chest day to Saturday" and it actually reschedules the workout. It reasons across multiple tools — ask "what should I focus on next week?" and it calls consistency-by-weekday, strength-gain-percent, and workout-history tools before synthesizing an answer. The dashboard greets the user with a specific, data-backed observation generated by the agent, not a generic "welcome back".
Architecturally the system is two tiers: a Flutter frontend (Riverpod, Dio, feature-first layered architecture) and a Python FastAPI backend that runs the AI pipeline. The chat pipeline runs as a deterministic staged sequence rather than handing everything to the LLM: input refinement extracts hints like time windows and thresholds; retrieval embeds the message with nomic-embed-text and queries Qdrant for relevant history; a deterministic regex intent router (no LLM) classifies the query for predictable behavior; tool filtering passes the agent only the tools allowed for that intent; a LangGraph ReAct agent with a recursion guard executes, emitting tool-call and token events as they happen; finalization streams the assembled response and a completion marker.
The key engineering decisions are what separate this from a typical LangChain tutorial build. Deterministic intent routing runs before the LLM reasons — most agent stacks let the model pick tools freely, which causes drift (using nutrition tools for a lifting question, or inventing data); FitDesk constrains the tool space first. Intent-scoped tool allow-lists mean each intent maps to an explicit set of tools, and the LLM literally cannot call tools outside the current list — defense in depth: deterministic routing first, model reasoning second. Real-time tool-call streaming via SSE emits four event types (token, tool_call, message, done) so users see the agent working in real time ("Tool call: get_workout_history") instead of staring at a black-box spinner — observability becomes a user-facing feature. The entire AI pipeline runs locally on Ollama (llama3.2 for reasoning, nomic-embed-text for embeddings) with Qdrant for retrieval, so there is no external LLM dependency in production: no API bills, no vendor lock-in, no rate limits.
Screenshots
What was hard → How I solved it
Preventing the agent from hallucinating fitness data or inventing numbers when the user asked data-grounded questions
Built a deterministic regex-based intent router that classifies queries before the LLM reasons, constraining the tool space up front
Stopping agent drift — keeping the model from picking the wrong tools (e.g. nutrition tools for a lifting question)
Implemented intent-scoped tool allow-lists so the LLM literally cannot call tools outside the current intent — defense in depth on top of routing
Grounding every answer in the user's real training history while still allowing the agent to take actions like logging meals and rescheduling workouts
Designed a constrained set of purpose-built tools (history, schedule, analytics, mutations) that return structured data the model interprets into plain language, never inventing numbers
Running the full AI pipeline locally with no external LLM dependency, avoiding API bills, vendor lock-in, and rate limits
Ran reasoning on Ollama (llama3.2) and embeddings on nomic-embed-text with Qdrant for retrieval, keeping the entire pipeline local-first
Giving users real-time visibility into the agent's reasoning instead of a black-box spinner
Streamed tool calls and tokens over SSE with four event types (token, tool_call, message, done) so users see "Tool call: get_workout_history" in real time
What it changed
Demonstrates the rare combination of production Flutter experience and hands-on agentic AI engineering in a single solo build
Proves that a full AI layer can be added to a Flutter app cleanly, without compromising mobile architecture or performance
Showcases a polished Flutter frontend that integrates properly with an agent backend, streams responses cleanly, and handles real tool calls
Serves as a production-minded agentic system with deliberate reliability engineering (deterministic routing, tool scoping, observability) rather than a thin LLM wrapper
Establishes a reusable pattern for action-taking AI assistants grounded in structured user data, not generic chat
