Overview
Android Agent is an on-device AI assistant that controls a physical Android phone through natural language. You type commands like "open YouTube and search for jazz piano tutorials." The agent figures out how to do it by reading the screen, deciding what to tap, typing text, and verifying the result.
Demo
How It Works
The app runs an LLM tool-calling loop in Flutter. At each step the model receives the current screen state, decides which action to take, and executes it through the Android Accessibility Service.
Architecture
The app is split into three layers. Flutter handles the UI and agent loop, native Kotlin bridges to Android APIs, and the OS provides the actual automation and capture services.
Eight tools are exposed to the model:
- run_adb — execute shell commands (tap, swipe, type, launch apps)
- launch_app — open an app by package name
- get_ui_dump — capture the screen's semantic XML tree
- get_screenshot — capture a visual screenshot for multimodal models
- search_packages / list_packages — discover installed apps by name
- current_app — identify the foreground activity
- type_in_field — locate an input field, focus it, type, and submit
Key Technical Decisions
No root required. Everything runs through Android's Accessibility Service. The agent reads uiautomator XML output and performs actions via performTap, performSwipe, and performTextInput. Exactly the way humans operate the device.
Budget-aware XML parsing. Raw UI dumps can exceed 500+ lines. The parser builds a tree, scores nodes by importance (clickable, focusable, text content, resource IDs), and serializes the most informative subset within a configurable line budget. This keeps token usage low without losing the elements that matter.
Visual + semantic understanding. When enabled, screenshots are sent alongside the XML so the model can reason about layout, colors, and visual context that the hierarchy alone doesn't convey. If screenshot permission hasn't been granted yet, the engine auto-prompts the user at the right moment instead of failing silently. The app also auto-detects when a model doesn't support images and falls back gracefully.
Session persistence. Conversations are stored locally with SharedPreferences and flutter_secure_storage (for API keys). Users can switch between up to 20 sessions, pick up where they left off, or start fresh.
Provider-agnostic. The API endpoint and model name are fully configurable. It works with OpenAI, Anthropic-compatible proxies, local Ollama instances, or any OpenAI-format API.
What Makes It Interesting
The core challenge is the observation-action loop under token constraints. Each screen state needs to be compressed enough to fit the model's context window, but detailed enough for the model to make correct decisions. The importance-scored XML pruning and selective screenshot capture are the key tricks that make this practical on real devices.
Requirements
- Android — API level 21+ (5.0 Lollipop or higher)
- Flutter SDK — 3.x with Dart 3.12+
- LLM API — Any OpenAI-compatible endpoint (OpenAI, Ollama, LM Studio, etc.)
No root access required. The app uses Android's Accessibility Service for all device interactions.