NekoAI — AI-Powered Desktop Pet

NekoAI is an open-source, AI-powered desktop pet that lives on your screen. It wanders around your windows, reacts to what you do, and when you need it — it thinks, answers, and helps, right there on your desktop. A love letter to 90s/00s desktop companions (Neko, eSheep, Shimeji) rebuilt with a modern stack and a real AI brain inside.

Key Features

Feature	Status
Animated sprite pets that roam your desktop	Live
8-direction cursor following & movement	Live
AI chat via animated speech bubble	Live
Persistent memory — name, projects, preferences	Live
Multi-provider AI (Claude, OpenAI, Gemini, Ollama local)	Live
Dynamic mood — energy changes with time and idle time	Live
Multiple pets — Classic Neko, Ghost, Shiba	Live
Proactive nudges ("coding 90 min — take a break!")	Live
NekoCapture — screenshots with metadata footer	v0.5
NekoAnnotate — full-screen drawing overlay	v0.6

Architecture

src-tauri/ (Rust backend)
├── lib.rs              — App setup, tray, Tauri commands
├── desktop_monitor.rs  — Win32 APIs: active window, idle time
└── storage.rs          — SQLite: conversation history, user facts

src/ (TypeScript/React frontend)
├── ai/                 — Provider factory, memory, fact extraction
├── components/         — SpeechBubble, SettingsPanel, ContextMenu
├── hooks/              — usePetMovement, useMoodEngine, useDesktopContext
└── pets/               — PetRenderer, PetBrain, sprite loader

AI Memory System

NekoAI builds a persistent context for every conversation:

• Pet personality defined per-pet in pet.json via system_prompt
• User facts extracted from conversations and stored in SQLite (user_facts table)
• Conversation history — last 20 messages as rolling context
• Dynamic mood — energy/happiness/curiosity subtly influences AI tone

Technical Challenges Solved

Frameless window resizing on Windows: Creating a truly frameless window removes WS_THICKFRAME from the OS. JavaScript APIs cannot restore it at runtime. A Rust-side Tauri command calls window.set_size() directly, bypassing the JS API limitation. This allows the speech bubble and settings panel to expand/collapse dynamically.

Pixel-perfect sprite scaling: All pet sizes are integer multiples of the native 32px sprite (S=32, M=64, L=96, XL=128) to prevent anti-aliasing artifacts on pixel art.

Privacy First

• No backend server — all data stays on the user's machine
• Only outbound calls are the AI API calls the user configures
• Supports fully local AI via Ollama (100% offline, zero API costs)

Supported AI Providers

Provider	Models	Requires
Anthropic	Claude Haiku, Sonnet	API Key
OpenAI	GPT-4o mini, GPT-4o	API Key
Google	Gemini 1.5/2.0 Flash	API Key
Ollama	Llama 3, Mistral, Phi-3	Local install