AI Agent Project Structure: Folders, Files, and Responsibilities
Most AI agent tutorials jump straight to LangChain chains and tool calling. Production agents fail for a simpler reason first: nobody can find anything. Logic, prompts, API keys, and tests end up in one 800-line script. The next developer cannot tell where the execution loop lives versus where weather API wrappers live.
A clear project structure separates concerns before you scale — agent core, tools, models, prompts, API, tests, and data each get a home. This guide walks through a Python-style layout you can adapt to Node, .NET, or Go: what each folder does, which files belong there, and how the pieces connect at runtime.
Project Configuration at the Root
Start at the repository root with files every teammate and CI job expects. README.md documents setup, environment variables, how to run the agent locally, and example prompts — not marketing copy, but copy-paste commands that work on a fresh clone.
requirements.txt (or `pyproject.toml`) pins dependencies: LLM SDK, HTTP client, vector DB driver, test runner. `.env` holds API keys and model endpoints — load with python-dotenv or similar, and never commit it. `.gitignore` must exclude `.env`, `logs/`, `__pycache__/`, and local vector index files. `docker-compose.yml` is optional but valuable when your agent needs Redis for memory, Postgres for state, or a local Qdrant/Chroma instance — one command spins up dependencies for dev and integration tests.
Quick reference
- README: install steps, env var table, curl examples for the API.
- requirements.txt: pin major versions — LLM SDKs break often.
- .env.example committed; .env in .gitignore with a loud comment in README.
- docker-compose: agent app + Redis + vector DB for local parity with prod.
- Keep secrets out of prompts/ and config.py — inject at runtime only.
- Add a Makefile or scripts/ for common tasks: test, lint, run-api.
Remember this
Root config files set the contract — how to run, what to install, and where secrets live.
The src/ Layout — Six Modules
Put application code under `src/` so imports stay clean and tests can target packages explicitly. Six modules cover almost every agent codebase:
`agent/` — the brain: orchestration loop, executor, shared state, memory read/write. `tools/` — callable capabilities the LLM invokes: search, calculator, weather, internal APIs. `models/` — LLM client wrappers and embedding clients; swap OpenAI for Anthropic here without touching agent logic. `prompts/` — system and task prompt templates, versioned as code. `utils/` — logging, config parsing, shared helpers. `api/` — HTTP layer (FastAPI, Flask) that exposes the agent to other services or a UI.
This split mirrors how agents actually run: API receives a request → agent loads prompts and state → model decides → executor calls a tool → state updates → response returns.
Quick reference
- agent/: agent.py (loop), executor.py (tool dispatch), state.py, memory.py.
- tools/: one file per tool or tool family — keep schemas next to implementations.
- models/: llm_client.py, embeddings.py — centralize retries and token counting.
- prompts/: system_prompts.py, agent_prompts.py — no prompts buried in agent.py.
- utils/: config.py reads env; logger.py structured JSON for production.
- api/: routes.py + schemas.py (Pydantic) — validate input before the agent runs.
Remember this
src/ splits the agent loop, tools, models, prompts, utilities, and HTTP — each module has one job.
Inside agent/ — Loop, Executor, State, Memory
The `agent/` package is where product behavior lives. `agent.py` defines the main loop: read user goal and context, call the LLM with available tools, parse the model's tool-call or final answer, and decide whether to continue or stop. `executor.py` maps tool names to Python functions, validates arguments against schemas, catches errors, and returns structured results the model can read on the next turn.
`state.py` holds session-scoped data: current plan step, partial results, user id, conversation turn count. `memory.py` handles longer recall — write to vector store or Redis, retrieve relevant chunks before each LLM call. Keep state in-memory for a single request; use memory when sessions span hours or days.
Resist putting tool implementations inside `agent.py`. The loop should only know tool *interfaces* — names, JSON schemas, descriptions — registered from `tools/` at startup.
Quick reference
- Max-iteration guard in agent.py — prevent infinite tool loops.
- executor.py: timeout per tool, idempotent retries where safe.
- state.py: immutable updates or explicit dataclass — easier to debug.
- memory.py: separate short-term (Redis) from long-term (vector DB).
- Log every LLM call and tool invocation with correlation id.
- Unit-test executor with mocked tools — no LLM needed.
Remember this
agent.py orchestrates; executor.py runs tools; state.py tracks the session; memory.py recalls past context.
Tools, Models, and Prompts
`tools/` holds one module per capability. Each tool exports a name, description (for the LLM), JSON schema for arguments, and an async or sync handler. Example files from the reference layout: `search.py`, `calculator.py`, `weather.py`. Add tools by registering them in a central registry — do not hardcode a giant if/elif in the executor.
`models/` wraps vendor SDKs. `llm_client.py` sends chat completions, handles streaming, retries on 429, and normalizes responses. `embeddings.py` batches document embedding for RAG. When you switch from GPT-4 to Claude, you change one file.
`prompts/` stores templates as Python strings or Jinja files. `system_prompts.py` defines persona, safety rules, and output format. `agent_prompts.py` holds task-specific instructions — research mode vs coding mode. Version prompts in git; diff them in code review like any other logic.
Quick reference
- Tool schema: name, description, parameters — match OpenAI/Anthropic function format.
- Validate tool args with Pydantic before calling external APIs.
- models/: expose chat(messages, tools) and embed(texts) — hide vendor quirks.
- prompts/: inject {{context}} and {{tools}} at runtime — no string concat in agent.py.
- Keep prompt tokens measurable — log prompt size per request.
- Start with 3–5 reliable tools; add more only after evals pass.
Remember this
Tools are plugins, models are adapters, prompts are versioned templates — keep all three out of the agent loop file.
API Layer and Utilities
`api/` exposes the agent to the world. `routes.py` defines endpoints: `POST /chat`, `POST /run-task`, health checks. `schemas.py` uses Pydantic (or similar) to validate request bodies — session id, user message, optional file refs — before they reach `agent.run()`. Return structured JSON: answer, tool trace (optional), token usage, errors.
`utils/` is shared infrastructure. `config.py` loads settings from environment with sensible defaults. `logger.py` configures structured logging — JSON in production, readable text locally. `helpers.py` holds small pure functions: truncate context, format tool results, parse model JSON when the vendor returns markdown fences.
The API should stay thin. No business logic in route handlers beyond auth, rate limits, and calling the agent service. That keeps CLI and API able to share the same `agent/` package — `main.py` chooses the entry mode.
Quick reference
- FastAPI + uvicorn is a common default for Python agents.
- Return 422 on validation errors before spending LLM tokens.
- Optional: stream SSE from api/ while agent loop runs.
- utils/config.py: pydantic-settings or dataclass from env.
- Correlation id: pass from HTTP header through agent and tools.
- Rate limit at API boundary — not inside the LLM loop.
Remember this
api/ validates and exposes; utils/ configures and logs — neither should contain the agent loop.
Tests, Data, Logs, and main.py
`tests/` mirrors `src/`. `test_agent.py` mocks the LLM and asserts the loop stops after N steps or calls the right tool. `test_tools.py` hits tool handlers with fixture inputs — no network unless marked integration. `test_api.py` uses TestClient to verify schemas and status codes.
`data/` holds non-code assets: `examples.json` for demo inputs, `knowledge_base/` for RAG source documents, eval datasets for regression. Do not commit huge corpora — document download scripts instead.
`logs/` receives runtime log files in dev; use `.gitkeep` so the folder exists but stays empty in git. In production, ship logs to your aggregator — files are a local convenience.
`main.py` at the root is the entry point: parse CLI args (`--serve`, `--playground`, `--one-shot "query"`), load config, wire the tool registry, and start the agent or API server.
Quick reference
- pytest + pytest-asyncio for async agent and API tests.
- Fixture LLM responses — test tool selection without API spend.
- data/knowledge_base/: chunk and index in CI for RAG integration tests.
- logs/ in .gitignore except .gitkeep.
- main.py: if __name__ == "__main__" dispatches CLI vs uvicorn.
- Add eval harness later — data/ is the natural home for golden sets.
Remember this
tests/ prove behavior, data/ feeds RAG and evals, logs/ aid debugging, main.py wires how you run the project.
A maintainable AI agent project looks boring on purpose: config at the root, six packages under src/, tests that mirror production modules, and one main.py to run it all. You can adopt this layout on day one with three tools and a single LLM client — the folders stay empty until you need them, but the boundaries prevent the monolith script that every team regrets six months in.
Before adding LangGraph, CrewAI, or MCP, sketch your tree on paper. If you cannot point to where a new tool or prompt template lives, fix the structure first. Related on this site: What Is Agentic AI, Core Layers to Master Agentic AI, and Top AI GitHub Repositories for frameworks that fit this layout.
Related Articles
Explore this topic