How to build a custom AI chatbot in 2026

Building a custom AI chatbot no longer requires a machine learning team. The combination of a frontier model API, a vector database, and a few hundred lines of Python gets you an assistant that answers questions from your own documents. The whole stack can run in a weekend if you already code.

The approach below uses retrieval-augmented generation, or RAG, which grounds the model's answers in your files rather than its training data. That is the technique that separates a useful internal tool from a generic chatbot that invents answers.

Step 1: Choose a model API

Pick a model based on your content. Long documents favour Claude, which handles large context windows reliably. High-volume, low-latency work favours GPT 6.1. Mixed media content favours Gemini 4 Argon. Our model benchmark comparison covers the trade-offs in detail.

Create an API key in the provider's dashboard. Keep the key in an environment variable, never in source code, and set a monthly spend limit before you start. Our API key security guide covers the setup.

Step 2: Chunk and embed your documents

Split your documents into chunks of 500 to 1,000 words, with some overlap between chunks so no sentence is cut in half. Pass each chunk through an embedding model, which converts text into a list of numbers that captures meaning. Store the embeddings in a vector database such as Chroma, Pinecone, or pgvector.

The chunk size matters more than the database choice. Chunks that are too small lose context, and chunks that are too large dilute the relevant passage. Start at 700 words and adjust based on answer quality.

Step 3: Retrieve and generate

When a user asks a question, embed the question the same way, then find the five most similar chunks in your database. Pass those chunks to the model with a prompt that instructs it to answer only from the provided context.

The prompt is where most custom bots fail. A weak prompt lets the model answer from memory, which reintroduces hallucinations. A strong prompt says: answer from the context, quote the source chunk, and say when the context does not contain the answer.

Step 4: Add guardrails and logging

Log every question and every retrieved chunk. The logs show where the bot fails, which is how you improve the chunking and the prompt. Add a simple feedback button so users can flag wrong answers, and review flagged answers weekly.

Cost control comes next. Cache frequent questions, set a token budget per session, and route simple questions to a cheaper model. A well-tuned RAG bot answers most questions from a handful of chunks, which keeps the token bill in single-digit dollars per month for small document sets.

Testing before launch

Test the bot against a set of real questions before exposing it to users. Write down the twenty questions your audience actually asks, run them through the bot, and check the answers against the source documents. Fix the chunking or the prompt for every wrong answer.

The test set is also your regression suite. When you change the model or the prompt, rerun the questions and compare the answers. A bot that worked last week and fails today is a sign that something changed upstream.

Cost breakdown

A small RAG bot serving a few hundred users costs between AU$5 and AU$30 per month in API fees, depending on the model and the traffic. The vector database is the other line item, and the free tiers of Chroma and pgvector cover most small deployments.

The OpenAI and Anthropic pricing pages publish the token rates, and our model benchmark comparison covers the cost trade-offs.

Step 5: Deploy

Deploy the bot behind a simple web interface or a Slack integration. Start with a single document set and a small group of testers before opening it up. ## Security and privacy

A custom chatbot handles your documents, so the security posture matters. Run the bot behind authentication, limit the document set to what the audience needs, and log access. The vector database stores embeddings, not the raw documents, which limits the exposure if the store is breached.

The Australian Cyber Security Centre publishes guidance on securing AI applications, and the OAIC privacy guidance applies when the documents contain personal information.

The build order

The build order is: model API, then embeddings, then the vector store, then the retrieval logic, then the interface. Each layer is testable on its own, and the failure points are easier to isolate when the layers are separate.

The OpenAI and Anthropic developer docs both publish working RAG examples you can adapt, and the Chroma documentation covers the vector store setup.

The fastest path to a useful bot is a narrow one. Pick one document set, one audience, and one interface. Expand only after the bot answers that audience's questions accurately. Our student AI tools guide covers the consumer side of the market if you are choosing rather than building.