Use Xantly with LlamaIndex

LlamaIndex's OpenAI integration accepts an api_base argument on every OpenAI(...) constructor. Point it at Xantly and every RAG pipeline, query engine, and agent runs through smart routing + semantic cache + memory.

Prerequisites

• Python 3.9+
• pip install llama-index llama-index-llms-openai llama-index-embeddings-openai
• A Xantly API key, create one

Setup

from llama_index.llms.openai import OpenAI

llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

print(llm.complete("What is LlamaIndex?").text)

Global settings (recommended)

Set the LLM once and every LlamaIndex module picks it up:

from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

Settings.embed_model = OpenAIEmbedding(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="openai/text-embedding-3-small",
)

RAG pipeline

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)
Settings.embed_model = OpenAIEmbedding(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="openai/text-embedding-3-small",
)

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)

query_engine = index.as_query_engine()
resp = query_engine.query("What does this codebase do?")
print(resp)

Agent

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI

def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

tools = [FunctionTool.from_defaults(fn=multiply)]

llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="anthropic/claude-sonnet-4.6",  # strong tool-calling
)

agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)
print(agent.chat("What is 17 * 23?"))

Chat engine

from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core import VectorStoreIndex

memory = ChatMemoryBuffer.from_defaults(token_limit=3900)
chat_engine = index.as_chat_engine(chat_mode="context", memory=memory)
print(chat_engine.chat("Summarize the key points."))

Streaming

for token in llm.stream_complete("Stream a haiku."):
    print(token.delta, end="", flush=True)

Model choice

Verify

resp = llm.complete("say pong")
print(resp.text)

Open your Xantly dashboard, the call should be logged with routed-to model, cache status, and USD cost.

What you get

• RAG with smart routing. The query step goes to auto-quality via BaRP; embeddings route to the cheapest compliant embedding model.
• Semantic cache on RAG queries. Users asking similar questions about the same corpus hit cache at near-zero cost.
• Memory cascade. LlamaIndex's ChatMemoryBuffer handles session-local context; Xantly's L3 memory handles cross-session project knowledge.
• Waterfall fallback. Long RAG pipelines don't die mid-query on provider outages.
• Multi-modal via OpenAIMultiModal with api_base, image inputs route through Xantly.

Gotchas

api_base not base_url. LlamaIndex uses api_base (the OpenAI v0 legacy name, kept for compatibility). Both OpenAI and OpenAIEmbedding accept it.

/v1 required. Include it, LlamaIndex doesn't auto-append.

Settings is global. If you set Settings.llm, it applies to every module in the process. Override per call with the llm= kwarg on query engines/agents.

Async variants. Use aquery(...), acomplete(...), etc. on LlamaIndex async APIs, the underlying openai client handles async transparently.

Embedding dimensions. text-embedding-3-small defaults to 1536 dims. Set dimensions=512 to shrink for cost savings, Xantly respects the parameter.

Next steps

• LangChain (Python), the other big RAG framework.
• OpenAI SDK (Python), lower-level control.
• Embeddings API, Xantly's embedding endpoint details.
• Multi-Agent Orchestration, Xantly-native chaining.