# Use Xantly with LlamaIndex (Python)

LlamaIndex's OpenAI integration accepts an `api_base` argument on every `OpenAI(...)` constructor. Point it at Xantly and every RAG pipeline, query engine, and agent runs through smart routing + semantic cache + memory.

## Prerequisites

- Python 3.9+
- `pip install llama-index llama-index-llms-openai llama-index-embeddings-openai`
- A Xantly API key, [create one](/dashboard/api-keys)

## Setup

```python
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

print(llm.complete("What is LlamaIndex?").text)
```

## Global settings (recommended)

Set the LLM once and every LlamaIndex module picks it up:

```python
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

Settings.embed_model = OpenAIEmbedding(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="openai/text-embedding-3-small",
)
```

## RAG pipeline

```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)
Settings.embed_model = OpenAIEmbedding(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="openai/text-embedding-3-small",
)

docs = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(docs)

query_engine = index.as_query_engine()
resp = query_engine.query("What does this codebase do?")
print(resp)
```

## Agent

```python
from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool
from llama_index.llms.openai import OpenAI

def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

tools = [FunctionTool.from_defaults(fn=multiply)]

llm = OpenAI(
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="anthropic/claude-sonnet-4.6",  # strong tool-calling
)

agent = ReActAgent.from_tools(tools, llm=llm, verbose=True)
print(agent.chat("What is 17 * 23?"))
```

## Chat engine

```python
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core import VectorStoreIndex

memory = ChatMemoryBuffer.from_defaults(token_limit=3900)
chat_engine = index.as_chat_engine(chat_mode="context", memory=memory)
print(chat_engine.chat("Summarize the key points."))
```

## Streaming

```python
for token in llm.stream_complete("Stream a haiku."):
    print(token.delta, end="", flush=True)
```

## Model choice

| Model ID | When |
|---|---|
| `xantly/auto-quality` | Query engines + RAG default. |
| `xantly/auto-value` | High-volume RAG where quality tolerance is higher. |
| `anthropic/claude-sonnet-4.6` | Agents, reliable tool-calling. |
| `openai/gpt-5.4` | Classic LlamaIndex flows. |
| `openai/text-embedding-3-small` | Fast, cheap embeddings. |
| `openai/text-embedding-3-large` | Higher-quality embeddings for dense retrieval. |

## Verify

```python
resp = llm.complete("say pong")
print(resp.text)
```

Open your Xantly dashboard, the call should be logged with routed-to model, cache status, and USD cost.

## What you get

- **RAG with smart routing.** The query step goes to `auto-quality` via BaRP; embeddings route to the cheapest compliant embedding model.
- **Semantic cache on RAG queries.** Users asking similar questions about the same corpus hit cache at near-zero cost.
- **Memory cascade.** LlamaIndex's `ChatMemoryBuffer` handles session-local context; Xantly's L3 memory handles cross-session project knowledge.
- **Waterfall fallback.** Long RAG pipelines don't die mid-query on provider outages.
- **Multi-modal** via `OpenAIMultiModal` with `api_base`, image inputs route through Xantly.

## Gotchas

**`api_base` not `base_url`.** LlamaIndex uses `api_base` (the OpenAI v0 legacy name, kept for compatibility). Both `OpenAI` and `OpenAIEmbedding` accept it.

**`/v1` required.** Include it, LlamaIndex doesn't auto-append.

**`Settings` is global.** If you set `Settings.llm`, it applies to every module in the process. Override per call with the `llm=` kwarg on query engines/agents.

**Async variants.** Use `aquery(...)`, `acomplete(...)`, etc. on LlamaIndex async APIs, the underlying `openai` client handles async transparently.

**Embedding dimensions.** `text-embedding-3-small` defaults to 1536 dims. Set `dimensions=512` to shrink for cost savings, Xantly respects the parameter.

## Next steps

- [LangChain (Python)](/docs/use-with-langchain-python), the other big RAG framework.
- [OpenAI SDK (Python)](/docs/use-with-openai-sdk-python), lower-level control.
- [Embeddings API](/docs/embeddings), Xantly's embedding endpoint details.
- [Multi-Agent Orchestration](/docs/multi-agent-orchestration), Xantly-native chaining.