Use Xantly with LangChain (Python)

LangChain's ChatOpenAI and ChatAnthropic both accept custom base URLs. Point them at Xantly and every chain, agent, and RAG pipeline gets BaRP routing, semantic cache, and memory underneath, with zero LangChain-specific code changes.

Prerequisites

• Python 3.9+
• pip install langchain langchain-openai langchain-anthropic
• A Xantly API key, create one

Setup (OpenAI-compatible)

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

print(llm.invoke("What is LangChain?").content)

Setup (Anthropic-compatible)

from langchain_anthropic import ChatAnthropic

llm = ChatAnthropic(
    base_url="https://api.xantly.com",  # bare, no /v1
    api_key="xantly_sk_...",
    model="anthropic/claude-sonnet-4.6",
    max_tokens=1024,
)

print(llm.invoke("What is LangChain?").content)

Pick whichever shape matches the rest of your stack, Xantly speaks both natively.

LCEL chain

from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a concise technical writer."),
    ("human", "Explain {topic} in one paragraph."),
])

chain = prompt | llm
print(chain.invoke({"topic": "vector databases"}).content)

Tool-calling agent

from langchain.agents import create_tool_calling_agent, AgentExecutor
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.tools import tool
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="anthropic/claude-sonnet-4.6",  # strong tool calling
)

@tool
def multiply(a: int, b: int) -> int:
    """Multiply two integers."""
    return a * b

prompt = ChatPromptTemplate.from_messages([
    ("system", "Use tools when they apply."),
    ("human", "{input}"),
    ("placeholder", "{agent_scratchpad}"),
])

agent = create_tool_calling_agent(llm, [multiply], prompt)
executor = AgentExecutor(agent=agent, tools=[multiply])
print(executor.invoke({"input": "What is 17 * 23?"})["output"])

RAG with Xantly embeddings

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Same base URL works for embeddings
embeddings = OpenAIEmbeddings(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="openai/text-embedding-3-small",
)

docs = ["Xantly is an AI gateway.", "LangChain is an LLM framework."]
splits = RecursiveCharacterTextSplitter().create_documents(docs)
store = FAISS.from_documents(splits, embeddings)

llm = ChatOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-quality",
)
# ... use store + llm in a retrieval chain

Streaming

for chunk in llm.stream("Stream a haiku."):
    print(chunk.content, end="", flush=True)

Model choice

Verify

llm = ChatOpenAI(
    base_url="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
    model="xantly/auto-speed",
)
print(llm.invoke("say pong").content)

Check your Xantly dashboard, the call should log with routed-to model, cache status, cost.

What you get

• Every LangChain primitive. Chains, agents, RAG, structured outputs, memory, all proxied through Xantly's routing.
• Waterfall fallback. Provider outages don't derail long-running agent executions.
• Semantic cache. LCEL .invoke() on repeat prompts hits cache at near-zero cost.
• Memory across chains. Xantly's L3 memory complements LangChain's ConversationBufferMemory, opt in per call via model_kwargs={"extra_headers": {"X-Xantly-Memory-Mode": "recall"}}.
• Cost per chain step. Every sub-call is logged individually.

Gotchas

base_url must match the shape. OpenAI-style: https://api.xantly.com/v1. Anthropic-style: https://api.xantly.com (bare). Wrong combination = 404s.

ChatOpenAI model field accepts provider-prefixed IDs. anthropic/claude-sonnet-4.6 through ChatOpenAI works, Xantly translates server-side.

LangChain's retry config. Set max_retries=0 on the LLM to avoid double-retries (Xantly waterfalls internally).

openai vs openai-compatible providers. LangChain treats them interchangeably here, ChatOpenAI is fine; you don't need a separate ChatOpenAICompatible class.

Next steps

• LangChain (TypeScript), langchain.js equivalent.
• LangGraph, graph-based agent runtime on top of LangChain.
• LlamaIndex (Python), the other big RAG framework.
• Multi-Agent Orchestration, Xantly's native agent chaining.