Use Xantly with DSPy

DSPy is Stanford's "programming-not-prompting" framework, you declare signatures and modules, DSPy optimizes the prompts. Its dspy.LM class accepts any OpenAI-compatible endpoint. Point it at Xantly and DSPy's optimizer runs + program execution both route through smart routing + cache + memory.

Prerequisites

• Python 3.9+
• pip install dspy
• A Xantly API key, create one

Setup

import dspy

lm = dspy.LM(
    "openai/xantly/auto-quality",  # protocol/model
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
dspy.configure(lm=lm)

That's the entire setup, now every DSPy module uses Xantly.

Signature + module

import dspy

class QA(dspy.Signature):
    """Answer the question concisely."""
    question = dspy.InputField()
    answer = dspy.OutputField()

qa = dspy.Predict(QA)
result = qa(question="What is DSPy?")
print(result.answer)

Chain-of-thought

cot = dspy.ChainOfThought(QA)
result = cot(question="What's 17 * 23?")
print(result.reasoning)
print(result.answer)

ReAct

def search(query: str) -> str:
    """Search the web."""
    return f"results for {query}"

agent = dspy.ReAct(QA, tools=[search])
result = agent(question="What is the capital of France?")
print(result.answer)

Optimizer (MIPROv2, BootstrapFewShot)

DSPy's differentiator, it optimizes your prompts against a train set:

from dspy.teleprompt import BootstrapFewShot

trainset = [
    dspy.Example(question="What is 2+2?", answer="4").with_inputs("question"),
    dspy.Example(question="What is 3*3?", answer="9").with_inputs("question"),
]

def accuracy(example, pred, trace=None):
    return pred.answer.strip() == example.answer

compiler = BootstrapFewShot(metric=accuracy, max_bootstrapped_demos=4)
optimized = compiler.compile(qa, trainset=trainset)

The compiler issues many LM calls during optimization. Xantly's semantic cache absorbs the repeated ones → 40-70% cost savings on big training runs.

Per-module models

Use different tiers for different modules:

fast = dspy.LM(
    "openai/xantly/auto-speed",
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
quality = dspy.LM(
    "openai/xantly/auto-quality",
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)

classifier = dspy.Predict(MyClassifierSig, lm=fast)
reasoner = dspy.ChainOfThought(MyReasoningSig, lm=quality)

Streaming

for chunk in dspy.streamify(qa)(question="Stream an explanation of DSPy."):
    print(chunk, end="", flush=True)

Model choice

Verify

import dspy

lm = dspy.LM(
    "openai/xantly/auto-speed",
    api_base="https://api.xantly.com/v1",
    api_key="xantly_sk_...",
)
dspy.configure(lm=lm)
print(dspy.Predict("input -> output")(input="say pong").output)

What you get

• Big savings on optimizer runs. BootstrapFewShot / MIPROv2 fire hundreds of LM calls during compilation. Xantly's semantic cache turns repeats into free hits. Common 40-70% cost reductions.
• Waterfall fallback during long compilation. A mid-optimization provider outage doesn't wreck your run.
• Per-signature cost tracking. See which module burned what in the dashboard.
• Memory for ReAct. Set X-Xantly-Memory-Mode: recall via default_headers on the dspy.LM to persist tool-use patterns across runs.
• Tool-call translation. Switch dspy.LM between providers and tool-using modules keep working.

Gotchas

The openai/ prefix is required. DSPy uses LiteLLM under the hood. Full model ID: openai/xantly/auto-quality (openai is the protocol, xantly/auto-quality is the model).

api_base not base_url. DSPy matches the LiteLLM kwarg name.

/v1 required in api_base.

Optimizer cost explosion. If you set max_bootstrapped_demos=16 with a quality model, expect thousands of calls. Use a value tier LM for compilation and a quality tier for the final compiled program.

dspy.settings is thread-local. If you multithread, each thread needs dspy.configure(lm=...).

Next steps

• Guidance, Microsoft's alternative to DSPy.
• LangChain (Python), classic chaining framework.
• Instructor, structured-output library.
• Cost-Optimized Routing, critical for expensive optimizer runs.