What LangSmith is built for
LangSmith is LangChain's LLM observability and evaluation platform — tracing, step-by-step visibility into agent behavior, and LLM-as-judge evals, framework-agnostic across OpenAI, Anthropic, and other SDKs via OpenTelemetry support. Its monitoring dashboards include cost tracking alongside latency and error rates, computed from the requests your application actually logs.
The gap: cost tracking runs on your logs and an internal rate table
Like other LLM observability tools, LangSmith's cost figures come from multiplying your logged token counts against a pricing table built into the product. That means a rate change at a vendor only reaches your LangSmith dashboards after LangSmith updates its own table to match — typically after you have already been billed at the new rate. Nothing about tracing and evaluation is built to watch a vendor's pricing page directly.
What VendorSpies does instead
VendorSpies is not a tracing or evaluation tool — it never logs a prompt, an agent step, or a request. It watches the public rate cards of 30+ AI vendors directly and sends a free alert the moment one changes, independent of anything your application has called.
Most teams run LangSmith plus VendorSpies
If LangSmith is already wired into your app for tracing and evals, that is a different job worth keeping exactly as it is. Add VendorSpies alongside it for the piece tracing was never meant to cover: an early warning the moment a vendor changes the rate your traces are being priced against.