10 posts tagged with "Prompt management"

Braintrust is the first hosted-only platform in this series, and the first that runs your tool code instead of only storing the tool's schema. Six platforms have been compared here before Braintrust: Langfuse, Phoenix, Opik, Helicone, MLflow and Laminar. Every one of them stores a tool's JSON Schema at most. Braintrust takes your Python, bundles it, and runs it on its own infrastructure. That one decision sets the Python version you may use, sets whether your handler can open a file, and sets where your data has to live. This comparison is built around that decision.

Braintrust is not open source, and self-hosting needs the Enterprise plan. There is no free self-hosted install to test against, so everything below ran on their hosted Starter tier.

MLflow is the open-source ML/GenAI platform originally built at Databricks, and by far the largest, oldest project we've compared AcruxCore against — 27,000+ GitHub stars, 444 contributors, shipping since 2018. Its GenAI side is newer: a Prompt Registry, a tracing store, LLM-as-judge evaluation, and — the real surprise of this comparison — an AI Gateway that sits in the request path, the same design AcruxCore uses. We built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call through a gateway, inspect the trace, build a dataset, and call it from an SDK script.

Opik is Comet's open-source LLM-ops platform, and unlike some competitors we've covered it ships with genuinely no login wall when self-hosted — you docker compose up and you're working inside a real workspace immediately. We built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call, inspect the trace, build a dataset, run an experiment, and call it from an SDK script.

Langfuse is the LLM-ops platform we get compared to most often, and it deserves a real answer, not a table copied from two docs sites. So we built the same prompt — vip-support-triage, a support agent that changes tone for VIP customers and lists their open tickets — on both platforms, then ran the identical sequence on each: create the prompt, version it, send a live call, inspect the trace, build a dataset, run an experiment, and call it from an SDK script.

Most tool comparisons are written from docs and marketing pages. We didn't do that here. We ran nine LLM-ops platforms ourselves — LangSmith, Langfuse, PromptLayer, Arize Phoenix, Opik, MLflow, Helicone, and Laminar, plus our own AcruxCore as the baseline — and did the same thing on each one: create a prompt, version it, run it live with a real model key, inspect the resulting trace, and try to build an eval. Then we wrote a small script against each platform's own SDK and ran that too, and timed every one of them against a direct call to the provider.

Tracing and monitoring — what most people mean by "LLM observability tools" — is only one of nine angles below; prompt management, evals, guardrails, and tool-calling get the same hands-on treatment.

This is the AcruxCore leg of a hands-on comparison series. We ran the exact same steps — create a prompt, version it, generate a trace, try to build a dataset — against the hosted product ourselves, so the LangSmith, Langfuse, and PromptLayer write-ups have a fair baseline to compare against. No marketing framing here — just what the dashboard actually does. The main comparison post pulls the findings from all four platforms together.

Langfuse is one of the best-known open-source LLM engineering platforms — it covers prompt management, tracing (recording what an LLM app actually did, step by step), evaluation, and datasets. This is the Langfuse leg of a hands-on comparison series: we logged into Langfuse's hosted EU cloud with a real account and clicked through the whole loop ourselves — open a prompt, version it, inspect a real trace, and set up a dataset for testing. This post is a plain, factual account of what we found — the good and the rough edges. The main comparison post pulls the findings from Langfuse, LangSmith, PromptLayer, and AcruxCore together.

We logged into the real, hosted version of LangSmith (US region) and worked through the same four steps we're using across every platform in this comparison: create a prompt, run it and inspect the trace, and build a small eval dataset. This post is just the LangSmith leg — screenshots, actual UI, no marketing copy. The main comparison post pulls the findings from LangSmith, Langfuse, PromptLayer, and AcruxCore together.

This is a hands-on look at PromptLayer, one of the products we compare ourselves against. Instead of reading their docs, we logged into a real hosted PromptLayer account and did the same four things we do on every platform in this series: create a prompt, version it, run it and inspect the trace, and see what evaluation looks like. This post is just the PromptLayer leg — the main comparison post pulls the findings from all platforms together.

If you're building on LLMs, you eventually need four things: prompts you can change without redeploying, a reliable path to model providers, visibility into what each call did, and a way to tell whether a change helped. LangSmith and Acrux Core both address this space from different starting points. This is an honest look at where they overlap and where they differ.