Subscribe
Sign in
Langfuse
p/langfuseOpen Source LLM Engineering Platform
Start new thread
Clemens Rawert

Langfuse Prompt Experiments - Open Source LLM Engineering Platform

by•

Langfuse is the #1 open source LLM engineering platform. It helps teams iterate on and improve their LLM applications: LLM tracing, metrics, llm-as-a-judge, evaluations, prompt management, datasets testing and more. Create a free account or self-host Langfuse.

Add a comment

Replies

Best
Clemens Rawert
Maker
📌
Hi Product Hunt 👋👋👋 I’m Clemens, co-founder of Langfuse. We are so excited to be live on Product Hunt again today, launching the biggest feature yet: Langfuse Prompt Experiments! This new feature is designed to close the loop of developing and improving LLM applications. With Prompt Experiments, you can test and evaluate different prompt versions and LLMs on hundreds of examples simultaneously; Perform live LLM-as-a-judge evaluations and compare results in our new Dataset Comparison View to optimize prompts for your specific use case. You can quickly make prompt changes, see whether they improve metrics without causing new issues, and then deploy to production with confidence. Move faster on your LLM Application development than ever before. A few things you should know about Langfuse: 👣 Tracing: TS & Python & API + OpenAI, Llama Index, LangChain, LiteLLM + many more ✏️ Prompt Management: Collaboratively manage and deploy prompts from within Langfuse ⚖️ Evaluation: Automatically run evals to control for hallucinations or other criteria you care about 💾 Datasets: Collaboratively manage fine-tuning, testing and golden datasets 🧪 Dataset Experiments: Test different prompts and models on hundreds of traces simultaneously 📊 Metrics: Dashboards and analytics on cost, latency and quality 🕹️ LLM Playground: Engineer your prompts and directly see results 🏎️ Export & Fine Tune: Open GET API and csv/JSON exports to build downstream use cases 🚄 Scale: We’ve invested significantly in scaling and resilience as we’ve scaled to thousands of users and handle many many millions of events a day 🧑‍🤝‍🧑Community: Join our thousands of users on GitHub Discussions and Discord See for yourself: ⭐ GitHub: https://github.com/langfuse/lang... ⏯️ Interactive Demo: https://langfuse.com/demo 📒 Docs: https://langfuse.com/docs 💬 Discord: https://langfuse.com/discord Thanks again, PH community. This is where we started, and we are incredibly grateful for all of the community feedback which led to the development of this major new feature. We’re excited and will be in the comments the whole day to hear your thoughts & feedback!
Marc Klingen
Maker
Wohoo, so happy to be back on Product Hunt. Will be in den comments all day, happy to answer any questions you might have!
Jonas Beisswenger
Congratulations on the launch!! 🚀We have been using Langfuse from day one and it helped us tremendously in getting deep visibility into the performance of our AI stack!
Marc Klingen
Maker
thank you, @jonas_beisswenger. Let us know if you have any feedback based on your experience with Langfuse OSS!
Tobias Kemkes
Congrats on the launch! The prompt experiments look super exciting 🔥
Marc Klingen
Maker
@tobiaskemkes Thanks, Tobias! I just read about your prompt-focused launch yesterday. I need to give it a try this weekend.
Marc Metz
Congratulations, Clemens and the Langfuse team! Launching Prompt Experiments is a game-changer for fine-tuning LLM applications.
Clemens Rawert
Maker
thanks @marc_metz, prompt and mode changes can be very risky. Glad that we can help mitigate some of the risk with langfuse now via strong evals and tests -- without needing to use our SDKs
Mathias Nestler
I really love that feature. Such a game changer, saves tons of time and keep quality of AI integrations high. Love it <3
Marc Klingen
Maker
Thank you, @mathias_n, for your feedback! Please DM or share here if you have any additional ideas on how to improve this.
hritik choudhary
Langfuse Prompt Experiments is a powerful tool for optimizing and managing LLM applications! The ability to iterate and test different prompt versions, while ensuring efficiency and reliability, will certainly speed up development cycles. The integration of prompt management, evaluation, and dataset comparison is a game-changer for teams working with LLMs. How do you handle edge cases in prompt testing to ensure your models are robust across various scenarios?
Clemens Rawert
Maker
> How do you handle edge cases in prompt testing to ensure your models are robust across various scenarios? @hritik_choudhary, this mainly depends on the datasets used for the experiment. The closer they resemble production, the more robust the test will be.
Kevin Wu
When Langfuse was released early, I was one of the users, and I integrate into my chatbot. It was smooth. Congratulations on launch.
Marc Klingen
Maker
Thank @0xinhua for joining early and providing continuous feedback. Excited to build Langfuse with you and the community!
Goscha Graf
🚀 Congrats, this is BIG! 🚀 Very excited to see how far you have come in just one year! And so excited to see how much further you will go from here. ✨ Very exciting product and a great team! ✨ ⭐️⭐️⭐️⭐️⭐️ Recommended!
Marc Klingen
Maker
Thank you @justgoscha! Let us know if you have any feedback or questions!
Dan
Congrats team on the producthunt launch and the whole launch week! Really big fan of the new features around experimenting with prompts and the new eval features! Glad to have not to do any more vibe-based evaluation!
Marc Klingen
Maker
@dan_meier1 this is the main goal, move to more structured evaluation which helps to confidently make decision - without having to set up a whole testing suite and thus making this more accessible to everyone involved in prompt engineering
Gabor Cselle
Congrats on the launch today, I've been enthusiastically following your launch week!
Marc Klingen
Maker
@gabor thank! Join us on discord or GitHub if you have ideas of what you would like to see in Langfuse
Aditya Lahiri
Congratulations on the launch team. OpenFunnel agents are happy and excited users of Langfuse <3
Marc Klingen
Maker
@aditya_lahiri let’s go! Openfunnel agents might now get better even faster 🚀
Max Brenner
sleek! congrats
Marc Klingen
Maker
thank you @maxjbre! can't wait to see what you'll do at MagicAds with it to move faster without breaking things
Felix Vemmer
Congrats on the launch. Excited to test this for my own and clients projects! 💪
Clemens Rawert
Maker
thanks @felix_vemmer, ping me when you have questions or feedback! hope this helps you move faster
Mika Sagindyk
Congrats on launching your prompt experiments feature! Excited to see what follows 🔥
Clemens Rawert
Maker
thanks @mikasagi, how do you manage and evaluate prompts at Hemy?
Samar Ali
Launching soon!
What differentiates it from other platforms? Would love to learn more about the features and benefits. Keep up the great work
Marc Klingen
Maker
Hi @samalyx, thanks for asking! We have some pages on this that might be helpful: - features: https://langfuse.com/docs - why teams use langfuse: https://langfuse.com/why let me know in case you have any detailed questions on features that you are interested in
Huzaifa Shoukat
Congrats on the launch! Langfuse looks like a game-changer for LLM development. How do you see it transforming the way teams build and optimize their language models?
Marc Klingen
Maker
@ihuzaifashoukat teams working with LLMs often have similar issues: evaluation of outputs, various inputs, reasoning steps within agentict systems. We have summarized this here in "challenges of building llm applications": https://langfuse.com/docs
Stelios Sotiriadis
@clemo_sf, congrats on your launch! It's a really cool product!
Raghu_Rtr
great product man
David Nordhausen
Looks fantastic congrats on the launch team
Marc Klingen
Maker
Thanks @david_nordhausen, curious what prompts you will use this with first!