Dagster
p/dagster-cloud
The data orchestrator that puts developer experience first.
flo merian
Dagster+ — Ship data pipelines with extraordinary velocity
Featured
25
Dagster+ is a category-redefining data orchestrator that brings together unique capabilities into one tool in order to speed up your development work, reduce your costs, and simplify your data platform. Try Dagster+ for free, no credit card required.
Replies
Pete Hunt
Today we're launching Dagster+, the next evolution of Dagster Cloud. Dagster+ is targeted at data engineers, ML engineers, and analytics engineers, and is a major rethinking of what a data orchestrator is, where it sits in the stack, and how data platforms are constructed. Since our launch in August 2022, we have seen enormous growth in both the Dagster Core open-source project and our commercial product. We now have hundreds of commercial customers and the fastest-growing open-source community in the category. With this growth comes broader and deeper relationships with a diverse set of customers. As we engaged with these customers throughout 2022 and 2023, we heard a few key challenges: • They were spending too much money on infrastructure, especially on the data warehouse, ETL tools, and AI workloads, and struggled to understand, optimize, and control their spend. • Their stacks were sprawling and complex, with hundreds of different integrations and point solutions with varying levels of maintenance, and no single pane of glass for understanding the data platform as a whole. • They wanted to empower individual teams to work autonomously end-to-end, but struggled to balance this with the centralized needs of the data organization. Dagster+ is a suite of tools that sits on top of Dagster's data orchestration engine, and replaces the existing Dagster Cloud product. Dagster+ brings four new capabilities above and beyond our existing commercial product: 1️⃣ We have built a brand new data catalog and data discovery capability in Dagster+, which enables all users of the data platform to discover and leverage data autonomously. 2️⃣ Dagster+ builds on Dagster Core’s asset checks feature, introducing additional capabilities around anomaly detection, freshness, and alerting. 3️⃣ We’ve made massive improvements to our Branch Deployments feature from Dagster Cloud, and are launching a killer new feature, Change Tracking, with Dagster+. 4️⃣ Dagster Insights enables data teams to manage their spend on tools like Snowflake, Fivetran and OpenAI without doing any additional integration work. With Dagster+, Insights is now generally available and comes with many new features and integrations. All of these capabilities are enabled by Dagster's unique asset-oriented approach to data orchestration. I'm really proud of the product we've built, and I'm proud to be a part of the team that delivered it. We hope that Dagster+ helps you reduce your spend, eliminates the need for many point solutions, and empowers your teams to work more autonomously and less chaotically.
Pete Hunt
@deshraj_singh Thank you! Team effort!
Tanvir Hasan
Huge congratulation to the Dagster team on this major update! Super excited about the native Kubernetes support and declarative assets! Are there any plans to expand real-time observability or add even more integrations with other data tools?
Pete Hunt
@tmwky absolutely! Our real time observability system supports custom metadata, and we are always looking at supporting more integrations. Any in particular you're interested in?
Micha Cassola
Wow! Sounds like a wonderful solution to data management and observability! Do you have more concrete examples? What I've read is still all a bit theoretical.
Fraser Marlow
@michacassola All will be revealed at the launch today at 12:00 noon EST: join us
Ari Bajo
Exciting! Can you tell me more about Change Tracking?
Leslie Stinson
@ari_bajo_rouvinen Hey Ari! Our live launch is coming up and one of our engineers Jamie will be walking through the feature: https://youtu.be/_Z4xxZYEQNs We will also share more content around this feature on our website later today :)
Daniela Lopez
Sounds like a total game-changer. Congratulations to the Dagster team on this impressive release!
Leslie Stinson
@dani_does_content Thanks, Dani!
Andrew Grigorev
Congrats on the launch of Dagster+! 🎉 It’s fantastic to see how you’ve listened to the community and developed new featuers that address the real challenges faced by data teams.The introduction of the data catlog, change tracking, and enhanced spend managment tools - shows a deep understanding of what’s needed to simplify and streamline data operations. Really excited to see how Dagster+ will transform data orchestration and empower teams to work more efficiently and autonomously. Cheers to a future with less chaos and more inovation in data handling! 🚀
Fraser Marlow
Today's launch will happen on a YouTube Live at 12:00 noon EST / 9:00 AM PST - we hope you can join us:
Moon
Congratulations on the launch! Nice tool
Trent Kennelly
Really cool product! I love the new, more autonomous way of working using Dagster. Excellent work!!
Vasily Kolosov
Congratulations on the launch! I'm particularly intrigued by the data catalog and discovery capability, as well as the improvements in spend management with Dagster Insights. How do you envision these new features transforming the data platform landscape for your customers?
Fraser Marlow
@vasily_kolosov1 Great question. We already see many Dagster Cloud (now Dagster+) users consolidating their stack and deriving significantly more value from the metadata that is naturally collected by the orchestrator. This in turn allows a data platform where all data practitioners can collaborate around the definition, creation, and maintenance of critical data assets in the organization.
Pete Hunt
@vasily_kolosov1 in addition to what Fraser said, there's a real opportunity for data platform teams to enable their customers to self-serve. Enabling this sort of organization decentralization is key to unlocking productivity at scale.
Leslie Stinson
Hello PH community! We are so excited to launch Dagster+ today and thank you all for your support. We also have a live launch happening at 12pm ET today -- tune in to check out the full walkthrough of our new capabilities: https://youtu.be/_Z4xxZYEQNs
Riley Peronto
Great work, looks like a major upgrade. Where can I learn more about your cost optimization use case? We solve a similar problem in observability.
Leslie Stinson
@riley_peronto Hey Riley! We'll be discussing this in detail in the upcoming live launch: https://youtu.be/_Z4xxZYEQNs And more content around our cost optimization feature will be available later today on our website.
Leslie Stinson
Chris Davis
Small nit: the pricing calculator would be more helpful if it helped me translate credits to something more tangible. For instance, showing the credit breakdown for one of the [example projects](https://github.com/dagster-io/da...) could be illustrative.
Fraser Marlow
@thedatadavis good point. All consumption-based pricing models take some exploration to understand how it will play out in your use case. Feel free to contact our sales team to discuss how this applies in your particular circumstance: https://dagster.io/contact
Very cool project. Supported and starred. Good luck with the launch