Subscribe
Sign in
DataFuel.dev — Turn websites into LLM-ready data.
DataFuel API scrapes entire websites and knowledge bases in a single query. Get clean, markdown-structured web data instantly for your RAG systems and AI models. No complex scraping code needed.
60
Replies
Best
Hi Sacha! DataFuel sounds like a game-changer for anyone dealing with web data. The feature to scrape behind logins is particularly impressive. How did you manage to simplify such a complex process? Looking forward to trying it out!
@web3_antivirus
Thanks a lot Alex!
The main processes that I tried to tackle are:
- handle retries
- handle automatic discovery of sub urls without sitemap
- handle login
- handle status and heavy background job
I hope It helps users feel scraping is getting easier at the least
Congrats on awesome product, this is exactly what I am looking for my project! Do you have any plans for images?
@edmundas_eddy Great idea! An image is definitely likely to be very useful. If you have a specific need, please feel free to contact me on Twitter @dumay_sacha or by email at sacha@datafuel.dev.
I’d be happy to help you and discuss the details further.
Do you want the markdown to include an image, or is it something else you’re looking for?
@sacha_dumay we help our customers generate proposals with AI for new customers. Sometimes they only have the website of the potential customer and it would be great to also get the website logo and some other images to add to the proposal for customisation. So images and links to images in the would be great
@edmundas_eddy great the image link should be available right away. You can also use our json schema with AI to only get information you need cleaned and structured. Please give a shot to our product! Thanks
Very useful space, wondering what’d be your main answer to competition like Firecrawl and MultiOn’s offering.
What would you say is the main differentiator of DataFuel?
@jalcantara Great question! I believe that with Datafuel, you get:
- Automatic retries (if a proxy fails, we pause and automatically retry with a more expensive proxy to ensure reliability).
- A convenient way to scrape password-protected websites or knowledge bases.
- An easy way to obtain filtered JSON data via AI (GPT-4).
Thanks @sacha_dumay, but that’s just what all of these do, I mean what sets data fuel as a different offering?
@jalcantara I don't believe they exactly do that.
I am considering also adding the embedding in vector database included in our API
What do you mean @sacha_dumay ? Generating and returning chunked embeddings (split+Emb model)? or allowing the saving of those (pinecone)?
Not a bad path, that's in the end what a loft of folks use it for. Similarly if you summarized the page, etc. A narrower focus in which you execute well is a better niche than being one more player in a field.
@jalcantara yes exactly basically adding chunking and embed it for the user right after the scraping.
If you are interested in that now please contact me at sacha@datafuel.dev and I can get you started.
Thanks a lot for those feedback !
Congrats on the launch!! Does it also scraps websites built with client side technologies like ReactJs t?
@sudhakarb yes for sure! do you have an example? did you try it ? You can try for free 20 URLs.
@kingromstar Here’s the corrected version:
Yes, it does. I'm still trying to figure out how to differentiate more.
Do you have any suggestions or features you'd like to see?
Impressive evolution of the product! No doubt it can become an essential tool for anyone looking to train AI models without the hassle of sourcing data
The markdown format is such a clever addition. Makes it so much simpler to work with for my projects.
@gabriel_silass lovely yes markdown is great for human and AI, much easier to grasp and cheaper to feed into an LLM
Congrats Sacha! This looks awesome for a project we have. Can’t wait to try it.
The Access Gated Content feature looks promising! Truly innovative, and I’m keen to see how far this will go.
@furqanramzan Thanks a lot ! yes It is definitely promising, I am thinking to add SSO login too!
@arkim_phiri thanks for the support! I ll keep working to improve it and give more value
Congrats on the launch! DataFuel sounds like a game-changer for anyone working with web data. How do you see it being used in conjunction with other AI tools and platforms?
@ihuzaifashoukat yes I think using RAG or fine-tuning in combination with DataFuel can be super useful !
Think knowledge based, chatbot builder, new LLM model to train, etc
@brice_fromm yes a few marketer are using our AI json schema feature to get email, name, addresses leads!
@melvin_vd glad you like it! and I hope you can use it soon in production!
Awesome! I need that. The result, converted to Markdown, is really impressive!
I recommend DataFuel.dev to make it easier to collect data from websites! If you need to quickly and efficiently collect data from sites for RAG (Retrieval-Augmented Generation) systems or AI models, DataFuel is the perfect tool.
@varrr_al thanks for the recommendation! RAG is really my to-go use case!
@ioannis_tsiokos great to hear! right now with 50% for 3 months, so it is a great deal I hope :)
@sacha_dumay cool! i tried subscribing to a startup plan, but I couldn't find any promo code input in the checkout page to enter the code PH50OFF. am i missing something?
@ioannis_tsiokos It was my mistake, it should work again. Please try again! Much appreciated
DataFuel.dev
- 🚀 Scrape entire websites or knowledge bases in one query—no need for custom scripts.
- 📝 Markdown-structured data—perfect for RAG, saving GPT-4 costs and improving accuracy.
- 🔒 Scrape behind logins—access data from password-protected pages effortlessly.
- 📦 JSON output—extract emails, names, addresses, training data, and more.
- ⛏️ No proxy or retry headaches—let us handle the hard stuff.
- 🎁 Free trial—your first 20 URLs are on us!
💥 Launch special: Get 50% OFF for the first 3 months! I’m so excited to share this with the Product Hunt community. Whether you’re training chatbots, building RAG systems, or need clean web data for your project, I’d love for you to give it a try. Check out DataFuel.dev and let me know what you think! Ask me anything here—I’d love to hear your thoughts and answer your questions. 🚀