Subscribe
Sign in
Start new thread
Data Scraping Studio (v 1.7)
p/data-scraping-studio-v-1-7
Crawl password protected websites using CSS Selectors
Visit Product
Vikash Rathee
Online Web Scraper — Cloud hosted data extraction tool
Featured
38
•

Replies
Desmond Duggan
Do you support pagination? I'm scraping from this page: http://m.ironman.com/triathlon/r...
Vikash Rathee
Maker
@desmondduggan2 Yes, we do support pagination. Here is the tutorial https://www.datascraping.co/doc/...
Helmi
Hmm just wanted to signup and was told that my password can't be longer than 30 characters. Not that i really need longer papsswords but how does that matter? Shouldn't passwords be encrypted and the length be totally... whatever?
Vikash Rathee
Maker
@helmi We don't want you to remember 30+ character password. Usually 5-8 character is the best choice and 95% of user keeps that in this length and we encrypt that. So it's simple industry standard we've implemented in our system. Can't help sorry!
Vikash Rathee
Maker
Hello Product Hunters, I started DataScraping.co around a year back by launching the desktop app "Data Scraping Studio" on product hunt (https://www.producthunt.com/tech...) as a point and click CSS selector scraping app with amazing capability to run multiple agents simultaneously and over 5000 users downloaded the application and uses now in their business. Over the time we noticed, one of the major ask from community was the online solution and REST API. To integrate and use the scraped data in existing system without installing the desktop app or to use across operating systems. And we keep recommending our user to try (YC Backed) Kimonolabs, but then suddenly they shutdown their platform on March, 16 leaving 100k+ users leaving behind on short notice. Today, I'm very excited to launch the online web scraper with the same capabilities and power of cloud along with REST API to get the same Kimono(or even better) experience back. Go to our website and signup to get 1000 pages credit free every month to test the features and you can go PRO anytime if you like it! I’ll be available all the day to answer any comments, feedback you have, really looking forward to it!.
O O
@vikashrathee IMO scheduled, cloud scrapers are the way to go! Was a BIG fan of Kimono, but what did it for me with them was the ability to get an RSS feed from my extraction and a super reliable Google Sheets Add-on that would update when the scraper would run. Still can't find anything similar! Do you/ will you implement something similar? 😁
Vikash Rathee
Maker
@sandrojazzar Hey Sandro, thank you for the comment. Yes we do have RSS feed in plan and will be adding that in coming weeks. Google spreadsheet is not the priority right now but yes we will see this if more user requests the same. For now there is an alternative using the Zapier can post the data to Google Spreadsheet from webhook (we do support webhook). You can give a try. https://zapier.com/zapbook/zaps/...
mp8
@sandrojazzar have you tried Scrapinghub.com they are they have a lot of the same features as kimono including scheduled runs.
Vikash Rathee
Maker
@marcelp81 @sandrojazzar No they don't have that to automate I guess. What they explained is manual import each time in this article. https://support.scrapinghub.com/... So better would be to use Zapier to post the data automatically in spreadsheet each time the scraping job completes using any tool of your choice.
Kieran Daniels
@vikashrathee Do you have anyone or and example of someone scraping Zillow / Realtor.com for buyer/seller leads? Thanks!
David Marshall (Red)
I find data scraping vital for exploring open data. First impressions are good, I'll run it alongside existing projects for a proper test or two.
Vikash Rathee
Maker
@redpuppy Thank you David. We'd love to help you in your tests, do let us know if you have any questions.
Sunil Kumar S V
'Recipies' kind of stuff would be a nice add on! (For some frequently used type of website)
Roger Oba
Really amazed with your product, service, and mainly pricing. Any plans on Mac App launch?
Vikash Rathee
Maker
@roger_oba Thank you for kind words Roger. We do have plan for Mac app but that will take some time. Try the hosted app and API by that time and we'll notify all users when Mac app launched.
Vikash Rathee
Maker
@roger_oba Thank you.
Hussein shtia
very good tools what payment gateway you use ? may i can help with Better solution ( in app purchase for desktop software ) thanks
Vikash Rathee
Maker
@hussein_shtia Right now we are using Paypal. You may send me the details, as I'm looking for one something better. Tried a lot but unfortunately due to RBI guidelines none of the payment company accepting our request. https://support.chargebee.com/su...
Vikash Rathee
Maker
@kieran_daniels @hussein_shtia Stripe and all others are unable to provide services in India due to RBI guidelines. Had a discussion with them last month.
Jonathan Toh
That price >.<
Vikash Rathee
Maker
@rainbowhat Yes it works, you just need to enable the "Fetch pages via browser > Enable Ajax/Javascript Load" and you should be all set. Right now the UI is not ready in hosted app to do so as in desktop app, and you'd need to manually open the agent in notepad and change the configuration as per below JSON object and upload the scraping agent again to do that. (The configuration UI will be added in hosted app by end of this month). "RequestViaBrowser": { "Enabled": true, "AjaxEnabled": true, "ImagesEnabled": false }
Hadi Farnoud
unfortunately it doesn't work for me. I've tried to use it on https://web.archive.org/web/2016... to extract data from a lost site. data is inside a div that appears on hover over images. my agent is 20fce2dc52
Vikash Rathee
Maker
@hadifarnoud Hi Hadi, I just copied your agent in admin account to test and found that, the data was extracted but was not visible in table due to space and line breaks. I just enabled the clean, trim function in your account and you are good to re-run your agent now. Here is the screenshot I tested this in my account. And your question is posted here to help others to understand what went wrong. (https://www.datascraping.co/doc/...)
Kieran Daniels
@vikashrathee @hadifarnoud Impressive customer service ^^ ;)
Erika Foo
Haven't use it, but will try later. By the way, I think Octoparse is a good choice too. https://www.octoparse.com/
Nicola Mastrandrea
Why should I prefer OWS to Import.io?
Vikash Rathee
Maker
@nick88msn - Save money : Import.io costs $299 monthly for 50k pages while datascraping.co charges $130 only. - Unlimited agents : Import.io support only 30 runs storage, while in datascraping.co you can create any number of agents and upto 30 days historical data for any number of jobs run during 30 days. - Sub accounts : Create sub account for your team to share the same account with role based security. - Crawl with Limit : Advance limit option to crawl data in parts instead all in one go without modifying the input lists. - Modify data using JavaScript - Currently available for desktop app, will be added for hosted app soon (https://www.datascraping.co/doc/...) - Dynamic input : Extract many website using a single agent. (https://www.datascraping.co/doc/...) - Password protect site crawling : Easy form submission to crawl password protected websites (https://www.datascraping.co/doc/...) And lots more...