@samir_doshi ;)

@datarade sounds like you should make a collection of Scrapers.

@datarade Jeez that's a lot! Are they all similar, or just different use cases?

@giannidalerta follow me and check out my blog. My plan is to continue these type of comments. ;)

@datarade I've seen a lot of your epic scraper comments on PH, so looks like you are the scraper king haha. What are your go-to scrapers nowadays for social networks like Linkedin + Twitter? Looking for something web based / usable on Mac. Data-miner.io is good, but a bit complex for us non-technicals. Many sites like Portia, Import, etc don't work on these sites.

@gpuliatti I'm curious to hear your perspective to) Pala tired acquisition and sudden shutdo not of kimono labs and how they handled that?

@jpmillions We're open source guys (and gals)… so we're definitely saddened to see users of a closed platform treated this way. OTOH, we've seen a lot of customers coming to try Portia from Kimono. :) We're actively working on ways to help people port their Kimono crawlers, so keen on hearing anyone in this boat! Email me directly (gabriel@scrapinghub.com) or sign up to the mailing list at the bottom of this post (https://goo.gl/CGxsFl). Both Portia and Scrapy are fully open source, and any crawlers (created or running) in our platform are fully exportable and interoperable with open technologies. While we are focused on the long-term and so doubt our platform will be shut down any time soon, if that ever happened, all of our users would be able to export their crawlers and use them on their own infrastructure. We've done this ourselves for some of our Professional Services clients who want us to build scrapers but also run things on their own infrastructure.

@gpuliatti I tried scraping a list of 27,000 urls... the browser crashed. Is there any easier way of adding URLs to the scrape?

@pablohoffman @gpuliatti @nwkwan I'm sorry for late response. We've released new service for data scraping https://www.producthunt.com/post... . It is able to extract long infinite scrolled pages. I would really appreciate your expert review of our DFK service

@ykguler We got hit by the Product bump and our dashboard was having some issues… looks like things are working again.

@tribaling A few use-cases from our past client projects: - Scrape eCommerce sites that sell your products, to check for price violations and review data. - Build a broad crawler covering thousands of sites to automatically discover contact and profiles information for a specific industry. - Parse all shop locations for a number of big brands to provide a locator for users looking for a specific type of shop. - Build a database of interesting candidates to hire, by matching various sources of internet profiles with a series of filters which you or the HR team are interested in. I know people building boutique businesses on basic web scraping… like someone who uses our platform to offer a service that allows people to monitor Amazon Kindle Books pricing, and get alerted when the price drops or the book goes on sale. In effect, bringing Amazon's data "back to the people" to allow them to make better choices. But of course, most of the $$$ value comes from being a Fortune 500 company and being able to understand a lot more about the world, your industry and your competition. We help both large and small increase their reach and get access to the best technology. :)

@johnalxndr Everything should be back up now! We do have a free plan, it's the big box on http://scrapinghub.com/pricing. You can get shared resources and 1 concurrent crawl for 0$ a month.

@thisdickie For something that needs to grab data from any generic news site like Flipboard, Portia may be a bit limited. I would recommend using Scrapy, and using one of the many content parsing libraries available for Python. Moz has a great write-up of the available ones on their blog: https://moz.com/devblog/benchmar...

@thisdickie Hey Sam, you might also want to check out http://www.diffbot.com/ as we power more than a few article reading apps

@wanderslth Hi Ken, we think Import.io is great… I've personally used their browser tools quite a few times to do something quick. I'd say the biggest benefit you get with Portia is actually our platform… every project has a great API (http://doc.scrapinghub.com/api/o...) which you can use to schedule your crawls and run crawls, as well as review and download data. You can use Crawlera's proxies, send data to various sources like Amazon S3 (plus images and files separately if you want) or even machine learning services for extra text analysis by enabling add-ons with a couple of clicks. Portia crawlers behave quite similarly to Scrapy crawlers, which means joint-projects are possible… Portia tackling the low-hanging fruit without needing engineers, and Scrapy doing the heavy lifting on the sites that need it. Having an easier option helps lower overall project costs, and Scrapy allows engineers to piece all parts together and crawl the tougher sites. This has allowed us to handle much bigger projects that we could with only one or the other.

@gpuliatti ah; cool -- I took a poke around the larger site and looks... comprehensive. :) I'll definitely bookmark for a deeper dive here soon. Thanks.