Subscribe
Sign in
Website Metadata Scraper API — Extract email, phone number and more from just a URL.
Website Metadata Scraper is an API to extract email, phone number, social profiles, review profiles and more from just a URL.
35
Replies
Ed DeCaria
This looks promising. I tried to test it (first in mashape, then in Postman) with a random url (sephora.com) but my request seems to be "pending" for awhile now. I was hoping/expecting that it would finish in close to real time -- it would have to return metadata within a few seconds to work for my use case. Maybe I'm doing something wrong ...
Share
hey @edecaria! there are two factors at play here:
1. we are going through quite a big queue at the moment, so it's taking longer than usual. i can see it's run through for you now though :)
2. we go through up to 30 pages per URL to find the components we need, and only save the data we find at the end of this. i will make an update now so that the data is saved as it's found, which will show results a lot quicker.
thanks for giving this a try!
@philip_kallberg Thanks for the reply. And yes I see my previous pending calls have gone through. So what do you thinking a typical turnaround time will be for a pair of POST + GET calls (successful)? Is it first-come, first-serve through the queue or do paying customers get bumped to front of the line? This API seems pretty solid. I'm currently using Diffbot to scrape social links, but tbh it doesn't have a very high success rate. For example, it doesn't pick up the sephora social handles, whereas your API does. Same for nike.com, which I just tried. I also see that my nike call is still crawl_status:pending even though I see the social handles on my GET request. Impressively quick work if you already made the change you noted above -- thanks!
@edecaria yeah our queue prioritises paying customers, the higher the plan the higher the priority. glad to hear you like it!
and yes i did update that for you, i should have thought of that before haha :) also note that some big sites will have captchas and other blocking mechanisms which isn't a problem but add to the load time.
@philip_kallberg Cool. That makes sense. Mind explaining why the GET requests count toward the limit? Seems that the real unit of value is the POST. Especially if I start making multiple GET calls to try to pick up the response as soon as it's done, seems like I could burn through even 1,000 monthly calls pretty quickly. I'd suggest a webhook/callback when the crawl finishes or better yet when a specified field is populated. That's a side issue, though. Really just trying to figure out how to ensure a reliably fast response. This API seems like a perfect fit for a product of mine, but if it's 5-10 seconds sometimes and 1+ minutes other times, I can't really depend on it.
@edecaria yeah i agree the pricing isn't optimal, happy to discuss.
why don't you try running through 20 urls and seeing the response time? if it's social info you want i'm sure that'll be pretty fast. most sites have those in footer so it'll be picked up on the first page.
please join me on the website chat and we can discuss your use case and see how this could fit in.
Really good! If you're able to connect this with Zapier, we'd be able to do everything automagically 😉
@thiagoafram glad to hear that, we could create a zapier app if there was enough demand 😉
@vinod_sharma1 you would have to do so on your side at the moment. in future we may change the structure of the api so you can run a batch job of xxxx urls and we return a job id which you can poll to receive a csv once complete 😃
Really helpful tool to gather more information real quick
Pros:Easy to use
Cons:None
good work here @philip_kallberg. I get the scraping for sales data etc, but I don't understand the use case of collecting review data etc (I may just be having a brain fart!).
@jwelch thanks for checking us out! when it comes to review data, we released our review scraper api [0] a few weeks ago and got to learn about a bunch of uses cases around scraping reviews. the process with that api looks like this:
1) find the review profile url (eg. yelp) for a given company
2) pass it to the review scraper api
3) receive reviews back
this metadata api let's us automate step #1, so eg. if you want to get reviews for your competitor, just give us their website URL and then we can get their reviews.
as for collecting review data, we picked up clients in very different industries since we launched that api a few weeks ago. the overarching theme is generating actionable insights based on review data - whether on industry, company or competitors. we do this ourselves @ Reviewshake but thought it'd be cool to open up our infrastructure to help other companies too.
hope this makes sense?
[0]: https://www.producthunt.com/post...
Really easy to use.
Now we can offer review management to our clients - on top of the awesome services we already offer.
Very easy to use.
We can now offer review management to our clients - on top of the amazing services we already provide :)
Pros:Very easy to use.
Easy to follow explanatory video.
API params are exactly the one one would expect - no unnecessary added complexity.
Cons:None so far
How many times have you had a long list of URLs, and wanted to extract some piece of information from them? This weekend I had enough of that and my project became to create a simple API for it...
You give us a URL and we return any contact data we can find (eg. email, phone), social data (eg. Facebook, Twitter, Instagram etc.) as well as review sites in use (eg. Yelp, Tripadvisor and 50+ others).
Here are a few use cases that come to mind:
- Sales... easily get emails/phone numbers for your outreach
- Find review profiles for a given company and feed them to the Review Scraper API [0] to get their reviews
- Use review data in your next email outreach campaign to customize your email, eg. "Hey X, Congrats on your 5-star review from Jason last week!" - this works wonders 😉
- Follow or monitor companies on their social profiles
There are already tools that return known email addresses for a given domain, but they don't actually go to the website to fetch that data. The number of times such tools have come up empty handed and I've found the email directly on the target website is staggering. This API also returns a lot more data than just email addresses. We've started using this internally and it's already made our life so much easier, and I hope it will for you too!
This is just the start - there is so much more we can achieve and I would love to hear from the community how you would use this? Feel free to sign up to the free tier and go wild :)
[0]: https://www.producthunt.com/post...
Massively useful tool!
Keep up the great product refinement work. Look forward to getting even more value out of it as it gets even better.
Massively useful tool!
Keep up the great product refinement work. Look forward to getting even more value out of it as it gets even better.
Pros:Time saver!
Cons:Where has it been all my life?
I loved that they offer a service that you don't have to code yourself. Pretty usefull for a lot of ocasions
Pros:Pretty easy to use
Cons:none
Will keep using for a week to see how it runs
Pros:Super easy to use and very quick to set up.
Cons:Would love to use for products like amazon but not yet supported
Have uploaded about 3k sites so far in its working like a charm!
Pros:Great bang for buck value
Cons:Too early to tell will keep an eye out but looks good so far
Was looking for an API like this. It saves the time and hassle of going to a merchants websites and trying to get all their details. Such a useful API!
Pros:Very very useful!! and easy to use when trying to gather information from websites.
Cons:None so far
I think you guys need to retool your pricing plan. I'd like to try this out a few times before I drop $99 on it. Do you have an option that will allow me just to try it?
Hey @rachel_cook3, thanks for checking us out! I'm afraid we don't have that option for the time being, we can offer you a 30 day money back guarantee though if that helps.
Looks interesting.
I’d like to get the Company Name + other data (address, number of employees, turnover etc) from URL’s that provide this, as opposed to individuals.
Is this possible?
@jezzacrisp thanks for checking us out! to be honest, there are other companies who are specialized to provide that kind of info (eg. clearbit). the idea for this api was to provide a simple + cheap solution for a very specific problem. hope this makes sense