AI Web Scraping Product Hunt

Web Scraping with Browser Bear

If you're looking for a way to web scrape data into your Bubble app, then look no further than Browser Bear. Browser Bear is an exciting addition to the web scraping scene because not only does it have a very easy and powerful API, it also makes use of AI to detect which elements of the page to extract and can even loop through those and then send them all to an endpoint on your Bubble application.

Scraping Product Hunt with Browser Bear

In this Bubble tutorial video, I'm going to show you how you can scrape the front page for Product Hunt and get a list of all of the products into your Bubble app. And before I launch into that, if you are learning Bubble, you are watching this video. It's good to see you. Give us a like and subscribe. But also we have got hundreds of Bubble tutorial videos, many of which you cannot find on our YouTube channel. You can only find them on our website planetnocode.com.

Setting Up Browser Bear

Let me show you how you can use Browser Bear and just how quick and easy it is. First of all, I'm going to dive into my Browser Bear account and I'm in fact using one of the templates that the developer Browser Bears created and that is using AI web scraping and I'm going to take you through each of the steps here.

Configuring the Scraping Task

So I have a go action and it's saying go to planethunt.com and wait until the network is idle. Then I'm going to save links and basically this is effectively I'm guessing behind the scenes constructing an appropriate prompt for whichever AI Browser Bear is using and so I'm saying articles, but maybe I should go with products. Let's try products instead. And then save data. So this is a looping action because I don't want it to just save one item on the Product Hunt homepage.

Testing the API

Let's see how it treats that. Yeah, these are just the defaults that it sets out. You know, you could use this to scrape other content like blog for example. Let's just hit save on that for now. And then I can test the API through the dashboard here and I can get the results.

Advantages of AI-Powered Scraping

Now something I've already noticed is I think because it's using the AI aspect it does take slightly longer to get a response than other web scraping tools that I've used that don't have AI. Of course the advantage of using the AI is I've not had to open up my browser inspect tool, find like a class or ID that is declared for each of the elements that want to extract. This is so much quicker to do in the setup process.

Setting Up Browser Bear in Bubble

So I've got that and basically I'm now going to go into my Bubble app. Here we go. And I've set up Browser Bear in the API connector because I need to send a command to run what Browser Bear calls a task. So I've got a new API setup here. Authorization, I've added in my API key. I've added in the essential shared header content type application JSON. And then I've got run task.

Configuring the API Call

And all of this data by the way can be found. I'm simply following along by the structure that they've got here. Which is that each of your tasks that you create in Browser Bear has got an ID and you run that and then you pass in parameters pretty standard API stuff. And the parameter that we need to add in is the webhook URL because we want to say when Browser Bear has finished doing the scrape and it's extracted the data use the AI where does it send that data to? And we're going to get on to that.

Setting Up the Webhook

So I've got my task ID. Where do I find my task ID? Well, I find it up here. Copy that. And I can paste it into that. And then I'm adding in the parameter of webhook URL. Right, how do we set that up in Bubble? Well, I've got another Bubble tab open here and I've added in a back-end workflow of Browser Bear and it is public and it can be run without authentication. Obviously you need to understand the appropriate risks of doing that. But Browser Bear can't authenticate it with Bubble.

Initializing the Webhook

And then I'm going to click detect date and it gives me this URL and I'm going to take this URL and paste it into the URL here. Notice that it's got initialize on the end and I'm actually going to go back and point two other things out. So it's got initialize that is basically putting it in learning mode. And I need it to do that so that Bubble can learn the structure of the incoming data.

Handling Different Bubble Versions

Also notice that it's got version test in here and actually this is using a Bubble in your temporary domain. So just be aware of what in how this will impact your endpoints if you add your own domain in. The difference between different versions of your Bubble app such as your developer version and your live version. The webhooks are going to change. So you're going to want to make the webhook URL somehow dynamic in the way that you send it to Browser Bear because you want to swap out different bits of this.

Running the Scraping Task

But for now this will work and I'm going to go back into the API connector and I'm going to say initialize call. Okay, I get a response back basically saying that Banner Bear is running the task. And then if I just refresh here, I can go into logs and I can see that it's currently processing this scrape and I'm just making sure that this is ready in initialize mode because I'm expecting in the next 20 to 30 seconds to get that data back.

Viewing the Scraped Data

Remember that what I'm scraping from Product Hunt. I can go back into Banner Bear and I can just view what's going on here. Let's see what happens might take a few more seconds. So I've got the data back from Browser Bear and here we go. This is all of the lovely data that I scraped from Product Hunt including this link this list of content here and then I'll go browse go down here. So you can see I've got I've got LinkedIn I've got AI designer. I've got free status page you can see that these are all posts from Product Hunt.

Final Steps and Considerations

So before I wrap this video I would just point out and remind you that if you actually wanted to run this you need to remove initialize because now it's in the position to actually send data to that endpoint and of course make sure that you make your version this part here dynamic so that in your live version it's not sending data to your dev version. If you've liked this video please like and subscribe on YouTube that really helps us and remember that you can find even more Bubble tutorial videos at PlanetNoCode.com