Web Scraping Reddit Search Results into a Bubble.io app

Introduction to Web Scraping with Browse AI

If you've ever wanted to scrape search results from Reddit just like this, I'm going to show you how. I'm going to show you how we can use a web scraper called Browse AI. Here's the homepage. And they're a great web scraper because not only can you scrape individual pages and highlight the particular content you want to scrape from them, you can also monitor pages and you get updates through to your app. Now they've got a series of pre-built robots.

Creating an AI-Powered Scraping Robot

We're going to be using one of those in this demo. And I'm going to show you each step of how we can create an AI powered scraping robot in Browse AI, how we can run that from our Bubble site, and then how we can get the data back into our Bubble app using webhooks. All of that will be covered in this video. But before I launch into it, if you're learning Bubble and you want to watch more tutorials just like this one, then you can click the link down in the description to get access to our full tutorial library on our website, planetnocode.com.

Setting Up the Browse AI Robot

So let's go into our Browse AI account and we'll choose a pre-built robot. And I'm going to use Reddit, extract posts from search and say use this automation. Okay, search I'm going to use is called Bubble.io. This is just a demo. We want to get things working, check. It's all working. And because I'm going to keep my credit spend on the platform quite low as I'm going to be doing a series of demonstrations like this one, I'm going to say Max post five and then we start the extraction.

The Reliability of Web Scrapers

Now they're a good and they're bad web scrapers. You're basically doing battle with any defenses that the website owner has put in place to try and stop web scraping. And you'll find different platforms and web scrapers are more reliable than others. In my testing or Browse AI seems to be quite impressive. In fact, we get a response back very quickly.

Integrating with Bubble Using Webhooks

So how do we go about adding this data into our Bubble app? Well, we can go into integrate and we can set up a webhook. And a webhook is a way of saying when this task is finished in Browse AI, send a success message and all of the data through to Bubble. So I'm going to go into my Bubble app and I need to be in backend workflows.

Setting Up the Backend Workflow

This is a bit of a cluttered app. I use it for a lot of demos and I need to select backend workflows at the bottom. If you don't see that, you have to enable it in settings. But let's create a backend workflow because this is going to be a place that Browse AI can send data to. So I'll say web scrape, Reddit, and then I need to put it into what I like to call training mode.

Configuring the Webhook

So this is where the data is going to be sent. And notice in particular that at the end it's got initialize. This is when it is in training mode. We actually need to ensure that we go back and remove initialize when we actually want this to run automatically behind the scenes. So I've copied that into my clipboard and I'm going to paste in here and add in the webhook.

Running the Task and Receiving Data

So when task is finished, data is going to be sent to that webhook. I now just need to rerun the task, but I'm going to make sure that I'm in this detecting request data mode, otherwise it will be a waste and I'll just say run task. So it's going to show us once more what's going on. It's added to a queue and hopefully we'll get back a result just as quick as before.

Analyzing the Received Data

That one took slightly longer than the first demo, perhaps because of, like I say, doing battle with, in this case Reddit, who wants to protect the data on their site. And it seems like Browse AI had to retry a couple of times, but we've got the data. So hopefully that now means that if I go back into Bubble, we've got this update page and this is saying, here is all of the data and this is training Bubble, the structure of the data to get back.

Saving Data to the Bubble Database

And most importantly for us, we've got these, this list of posts. So we've got, yeah, here we go, alternative to Bubble and we've got all of this data about each post. So I'm going to click save because that's training my Bubble app what to expect back in the, back in the API data that sent through. So how do I save this to the database? Well, I'm going to just create a new one called web scrape once I make it private. No, doesn't matter. And then I'm just going to put in the bare basics. I'm just going to say title, which is text, and then URL, which is also text.

Creating a Workflow to Save Data

So now if I go back to my workflow, I can create a new thing, web scrape, add in all fields, and then here's the key thing, it gave me a list of data. And we've got other tutorials demonstrating how you can effectively create a loop in Bubble using another backend workflow. But for the sake of this, I'm just going to save the first result. So I look for captured list posts. I'm going to say first item title, first item post.

Testing the Data Flow

Okay, now let's run the test and see if we actually get data through into our Bubble database. But remember what I said before, which is that we have to update our webhook and remove, initialize. So we can't update, we can only delete. So I'm going to delete that, paste it back in, remove, initialize and add it in. And now let's try again.

Verifying the Data in Bubble

So we will run exactly the same. And what we're expecting to see in our Bubble database is that we get data created in that web scrape data type. So nothing's happened just yet. So let's go back here and it's queuing up and I'm going to skip ahead to when it's done, right, seems to be done. We've got our results back from Browse AI and hopefully this means that if we go into Bubble and perhaps we need to refresh this, we will see data in web scrapes.

Troubleshooting and Fixing Issues

Okay, we've got, we've got an issue, so let's go ahead and debug it. So I'm going to integrate. Oh, I know exactly why this is. Right, it's failed. Okay, I overlook something here, which is we have to basically say that this can be run without authentication. That's it. Otherwise you have to supply an API key, probably not secure in this case to just put an API key in another platform like Browse AI.

Retrying the Process

So let's just go ahead and run that whole thing again and hopefully this time it will be a bit more successful. So run a task. Okay, we got the results back. Hopefully this time we'll see them in our Bubble app. So once more I'm going to hit refresh and, and there we go. After a slightly slow load there from Bubble, we can see that we have successfully saved the title and URL into our Bubble app.

Conclusion and Next Steps

So there you go. That's how you can use Browse AI to scrape posts from Reddit based on the search results and save them into your Bubble app. Here's how you can trigger a AI web scraping robot that you've built in Browse AI and you can trigger it from your Bubble app. We need to be using the Browse AI API to do this and so I'm on my extract post from Reddit search, but it's going to be the same whichever one you use. You just have to swap in the robot id and so it gives us the data here to send across.

Using the Browse AI API

And I'm going to open up the API documentation and go into robots. And we want to tasks, yeah, task, run a robot. So this is what we need to send across to Bubble. So I'm going to copy, here we go, put it into the shell version. So we have to send a post request to this URL. So I'm going to copy that and go into my Bubble app. And we use the API connector to connect to services like this.

Setting Up the API Connector in Bubble

So go down here, add one in, call it Browse AI, and then we're making a post request to this URL, but we have to swap in the robot id. So let's make that dynamic robot id. So it says there how I can add in params into it, making it dynamic. I'm unticking private just so that if I was in the workflow, private is whether this data is accessible to the user. It doesn't really matter.

Configuring API Parameters

Think in this case if my users get access to my robot id, but that would allow me to dynamically swap in in a workflow if I wanted to. So let's paste in the robot id that we're working with and go back to the documentation. So I have to supply the robot id. I've done that. I have to authorize a bearer API key in the header. So in Bubble, the way I do that is I say private key and header authorization and then I would write in bearer and I'm going to paste in my API key, which I've got, I think it's this one here. Of course I'll be deleting this before publishing this video.

Adding API Action to Workflow

We'll call this one scrape. Content from Reddit needs to be an action so that I can run it in the workflow. And we have to include some parameters. So let's go back to the documentation and just see how we do that. So the two property names, does it give it an example here? Okay, right. So data input property, I'm going to copy all of that. Just so I have an idea of the layout.

Finalizing API Configuration

Paste it, that can probably add into parameters here. So the parameters we need to add in are here. We need search text and we need max posts. So let's try that, say search text. Ah, no, this isn't going to work because it's got to have, it's nested inside this key here, input parameters so I can't put it in there. So I've got what was it, search text and this is where I said Bubble.io and then there is another one I guess. Yes that would be a comma and that would be Maxposts.

Testing the API Connection

Now that's a number, but I think it probably wants me to send it as a string. Let's just try that. So I'm going to see if I get an error and I do that by running initialize call. No, I get a success code and so I'm going to click save and just so I'm really quick to show you what's going on here, if I go back and then into run tasks, we should see a new task is running. Yes, created by the API and it's running now.

Understanding Webhooks and Data Flow

So if I have my webhook set up, that data will come through to my webhook as soon as it's finished running. Because remember it can take a web scrape per 5 seconds or if it's got to change some settings spoof being a user, it can take a few minutes to get a result back. That's why we use webhooks, so that our Bubble app isn't waiting and when Browse AI has got the data, it's going to send it through to our Bubble app, through to that backend workflow using a webhook.