Web scraping the New York Times with Browserbear

Introduction to BrowserBear

BrowserBear is a fantastic web scraping tool with a really easy but versatile API. In this Bubble tutorial video, I'm going to show you how you can add web scraping into your Bubble app. In particular, we're going to focus on how to take a screenshot. We'll be taking a screenshot of the New York Times homepage and showing how we can get that data back into Bubble.

Accessing Planet No Code Tutorials

Before I launch into that, if you're learning Bubble (which you probably are because you're watching this video), we've got hundreds of Bubble tutorial videos just like this one. Many of these can only be accessed by our members on our website planetnocode.com. Let's dive into BrowserBear.

Setting Up BrowserBear Task

I've signed up in BrowserBear and modified one of their example tasks. I'm saying let's take a screenshot of the New York Times. If I examine the steps here, I've got "go to the homepage of the New York Times" and then I simply say "take a screenshot". I do that by adding in a step.

BrowserBear's Versatility

Just to give you an idea, this is really how versatile BrowserBear is because there's all of these steps that you can add into a task. All you need to do, and I'll be demonstrating that in just a few seconds, is run this task from your Bubble app and it will run the task and return the data to your Bubble app.

Previous AI Features Tutorial

In a previous video, I've demonstrated their AI features, which is a really cool way to set up quick web scraping without having to deconstruct the page. Anyway, we're focusing on screenshots in this tutorial.

Running BrowserBear Task in Bubble

So, how do I run this task in Bubble? I'm in my Bubble API connector and I've set up BrowserBear. I've added in all of the essential fields up here including authorization, bearer, my API key, and my content type. If you're unsure where I've got this data, I've taken it out of the BrowserBear API documentation.

Setting Up the API Call

I'm on "runs, create run" and then I've taken out the essentials here. All I really need to send through to run the task is to send through a webhook because that's where I want to receive the data when BrowserBear has completed its operation.

Creating a Backend Workflow

If I go into a separate Tambam Bubble, I've set up an API workflow as a back-end workflow called BrowserBear. If I click detect data, I get this URL. Going back into my API connector, I'm adding the parameter of webhook URL and I'm pasting my endpoint for my Bubble API back-end workflow into there because that's where I'm going to receive the data when it's taken the screenshot.

Demonstrating the API Connector

I'll be demoing all of this within the API connector. Do check out our other videos such as our OpenAI tutorials where you'll see about how we can save data from an API tool, but this is just the technicality of using the BrowserBear API.

Updating Task ID

The other thing I need to update is my task ID because this is a different task from a previous video. To find my task ID, I can just click up here and click copy, then go back to Bubble, paste my task ID, and I can re-initialize the call.

Handling API Response

I get this instant response back basically because the process of taking a screenshot can take a little bit of time. That's why I have to use a webhook. I don't get the data right away back in the return values for my API call.

Checking BrowserBear Logs

I can go into BrowserBear and go into logs. Here it is. So it's still running. I'm going to go back here because I'm waiting for the data to come in. By putting it into the request Data mode or initialize, Bubble is ready to learn the structure of the content as it returns.

Troubleshooting Connection Error

We've actually got an error now. I should have tested this beforehand. BrowserBear is not able to connect to the New York Times. It's recommending I change a setting. Let's try that one and run the API again.

Reviewing API Call Results

Going into my logs, this is my most recent call. Let's see what happens here. It's now saying finished, so I'm going to go back here and this is the data that's been sent in. I'm looking for the ID of the screenshot, so I'm going to just approve it.

Viewing the Screenshot

I'm going to take this, copy it, open in a new tab, and here is The New York Times. What it's done, obviously, is scroll to get the full page. I've got this Terms or, you know, other websites might have a cookie notice.

Next Steps

I'm going to stop this video here and I'm going to have a go at using some of the additional tools in BrowserBear to try and get rid of that cookie notice or terms notice. I'm going to record a follow-up video to this right away.