AI Web Scraping Structured Data - Part 2

Introduction to Web Scraping and AI in Bubble.io

This is a follow-up video showing the next steps that I took with this web scraping and AI combined no-code app that I've quickly built in Bubble.io. And it looks a bit messy here, but let me do a very, very quick recap. I'm taking a web page, I'm scraping it, converting it into markdown, that's what you see here. And then passing that web page off to Claude, and actually I've swapped in Claude Sonnet 3.5 for Claude Haiku because I don't need a really powerful model. But effectively what I'm saying is identify different parts of this job advert, for example the closing date. And I'm even asking the AI to reformat the closing date. And I get a response back that looks like this.

Addressing JSON Response Issues

Now there's a few really important things going on here that I want to point out because I've already seen comments on previous videos of people saying they can't get the JSON response from OpenAI or from Claude and get their Bubble app to pick it up because it treats it as text. But before I show you that secret sauce, click the link down in the description if you want to learn more about building no-code apps, you want to join a community of no-code builders and you want to access hundreds of our Bubble tutorial videos. There's one place to do it and that's clicking that link.

Changes Made to the App

So there are a few changes that I've made, let me go through them one by one. Firstly, I've swapped in Claude 3 Haiku and it's working fine. Next, I trawled through the Claude documentation to see if they really did have JSON mode because if you do a Google search for it, even if you look on the Bubble forum, you'll find slightly different messages. The best that I've found from Anthropic themselves is to pre-fill the message. And what they're saying is that if in your API request you begin the assistant message and you can begin it with a curly bracket, and you can see that I've done that just here, you are forcing the response to start with whatever you include in that field.

Handling JSON Responses in Bubble

So one of the issues is there can be a preamble, it can say here is your structured data as JSON and you don't want that, you want to go straight into the JSON. Now the next, let me initialise that and see what happens. Because the next issue is that the JSON comes back with all of these escaped characters. Bubble is treating it as text rather than code. Now I can't find a really lean and effective way to do that. I've got another idea about bouncing it off at a backend end point in your own Bubble app, I might demo that in a follow up to this video. But effectively we are left with this. And also notice that it doesn't start with a curly bracket because in the prompt down here we say here's the curly bracket.

Using Plugins to Extract JSON Values

So, how do we go from this actually to extracting the JSON value and then potentially being able to save it into our Bubble database? Well I've installed a plugin, let me go ahead and look which one I picked in the end. I picked... Is it that one? Oh sorry I've tried several here. Which one is it? I've literally just installed it. No it's not this one. This one here. So there are several plugins out there that claim to do this but basically I'm wanting to supply some JSON and then extract data using a particular key.

Implementing the JSON Extraction Process

So I'm using read JSON and get value by key. And so what's happening now... I'll go back here. Is I've added in these extra steps. So in the previous video we covered the web scraping using UseScraper. We save it as a custom state so that we can debug, test it on our page. We then send it off with our prompt. Here we go, so we've got some instructions, we've got some formatting in XML code formatting. We say reformat the date and we insert the results of the web scrape into that prompt. We make it all JSON safe and we save that so we can see what's going on.

Extracting Specific Values from JSON

Now this is the key bit. I've added in the read JSON... This one here. This is what that plugin gives me. I've added that in. I've added the results of Claude putting back in that initial curly bracket so now it's a complete piece of JSON. And then I say key name closing date and I'm saving the result of that string back into a new custom state. And so that is how I get the date out of that field there.

Future Improvements and Conclusion

Now I want to be able to improve this because I think that we should be able to get back all values from the JSON, not just one key at a time. If you had five values you want to take out, you probably have to put this step in there five times and change the key name each time. And then you could make changes to a thing, create a new thing, get the date around to it that way. I want to find a better way of doing it. If you know a better way, please leave a comment. Otherwise I'm going to end this video here and I'm going to try and find a better way to do it and record a follow up to this follow up.