AI Web Scraping Structured Data - Part 2
Learn one method of extracting values from JSON returned by an AI like Claude by Anthropic.
Unlock AI-powered web scraping: Learn how to extract structured data from any webpage using Claude and Bubble!
Master JSON handling in Bubble: Discover the secret sauce for seamlessly integrating API responses into your no-code app.
Supercharge your Bubble workflows: Explore advanced techniques for custom events and data manipulation in this step-by-step tutorial.
Introduction to Web Scraping and AI in Bubble.io
This is a follow-up video showing the next steps that I took with this web scraping and AI combined no-code app that I've quickly built in Bubble.io. And it looks a bit messy here, but let me do a very, very quick recap. I'm taking a web page, I'm scraping it, converting it into markdown, that's what you see here. And then passing that web page off to Claude, and actually I've swapped in Claude Sonnet 3.5 for Claude Haiku because I don't need a really powerful model. But effectively what I'm saying is identify different parts of this job advert, for example the closing date. And I'm even asking the AI to reformat the closing date. And I get a response back that looks like this.
Addressing JSON Response Issues
Now there's a few really important things going on here that I want to point out because I've already seen comments on previous videos of people saying they can't get the JSON response from OpenAI or from Claude and get their Bubble app to pick it up because it treats it as text. But before I show you that secret sauce, click the link down in the description if you want to learn more about building no-code apps, you want to join a community of no-code builders and you want to access hundreds of our Bubble tutorial videos. There's one place to do it and that's clicking that link.
Changes Made to the App
So there are a few changes that I've made, let me go through them one by one. Firstly, I've swapped in Claude 3 Haiku and it's working fine. Next, I trawled through the Claude documentation to see if they really did have JSON mode because if you do a Google search for it, even if you look on the Bubble forum, you'll find slightly different messages. The best that I've found from Anthropic themselves is to pre-fill the message. And what they're saying is that if in your API request you begin the assistant message and you can begin it with a curly bracket, and you can see that I've done that just here, you are forcing the response to start with whatever you include in that field.
Handling JSON Responses in Bubble
So one of the issues is there can be a preamble, it can say here is your structured data as JSON and you don't want that, you want to go straight into the JSON. Now the next, let me initialise that and see what happens. Because the next issue is that the JSON comes back with all of these escaped characters. Bubble is treating it as text rather than code. Now I can't find a really lean and effective way to do that. I've got another idea about bouncing it off at a backend end point in your own Bubble app, I might demo that in a follow up to this video. But effectively we are left with this. And also notice that it doesn't start with a curly bracket because in the prompt down here we say here's the curly bracket.
Using Plugins to Extract JSON Values
So, how do we go from this actually to extracting the JSON value and then potentially being able to save it into our Bubble database? Well I've installed a plugin, let me go ahead and look which one I picked in the end. I picked... Is it that one? Oh sorry I've tried several here. Which one is it? I've literally just installed it. No it's not this one. This one here. So there are several plugins out there that claim to do this but basically I'm wanting to supply some JSON and then extract data using a particular key.
Implementing the JSON Extraction Process
So I'm using read JSON and get value by key. And so what's happening now... I'll go back here. Is I've added in these extra steps. So in the previous video we covered the web scraping using UseScraper. We save it as a custom state so that we can debug, test it on our page. We then send it off with our prompt. Here we go, so we've got some instructions, we've got some formatting in XML code formatting. We say reformat the date and we insert the results of the web scrape into that prompt. We make it all JSON safe and we save that so we can see what's going on.
Extracting Specific Values from JSON
Now this is the key bit. I've added in the read JSON... This one here. This is what that plugin gives me. I've added that in. I've added the results of Claude putting back in that initial curly bracket so now it's a complete piece of JSON. And then I say key name closing date and I'm saving the result of that string back into a new custom state. And so that is how I get the date out of that field there.
Future Improvements and Conclusion
Now I want to be able to improve this because I think that we should be able to get back all values from the JSON, not just one key at a time. If you had five values you want to take out, you probably have to put this step in there five times and change the key name each time. And then you could make changes to a thing, create a new thing, get the date around to it that way. I want to find a better way of doing it. If you know a better way, please leave a comment. Otherwise I'm going to end this video here and I'm going to try and find a better way to do it and record a follow up to this follow up.
Can't find what you're looking for?
Search our 300+ Bubble tutorial videos. Start learning no code today!
Flexible Pricing Plans to Fit Your No-Code Journey
Choose the plan that aligns with your goals and start building your startup today.
Have questions?
We have answers!
Find answers to common questions about our membership plans, programs, and more.
Both plans offer full access to our learning resources, community, and support. The Annual plan provides a significant discount (over 15%) compared to paying monthly, and it allows you to lock in your rate for a full year.
Absolutely! You can easily upgrade or downgrade your membership plan at any time by logging into your account and selecting the desired plan. Any unused portion of your current plan will be prorated and applied to your new plan.
As a Planet No Code member, you'll receive a discount on our Bubble coaching sessions. Monthly members receive a 10% discount, while Annual members receive a 17.5% discount. To redeem your discount, simply log into your account and book a coaching session through our platform.
Our 8-week intensive mentorship program is designed to provide personalized guidance and support to help you accelerate your startup journey. You'll be matched with a startup expert who will work with you one-on-one to set goals, overcome challenges, and make rapid progress.
To apply for the Mastery Program, simply click the "Request Invitation" button on our pricing page and fill out the application form. Our team will review your application and schedule a call with you to discuss your goals and determine if the program is a good fit for your needs.
We accept all major credit cards, including Visa, Mastercard, American Express, and Discover.
While we don't offer a free trial, we do provide a 14-day money-back guarantee. If you're not completely satisfied with your membership within the first 14 days, simply contact our support team, and we'll issue a full refund.
If you decide that Planet No Code isn't the right fit for you, you can cancel your membership at any time by logging into your account and navigating to the subscription management page. Click the "Cancel Membership" button, and your membership will be terminated at the end of your current billing cycle.