Convert a URL to LLM friendly input

Introduction

In today's digital landscape, leveraging large language models (LLMs) for web content analysis can offer substantial advantages. One of the effective ways to do this is by converting website URLs into Markdown format, making the content LLM-friendly. This blog post explores a straightforward approach using a service called Reader by Jina, and demonstrates how to integrate this process into a Bubble app for real-time usage.

Using Jina Reader for URL Conversion

Converting a URL into Markdown with Jina Reader is incredibly simple. Even without an API key, you can experiment with the service directly from their webpage. This serves as a valuable tool, especially if you want to process content from competitors' websites and run analyses using an LLM.

To grasp the power of Jina Reader, let’s explore its application within a Bubble app. Here, we’ll illustrate how to set up a simple form using the Bubble API connector that outputs results in Markdown format. This is especially useful for users interested in comparing website content with the assistance of an LLM.

Step-by-Step Guidance

1. Initial Setup: Begin by appending the target URL to the end of Jina's service endpoint (rgena.ai). If you forego an API key, brace yourself for rate limitations, but it's adequate for testing purposes. For the demo, run a test to see the structured Markdown data output.
2. Integrating with Bubble API Connector: Navigate to your Bubble app's API Connector, add Jina as a new service, and incorporate it as an action within a workflow. This allows user input to be dynamically converted into a Markdown format.
3. Creating a User Interface: Construct a new page in your Bubble app titled "Convert URL to Markdown". Although design won't be the main focus, using rows and columns is recommended for an optimized layout. The page will contain an input field for the URL, a button to trigger the request, and a text area to display the Markdown result.
4. Custom States for Temporary Data: Use custom states within Bubble to temporarily store and display data, avoiding database save operations. This ensures quick updates during user interaction, though the data will be lost upon page refresh or navigation.
5. Handling the Response: After initializing the API call, parse the returned text into the designated custom state. This setup facilitates displaying the Markdown directly on your app page.

Advanced Applications and Considerations

For more complex web scraping requirements, consider scraping results pages from platforms like Airbnb. Be cautious as many websites have anti-scraping mechanisms. For advanced tasks like extracting specific data points (such as ratings) and returning JSON structured data, tools like Claude’s JSON mode can simplify these processes.

Explore our video tutorials and guides for deeper insights on using AI APIs effectively, particularly with OpenAI and Bubble.