When AI Breaks! Cloudflare AI Gateway to the Rescue for Apps

Introducing Cloudflare AI Gateway

So you've built your AI powered SaaS app and you've launched it and you've got users and you know that AI is a mission critical part of your app. Wouldn't it be amazing to have the peace of mind that if the OpenAI API went down you could just fall back on using Claude by Anthropic or if Claude was no longer available you could swap over to OpenAI and that your app would handle those changes? It would have an AI fallback. Well, I'm going to show you something that I can't believe I've only just discovered in this video. But before I go into that if you're wanting to build no code AI powered apps then you should definitely after watching this video click the link down in the description. Because we've got hundreds of Bubble no code app building tutorials on our website planetnocode.com.

Cloudflare AI Gateway: A Game-Changer for AI Apps

Here is what I found and what I've been experimenting with in the last 24 hours. It is called the Cloudflare AI gateway. Now if you've been doing web development web design as long as I have you will definitely have heard of Cloudflare. But if you haven't, Cloudflare is kind of the gateway to the internet for many people not like your ISP. But in the sense that so many websites sit with Cloudflare sitting between you, Cloudflare and then the website. In fact, they've got a really cool demonstration about generating cryptography keys using lava lamps. It's well worth checking that out.

Key Features of Cloudflare AI Gateway

Anyway, I digress. One of the more recent offerings that Cloudflare has added in is this AI gateway and it effectively sits in the middle between your app and your AI provider. And you can connect directly through to OpenAI and you can get access to a whole load of useful data such as logs such as caching. At me, what if someone was to ask the same question to your AI? It doesn't need to check each time or can you just provide the cached answer? You can all do that with the AI gateway. But what I want to draw your attention to right now is this ability to provide fallbacks.

Setting Up AI Fallbacks

If you've worked with Bubble API connector, you've worked with AI APIs. This will all look fairly familiar to you. So all we're doing is saying we're going to connect to a particular endpoint that we get and we can copy straight out of our Cloudflare account once you've set this up and then we are saying here first of all use the workers AI so this is actually AI LLMs hosted on Cloudflare in this case llama, but this could be OpenAI and so we're saying first run this query through Llama, but what if llama fails? Well, we just add in another one. So then we're saying immediately if llama fails then use OpenAI and according to the documentation here you can just add as many fallbacks as you like just by adding another object to the array.

Benefits for Production Apps

So if you want that peace of mind and you have a production ready app and you've got users on it and you don't want them to be guessing all of these errors when one of these AI providers goes down you can use the Cloudflare AI gateway to add a fallback AI. You can have multiple fallback AIs so that you get working AI to your users consistently.

Future Explorations

I'm going to do more videos on this including rate limiting custom metadata getting log data through Cloudflare AI gateway. Like I said at the start of this video I can't believe I've only just heard of this and only just playing with it. So it's well worth checking out.