That is the question I have been looking at for the past 2 weeks. I have been using Airflow for 2 projects since 2018 and I have been mostly happy with it (thanks in part to great colleagues who made it work). But recently while I was doing research for a new use case I stumbled on a Flyte demo that caught my attention quickly. The specific feature (a) that drove my interest initially was the dynamic DAG architecture. ...
Installing Flyte on Google Cloud Kubernetes (GKE)
A step-by-step installation guide Are you looking to set up Flyte in your environment? The official Flyte documentation is great and the website has its own LLM powered chatbot. However Flyte is still in active development (currently a Linux Foundation graduate project) so there are a few points that require clarifications. Personnaly I feel I spent a bit too much time trying to sort out my first installation on GKE and none of the main LLMs proved very helpfull so I thought it was best to share more details. ...
Invoke OAuth protected FastAPI app from Google Cloud services
The problem is the following: We need to invoke a Fast API app running on GC Cloud Run from some other GC service (eg PubSub in our own case). However we have setup OAuth protection of our API routes (eg using Clerk in our own case) and backend services cannot signin to our OAuth provider. The solution is to allow for multiple authentication methods at the level of our REST API, something that FastAPI supports. ...
Google Cloud outbound traffic with a fixed IP address
When using Cloud Run we can send queries to the internet but by default the IP address will be dynamic. However when querying external services (egress) such as third-party providers or services hosted outside of our Google Cloud project we often need a fix IP address that we can white-list (instead of leaving the traffic open to the whole internet). How do we change this ? We can use a Serverless VPC Access Connector and Cloud NAT (Network Address Translation). The good news is that it is simple and quick (less than 15 minutes if you do it via the web console) to do. ...
Running ThetaData server on GKE
If you are reading this page you probably know that ThetaData is a market data (e.g. stocks, options) vendor. You interact with ThetaData via a REST API (or a WebSocket) but you need to be running the server locally. The reason being that between your local server and ThetaData data centers they use a proprietary protocol to speed up data transfers. So there is an extra layer of complexity compared to “pure REST API” vendors to make queries faster. But is it really faster ? If you are pulling large chuncks of data such as the option chain (all expiration dates and strikes) of a liquid stock like Apple yes it feels much faster than the other vendors I tried. ...
Multi Region Load Balancer for Google Cloud Run
In this example we assume we are working on a suite of backend servives as part of a larger travel booking app. More precisely we assume we have 2 apps (REST APIs) deployed on Cloud Run: Pricing deployed in both europe-west1 and us-east1 regions Booking deployed in europe-west1 region The reason we are deploying the pricing api in 2 regions is for the web-site to respond quicker when users are inquiring/comparing different options by having the queries running closer to where they are. The final booking stage is less sensitive to speed so we only deploy it in one place. ...