Back to work

LLM Gateway Kit

Cost and reliability layer for Swift apps, with patterns from the Shelfie gateway · AI Infrastructure

A Swift package that sits between an app and its LLM providers. It caches paraphrased prompts and similar images, falls back across providers, and refuses a call once a daily spending cap is hit. Its patterns come from the gateway in Shelfie, which runs its own server-side version.

View source on GitHub

limitations

Key features

  • Prompts are matched on embedding similarity, images on a perceptual hash
  • A cheap model runs first; a circuit breaker, a thrown error or a low confidence score escalates to the next tier
  • The budget check runs before the request goes out, with separate caps for identified and anonymous callers

Impact

A runaway loop or a burst of concurrent calls runs out of calls at the cap, because each call reserves its estimate first. The kit has no published cost benchmark, and the README says savings depend on your traffic.

Tech stack

Swift, Swift Package Manager, GitHub Actions

interested in something like this?

Get in touch