Show HN: Open-source proxy server for Llama2, GPT-4, Claude2 with Logging,Cache Hello hacker news, I’m the maintainer of liteLLM() - package to simplify input/output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: https://ift.tt/7Mg4JLW We’re open sourcing our implementation of liteLLM proxy: https://ift.tt/Smn9WrH... TLDR: It has one API endpoint /chat/completions and standardizes input/output for 50+ LLM models + handles logging, error tracking, caching, streaming What can liteLLM proxy do? - It’s a central place to manage all LLM provider integrations - Consistent Input/Output Format - Call all models using the OpenAI format: completion(model, messages) - Text responses will always be available at ['choices'][0]['message']['content'] - Error Handling Using Model Fallbacks (if GPT-4 fails, try llama2) - Logging - Log Requests, Responses and Errors to Supabase, Posthog, Mixpanel, Sentry, Helicone - Token Usage & Spend - Track Input + Completion tokens used + Spend/model - Caching - Implementation of Semantic Caching - Streaming & Async Support - Return generators to stream text responses You can deploy liteLLM to your own infrastructure using Railway, GCP, AWS, Azure Happy completion() ! https://ift.tt/JFL3054 August 12, 2023 at 05:38AM
Show HN: Open-source proxy server for Llama2, GPT-4, Claude2 with Logging,Cache https://ift.tt/MZFkmL3
Related Articles
Show HN: Nextflick.io – Watch a random movie trailer https://ift.tt/NSDBOX3Show HN: Nextflick.io – Watch a random movie trailer I want to introdu… Read More
Show HN: Build your own no-code editor with Reka.js https://ift.tt/GcCgqBWShow HN: Build your own no-code editor with Reka.js Much of the comple… Read More
Show HN: Aicmd – Write difficult shell commands using natural language for free https://ift.tt/SiNY7D1Show HN: Aicmd – Write difficult shell commands using natural language… Read More
Show HN: Shhhbb, an SSH BBS https://ift.tt/XJOyjCcShow HN: Shhhbb, an SSH BBS Hello all :) I made this BBS for fun and t… Read More
Show HN: JavaScript Version of Douglas Hofstadter's Copycat https://ift.tt/xrXek3JShow HN: JavaScript Version of Douglas Hofstadter's Copycat https://if… Read More
Show HN: PromptLab–Prompt Chain Iteration for Nontechnical Users https://ift.tt/S4ce1spShow HN: PromptLab–Prompt Chain Iteration for Nontechnical Users Hey H… Read More
Show HN: Her – An AI assistant powered by ChatGPT https://ift.tt/PabQKvxShow HN: Her – An AI assistant powered by ChatGPT https://ift.tt/Ls4lw… Read More
Show HN: Pubnix.pink, a public-access Void Linux system https://ift.tt/1TB4jGUShow HN: Pubnix.pink, a public-access Void Linux system This is a hobb… Read More
0 Comments: