Overview

Core AI engineer for a global B2C product (1M+ users). Built production LLM infrastructure handling 2K+ concurrent users and 20 calls/sec with 10× cost optimization. Implemented custom workarounds for open-source LLMs during the Llama 2 era (2023).

Key Responsibilities & Achievements

• Product Collaboration: Shaping the features, oppotunity and implementation on the product.
• LLM Pipelines: Developed complex LLM pipelines for interactive, real-time chat features and companion applications. This included creating innovative workarounds for the limitations of available open-source LLMs to meet product requirements.
• LLM Deployment: From serverless to self-hosting to self-optimization to using an external inference provider, reduce the cost by 10x.
• Scalability: Designed systems to support 2,000+ concurrent users.

Impact

Contributed to the successful launch and scaling of a global B2C product with innovative LLM-powered features, serving more than 1M users, and 2,000+ concurrent users while lower the cost at 10x times.