GenAI Platform Engineer – LLM Gateway
- Market p25–p75
- $26k–$98k
- Jobgether p25–p75
- $44k–$114kn=4
- Opened · 30d
- +9,081
- Closed · 30d
- −6,473
- Signal
- ▲ Expanding+2,608 net
- Opened · 7d
- +2798% of open
- All backend
- 8,067open
Accountabilities:
- Design, develop, and operate enterprise LLM Gateway features, primarily using Python, enabling secure and scalable access to multiple large language model providers.
- Build and maintain multi-provider model integrations, intelligent routing, failover mechanisms, API key management, and access control capabilities.
- Develop platform features for usage metering, budget management, cost transparency, and chargeback reporting.
- Integrate agentic AI capabilities into the platform while ensuring appropriate security and governance controls.
- Develop and maintain Model Context Protocol (MCP) servers, cross-provider tool calling, and agent-to-agent connectivity.
- Maintain and enhance the platform's open-source foundations by assessing upstream releases, managing custom extensions, and delivering upgrades without introducing regressions.
- Contribute to administrative interfaces, self-service portals, developer experiences, and technical documentation.
- Build and maintain automated quality gates within CI/CD pipelines, including unit, integration, contract, and performance testing.
- Implement observability solutions using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
- Monitor platform performance and proactively identify and resolve latency, reliability, error-rate, and infrastructure cost issues.
- Participate in production support and on-call rotations, leading root-cause investigations and resolving complex incidents.
- Conduct blameless postmortems and translate incident findings into sustainable engineering improvements.
- Apply secure-by-default engineering practices, including Single Sign-On (SSO), OAuth 2.0/OIDC, Role-Based Access Control (RBAC), and secure secrets management.
- Maintain dependency security, version pinning, and software supply-chain hygiene.
- Evaluate emerging generative AI technologies, including model routing solutions, guardrails, agent registries, and agentic frameworks, and provide recommendations for adoption.
- Support internal engineering teams in onboarding and effectively using the platform through technical guides, runbooks, code samples, and reusable AI agent skills.
- Contribute to InnerSource initiatives, participate in code reviews, and collaborate with open-source communities and technology vendors.
- Identify opportunities to improve platform capabilities through upstream contributions, technical experimentation, and continuous innovation.
Requirements
- Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
- At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
- Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
- Hands-on experience integrating LLM provider APIs, such as OpenAI, Azure OpenAI, and AWS Bedrock.
- Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
- Proven experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
- Practical knowledge of Docker, GitHub Actions, and automated testing integrated into software delivery workflows.
- Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
- Demonstrated ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
- Excellent written and verbal communication skills in English, with the ability to collaborate effectively across globally distributed teams.
- Ability to work independently, take ownership of technical deliverables, and contribute to cross-functional engineering initiatives.
Preferred qualifications and experience:
- Experience working with AI/LLM gateways or API gateways, particularly model routing, API key management, access controls, and cost tracking.
How Jobgether works
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1