Job Title: Prompt / LLM Ops / AI Systems Engineer
Role Summary
We are seeking a versatile engineer to bridge the gap between creative prompt design and robust system
orchestration. You will own the design, evaluation, and production operation of our foundation model-based
systems. As an LLM Ops Specialist, you will be responsible for building the infrastructure that makes these systems
scalable, optimized and observable.
Core Responsibilities
System Architecture amp; Orchestration:
- Design end-to-end compound AI systems integrating multiple models, retrieval systems, and agentic
workflows.
- Implement Retrieval-Augmented Generation (RAG) pipelines with optimized chunking, vector
embeddings, and re-ranking strategies.
Prompt Engineering amp; Versioning:
- Craft structured, few-shot, and chain-of-thought prompts to reduce hallucinations and ensure predictable
outputs.
- Develop and manage a Prompt Registry to track versioning and A/B test results.
LLM Operations (LLM Ops)
- Observability: Instrument real-time monitoring for LLM-specific metrics, such as:
- Time to First Token (TTFT)
- Token throughput
- Drift detection
- Cost Control: Implement semantic caching and intelligent model routing to reduce API spend.
- CI/CD for AI: Build evaluation suites and methodologies to test for regressions on every deployment.
Core Tech Skills
- Python Proficiency
- Hands-on with Programmatic Prompting Frameworks
- Experience with Retrieval-Augmented Generation (RAG) Pipelines
- Experience with Structured Data Control and Integrations into other Software Systems
Mandatory Soft Skills
- Excellent Written and Spoken Communication Skills, to the Degree of Clinical Precision
- Excellent Tech-to-Business Translations
- Analytical Thinking pertaining to Problem Decomposition and Adversarial Thinking
- Collaborative Agility - Must have ability and willingness for Cross-Team Functionality and Faster Feedback
Integration Loop
- Constant Experimentation Mindset and High Degree of Learning Agility
This job is provided by Shine.com
