AI Prompt Engineer
San Francisco, CA (On-Site M-F)
Our client is an early-stage, AI-native technology company building AI-powered call center and scheduling agents.
About the Role
As an AI Prompt Engineer, you will own critical behavioral slices of production voice agents used, shaping both shared and customer-specific behaviors across thousands of calls. You will design prompts, sub-agent architectures, and evaluation harnesses to iteratively improve automation, booking, and resolution rates using real call data.
Responsibilities
- Write, maintain, and version-control production prompts for intent classification, information extraction, scheduling and availability negotiation, verification flows, objection handling, and edge-case recovery across all customer deployments.
- Review failed or low-performing calls daily, identify root causes, and ship targeted prompt or configuration updates multiple times per week to measurably improve automation and booking metrics.
- Design and manage sub-agent architectures (e.g., routing, specialist agents, fallback handlers) that support complex multi-turn healthcare workflows while maintaining latency and reliability requirements.
- Build and maintain offline evaluation harnesses, including curated eval sets, automated prompt optimization workflows (e.g., GEPA-style approaches), and regression test suites for safe shipping of changes.
- Collaborate on human-in-the-loop onboarding flows that translate practice-specific intake forms, scheduling rules, and quirks into robust agent configurations, and define customer-specific evaluation metrics.
- Simulate real-world caller scenarios, monitor live production performance dashboards, detect drift or degradation early, and coordinate fixes with engineering and operations.
- Partner closely with software engineers to integrate prompts and agents into the broader AI stack, ensuring clean interfaces, observability, and reliable deployments in a high-volume environment.
Qualifications
- 2+ years of experience with AI/ML, NLP, or prompt engineering in production, including hands-on work shipping prompts or agents that real users relied on.
- Demonstrated experience writing, testing, and iterating prompts for tasks such as classification, information extraction, scheduling, or conversational flows in high-stakes or operational contexts.
- Strong analytical, data-driven mindset with comfort designing experiments, reading dashboards, and justifying changes with metrics (e.g., conversion or booking rate improvements).
- Excellent writing skills, including sensitivity to tone, register, and phrasing in spoken or TTS-delivered interactions.
- Comfort reading in Python and working familiarity with TypeScript, with the ability to collaborate effectively using modern AI coding tools.
- On-site availability five days per week in the San Francisco Bay Area.
Preferred Skills
- Prior experience with voice AI, TTS, ASR, or telephony platforms and real-time conversational systems.
- Automated prompt optimization experience using frameworks or approaches such as DSPy, GEPA, or similar techniques.
- Experience building and maintaining evaluation suites, test harnesses, or CI pipelines for LLM-based agents.
- Academic or practical training in linguistics, philosophy, cognitive science, or related fields that inform language and conversation design.
