Note: The job is a remote job and is open to candidates in USA. MRO is a company focused on healthcare software products, seeking a Principal AI Engineer to lead the technical vision and implementation of AI capabilities. This role involves designing production AI systems, collaborating with various teams to create scalable and compliant AI solutions, and ensuring high-quality standards in AI product development.
Responsibilities
- Own the end-to-end technical vision for MRO Prodigy's AI layer — a production system that uses RAG, generative AI, and structured data reasoning to automate answers to Healthcare Registry questionnaires
- Define the AI roadmap for Prodigy, balancing near-term customer commitments against foundational capability investments that scale the product to enterprise maturity
- Evaluate and make build/buy/integrate decisions for AI capabilities — foundation model selection, embedding strategies, retrieval architectures, and orchestration frameworks — and own the consequences of those decisions
- Serve as the technical authority on all AI design decisions for Prodigy; produce architecture decision records, set standards, and ensure the architecture is defensible, auditable, and extensible
- Architect and evolve Prodigy's multi-modal retrieval pipeline, combining unstructured clinical document ingestion with structured EHR/FHIR data to surface accurate, citation-backed answers to registry questionnaire items
- Design and refine the answer generation layer — prompt engineering, context construction, grounding strategies, and output formatting — ensuring generated answers are clinically accurate and audit-ready
- Own the question routing and data source classification logic that maps registry questions to the right retrieval path, structured data field, or generation strategy
- Build and maintain the answer validation and confidence scoring framework, defining the statistics and quality thresholds that govern when answers are auto-accepted versus routed for human review
- Stay hands-on and close to the work: run direct ideation and feedback loops with Prodigy's end users (abstractors, registry, and quality teams) and with production analytics and monitoring systems — turning real usage signals into prioritized improvements that demonstrably move value, not just model metrics
- Evolve the feedback loop architecture that captures human corrections and routes them into continuous model improvement — ensuring Prodigy gets measurably better with every customer interaction
- Define the evals framework for Prodigy: how accuracy is measured, how regression is detected, and what signals trigger retraining or prompt revision
- Establish guardrails for hallucination detection and factual grounding specific to clinical registry use cases, where answer accuracy has direct downstream compliance implications
- Architect AI infrastructure across GCP (Vertex AI, BigQuery, Dataflow) and AWS (Bedrock), ensuring the pipeline is scalable, cost-efficient, and operationally observable
- Collaborate with data engineering to maintain high-quality, well-governed clinical and FHIR data inputs; define feature engineering and chunking strategies that optimize retrieval precision
- Define MLOps standards for Prodigy: model versioning, deployment gates, rollback procedures, drift monitoring, and audit trail requirements consistent with HIPAA compliance
- Act as the AI technical mentor for the Prodigy squad and adjacent engineering teams — guiding developers on RAG patterns, LLM integration, responsible AI practices, and clinical data handling
- Collaborate with Security and Compliance to ensure Prodigy's AI layer meets HIPAA requirements, including PHI handling in prompts, data residency, and model audit logging
- Foster AI literacy across the broader engineering organization, helping teams understand when and how to apply AI safely in a regulated healthcare context
- Partner with Product Management to translate registry workflow complexity and customer feedback into technically sound AI capability improvements
Skills
- Bachelor's in Computer Science, AI/ML, or related field
- Deep AI/ML engineering experience
- Know how HIPAA applies to the systems you build
- Owned AI product quality end-to-end — not just contributed to it
- Own the end-to-end technical vision for MRO Prodigy's AI layer
- Define the AI roadmap for Prodigy
- Evaluate and make build/buy/integrate decisions for AI capabilities
- Serve as the technical authority on all AI design decisions for Prodigy
- Architect and evolve Prodigy's multi-modal retrieval pipeline
- Design and refine the answer generation layer
- Own the question routing and data source classification logic
- Build and maintain the answer validation and confidence scoring framework
- Stay hands-on and close to the work
- Evolve the feedback loop architecture
- Define the evals framework for Prodigy
- Establish guardrails for hallucination detection and factual grounding
- Architect AI infrastructure across GCP and AWS
- Collaborate with data engineering to maintain high-quality data inputs
- Define MLOps standards for Prodigy
- Act as the AI technical mentor for the Prodigy squad and adjacent engineering teams
- Collaborate with Security and Compliance to ensure Prodigy's AI layer meets HIPAA requirements
- Foster AI literacy across the broader engineering organization
- Partner with Product Management to translate registry workflow complexity into AI capability improvements
- Strong ML / data science / statistics theory foundation
- Hands-on LLM integration: prompt engineering, grounding, citation, hallucination mitigation, and output validation
- Experience with LangChain, LlamaIndex, or equivalent orchestration frameworks
- Built confidence scoring and auto-acceptance thresholds
- Designed human-in-the-loop feedback systems
- Production experience building RAG pipelines
- Full ML lifecycle ownership in production
- Strong Python and software engineering fundamentals
- Hands-on with vector databases and hybrid search
- Built evals frameworks that measure accuracy, precision, recall, and F1
- Solid AWS and GCP experience
- Experience building pipelines over mixed unstructured and structured data sources
- Clinical NLP or healthcare AI experience
- Knows how HIPAA applies to AI systems specifically
- Familiar with AI governance in practice
- Has owned AI technical vision before
- Can write an ADR, set an engineering standard, and make it stick across teams
- Communicates tradeoffs clearly to both engineers and non-technical stakeholders
- Track record of mentoring engineers and raising AI maturity on a team
- Master's or PhD preferred — or equivalent depth proven through shipped AI systems
- Azure familiarity a plus
- FHIR/HL7 and clinical document format familiarity strongly preferred
- Prefer direct experience with clinical documentation and abstraction workflows
Benefits
- Annual cash bonus
- Medical, dental, vision, life insurance
- 401(k) plan
Company Overview
Company H1B Sponsorship