Note: The job is a remote job and is open to candidates in USA. AnswersNow is a company focused on revolutionizing autism therapy through an innovative virtual ABA therapy platform. As a Data Scientist, you will optimize systems that transform clinical context into impactful recommendations, ensuring high-quality standards in AI solutions for autism therapy.
Responsibilities
- Architect and continuously improve the RAG pipeline that retrieves client-specific clinical context — session notes, treatment plan goals, historical performance data — and injects it into inference-time prompts
- Design the retrieval layer: chunking strategies, embedding models, vector store configuration, and retrieval ranking — optimizing for clinical relevance, not just semantic similarity
- Build a context assembly system that selects and structures the most relevant clinical information for each model invocation, given token constraints and clinical priority
- Evaluate retrieval quality rigorously: build test sets, measure recall and precision, and iterate on the pipeline based on where retrieval fails
- Design evaluation frameworks that assess AI recommendation quality beyond standard NLP metrics — working with clinical stakeholders to define what 'good' means for each use case
- Build automated evaluation pipelines that can test AI outputs at scale: LLM-as-judge evaluators, human review workflows, and clinical validity checks
- Maintain evaluation datasets that reflect the real distribution of clinical scenarios the model encounters in production
- Report evaluation results in terms that clinical and product stakeholders can understand and act on
- Systematically identify where foundation model capabilities fall short for AnswersNow's care model: what clinical reasoning the model gets wrong, what it hallucinates, what it doesn't know how to handle
- For each identified gap, recommend and implement the appropriate mitigation — improved retrieval, prompt engineering, output validation, or escalation to human review
- Stay current on foundation model capabilities and evaluate new models against our clinical requirements as they emerge
- Maintain a gap log and roadmap that gives product and clinical leadership visibility into current AI limitations and the plan to address them
- Monitor production AI outputs for quality, drift, and failure modes using the evaluation infrastructure you've built
- Define alerting thresholds and escalation paths for when AI quality falls below acceptable clinical standards
- Partner with the engineering team on observability — ensuring AI outputs are logged, traceable, and auditable
- Conduct root-cause analysis when AI quality issues are reported and drive systematic fixes
- Work closely with clinical leadership and BCBAs to understand the care model deeply enough to design AI systems that support it accurately
- Translate clinical domain knowledge into technical requirements: what context does the model need, what outputs are clinically acceptable, where does the model need to defer to the clinician
- Partner with the BI Engineer and data team on the data infrastructure that feeds the AI pipeline — session data, outcomes data, treatment plan content
- Communicate AI system behavior clearly to non-technical stakeholders: what the system does, what it doesn't do, and where human judgment remains essential
Skills
- 4+ years of experience in applied data science, ML engineering, or AI engineering in a production environment
- Deep understanding of RAG architectures: retrieval systems, embedding models, vector databases (Pinecone, Weaviate, pgvector, or similar), chunking strategies, and context assembly
- Experience designing and running evaluation frameworks for AI systems — you've thought hard about how to measure quality in domains where ground truth is ambiguous
- Strong Python skills; experience with LLM orchestration frameworks (LangChain, LlamaIndex, or similar)
- Clinical NLP experience or healthcare AI background is strongly preferred — you understand why clinical data is different from general text and what that means for AI system design
- You think like an engineer and a scientist: you build systems that can be measured, iterated on, and trusted — not black boxes
- Strong written communication: you can explain RAG pipeline design to a clinician and explain clinical requirements to an engineer
- Genuine interest in the clinical domain — you want to understand Applied Behavior Analysis well enough to build AI that actually helps BCBAs do their jobs
- Experience with Amazon Bedrock, SageMaker, or AWS AI/ML services
- Familiarity with HIPAA-compliant data handling for AI training and inference pipelines
- Background in clinical NLP, behavioral health informatics, or ABA/autism research
- Experience with fine-tuning or RLHF — even if this role doesn't require it, understanding the tradeoffs informs better RAG design
- Exposure to LLM-as-judge evaluation patterns or multi-model evaluation pipelines
Benefits
- Fully remote – work from anywhere in the U.S.
- Flexible hours with an async-friendly team culture
Company Overview