Lead Machine Learning & AI Evaluation Engineer page is loadedLead Machine Learning & AI Evaluation EngineerApply locations Boston MA time type Full time posted on Posted 17 Days Ago job requisition id R12995Join us as we work to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all.RESEARCHER – AI EVALUATIONWe are looking for a Lead AI /ML Scientist/ Engineer focusing on AI Evaluation to join the Research team in the Core AI Subdivision . Evaluating LLM s and applications integrating LLMs & agents presents unique challenges compared to traditional software or machine learning models due to their inherent non-deterministic nature and the complexity of assessing the quality of their multimodal outputs . Effective verific ation & validation of LLMs and applications integrating LLMs & agents is paramount for ensuring accuracy, reliability, safety, and the user trust . You will work with the team to establish scalable methodologies, designs, and tooling to accomplish this .About you:You love to own important work and find it difficult to turn down a good challenge. You are a senior Data Scientist, are excited and knowledgeable about the latest developments in AI & ML ; and you keep abreast of the emerging models, methods , tooling, and technologies . You have experience building , tuning, evaluating , and deploying ML models at scale . You have very strong oral and written communication skills and can work with colleagues from a variety of technical and non-technical backgrounds. You enjoy both learning and teaching, and you are excited to share your expertise across the company. You love collaboration and working closely with a team of other experts and with technical and non-technical stakeholders . Finally, you have a strong interest in improving the delivery of healthcare .About the team:The Core AI Subdivision is bringing Artificial Intelligence to bear against the hardest problems in healthcare . We are working with product and engineering leaders across the company to build AI into our Best -in- KLAS suite of products. We work together with athenahealth engineers to deploy state-of-the-art machine learning models and agents .Job Responsibilities :As a member of the Research team focusing on AI verification & validation, you provide subject matter expertise , practical technical guidance, and tooling for evaluating LLMs and applications employing LLMs & agents in their workflow . Your domain includes:Leveraging standardized benchmarks for initial assessmentCalculating and interpreting quantitative metrics such as accuracy, precision, recall, F1, perplexity, BLEU, ROUGE, text similarity, exact match etc.Human evaluationConventional testing such as unit, functional and scale/ load.Model explainability & output consistencyT esting to understand b ias, toxicity, fairness.P rompt variation /robustness testingFactual accuracy/coherence/relevance/fluency/hallucination testingSecurity testingMonitoring in production (especially important given the non-deterministic nature of LLMs)Overall observability (accuracy, perf metrics, traces/explainability, cost, usage…)Techniques/approaches for improving key aspects of overall model performance such as accuracy and latency e.g., advanced prompt engineering, RAG, domain specific fine tuning, reasoning, and self-checking.Incorporating end user feedback loopsEstablishing best practice for evaluation of applications integrating LLMsAutomating as much as practical to make AI evaluation reliable, scalable , and repeatable, including integration into re-training and CI/CD pipeline sAs a member of the Research team , you will :Identify opportunities to make AI evaluation determini stic, performant , and cost-effective.Understand and follow conventions and best practices for modeling, coding, architecture, and statistics; and hold other team members accountable for doing so.Apply rigorous testing of statistics, models, and code.Contribute to the development of internal tools and Core AI team standards.Typical Qualifications:Excellent verbal communication and writing skills.Bachelors in relevant field : math, computer science, data science, economics.At least 8 years of professional experience developing and evaluating machine learning models .At least 4 years enterprise experience training , evaluating, and deploying models with a particular focus on automated evaluation pipelines .Proficient in Python.Experience using machine learning models and librariesFamiliarity with NLP , computer vision , ambient computing techniques.Experience with commercial and open-source AI evaluation tooling, frameworks, and best practices.Experience using the AWS ecosystem a bonus , including Kubernetes, Kubeflow or EKS experience.About athenahealthOur vision: In an industry that becomes more complex by the day, we stand for simplicity. We offer IT solutions and expert services that eliminate the daily hurdles preventing healthcare providers from focusing entirely on their patients — powered by our vision to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all.Our company culture: Our talentedemployees — or athenistas, as we call ourselves — spark the innovation and passion needed to accomplish our vision. We are a diverse group of dreamers and do-ers with unique knowledge, expertise, backgrounds, and perspectives. We unite as mission-driven problem-solvers with a deep desire to achieve our vision and make our time here count. Our award-winning culture is built around shared values of inclusiveness, accountability, and support.Our DEI commitment: Our vision of accessible, high-quality, and sustainable healthcare for all requires addressing the inequities that stand in the way. That's one reason we prioritize diversity, equity, and inclusion in every aspect of our business, from attracting and sustaining a diverse workforce to maintaining an inclusive environment for athenistas, our partners, customers and the communities where we work and serve.What we can do for you:Along with health and financial benefits, athenistas enjoy perks specific to each location, including commuter support, employee assistance programs, tuition assistance, employee resource groups, and collaborativeworkspaces — some offices even welcome dogs.We also encourage a better work-life balance for athenistas with our flexibility. While we know in-office collaboration is critical to our vision, we recognize that not all work needs to be done within an office environment,full-time. With consistent communication and digital collaboration tools, athenahealthenablesemployees to find a balance that feels fulfilling and productive for each individual situation.In addition to our traditional benefits and perks, we sponsor events throughout the year, including book clubs, external speakers, and hackathons. We provide athenistas with a company culture based on learning, the support of an engaged team, and an inclusive environment where all employees are valued.Learn more about our culture and benefits here: athenahealth.com/careershttps://www.athenahealth.com/careers/equal-opportunitySimilar Jobs (2)Lead Foundational Models Machine Learning Engineerlocations Boston MA time type Full time posted on Posted 17 Days AgoLead Software Engineer - Machine Learninglocations Boston MA time type Full time posted on Posted 3 Days AgoUnited by our mission and driven by our entrepreneurial spirit, our work at athenahealth is collaborative, transformative, and above all, it’s meaningful. Our employees take pride in using technology and data-driven insights to inspire changes that will make the U.S. healthcare system better for everyone, including your friends, family and maybe even you.Notice to Job Seekers/Job Candidates: Recruitment Fraud AlertPlease be aware of questionable job offers that are not affiliated with athenahealth.athenahealth has been made aware of unauthorized career opportunities offered by individuals posing as representatives of larger U.S. companies, including athenahealth. The fictitious jobs are advertised on employment-search websites, such as Indeed.com and Craigslist.com, and prospective employees are required to share their personal and financial information (e.g. credit card, bank information), provide copies of their government-issued identification, and/or send money for application fees, processing charges or work permits.The victims who are told they are "hired" are often instructed to deposit a check (which is later returned as fraudulent) into their own account and to forward overpayment to individuals - usually via wire transfer.Important information for job seekers:athenahealth has a formal application process and we do not request you to interview on a Google Hangout or via text messaging.athenahealth will never request money for the opportunity to apply or work for athenahealth.athenathealth does not require completion of tax forms, bank account or credit card information as part of the recruiting process. #J-18808-Ljbffr