Director of AI Model Evaluation & Quality Validation - LLMs & Generative AI Seattle, WA

Company: Truveta Inc
Location: Seattle
Posted on: November 15, 2024

Job Description:

Director of AI Model Evaluation & Quality Validation - LLMs & Generative AISeattle, WADirector of AI Model Evaluation & Quality Validation - LLMs & Generative AITruveta is the world's first health provider led data platform with a vision of Saving Lives with Data. Our mission is to enable researchers to find cures faster, empower every clinician to be an expert, and help families make the most informed decisions about their care. Achieving Truveta' s ambitious vision requires an incredible team of talented and inspired people with a special combination of health, software and big data experience who share ourcompany values . Truveta was born in the Pacific Northwest, but we have employees who live across the country. Our team enjoys the flexibility of a hybrid model and working from anywhere. In person attendance is required for one week during the year for Truveta Planning Week. For overall team productivity, we optimize meeting hours in the pacific time zone. We avoid scheduling recurring meetings that start after 3pm PT, however, ad hoc meetings occur between 8am-6pm Pacific time. Who We Need Truveta is rapidly building a talented and diverse team to tackle complex health and technical challenges. Beyond core capabilities, we are seeking problem solvers, passionate and collaborative teammates, and those willing to roll up their sleeves while making a difference.If you are interested in the opportunity to pursue purposeful work, join a mission-driven team, and build a rewarding career while having fun, Truveta may be the perfect fit for you. This Opportunity We are seeking a highly skilled and experienced Machine Learning Expert to join our team and evaluate the effectiveness of our AI models utilizing Large Language Models (LLMs). The ideal candidate will have a strong background in machine learning, natural language processing (NLP), and a proven track record of working with LLMs. You will be responsible for assessing the performance, accuracy, and efficiency of our AI models, identifying areas for improvement, and providing actionable insights to enhance their effectiveness. You will work in an exciting and fast-paced environment, collaborating closely with multiple teams across the company. You will work as part of an organization that brings together talent from diverse backgrounds including software engineering, big data, machine learning and AI, and clinical experts , making our team an exciting place to work. We value and encourage diversity in the belief that our differences make us and our products better. In This Role, You Will:

• Evaluate the performance of AI models that leverage Large Language Models (LLMs) in various applications.
• Develop and implement metrics to measure the accuracy, efficiency, and overall effectiveness of LLM-based AI models.
• Conduct thorough analyses to identify strengths and weaknesses in model performance.
• Provide detailed reports and recommendations to improve model accuracy, efficiency, and scalability.
• Collaborate with the development team to refine and enhance AI models based on evaluation results.
• Stay up-to-date with the latest advancements in machine learning, NLP, and LLMs to ensure our models remain at the forefront of the industry.
• Ensure that models meet business requirements and deliver tangible value to our clients. Key Qualifications
  • Master's or Ph.D. in Computer Science, Machine Learning, Artificial Intelligence, or a related field.
  • 7+ years of experience in machine learning, with a focus on natural language processing and large language models.
  • Proven track record of evaluating and improving AI models, particularly those involving LLMs.
  • Strong programming skills in Python and experience with machine learning frameworks such as TensorFlow, PyTorch, or similar.
  • Familiarity with performance evaluation metrics and techniques for LLMs.
  • Excellent analytical and problem-solving skills.
  • Ability to communicate complex technical concepts to non-technical stakeholders. Preferred Qualifications
    • Experience with popular LLMs such as GPT, BERT, T5, or similar.
    • Knowledge of deployment and scaling of AI models in a production environment.
    • Previous experience working in a fast-paced, agile development environment.
    • Contributions to the AI/ML community through research papers, conference presentations, or open-source projects. This position offers a unique chance to work on state-of-the-art language models and contribute to transformative research with the vision of Saving Lives with Data. You will be part of a diverse and team who are passionate about pushing the boundaries of machine learning and natural language understanding in the healthcare domain. Join us and make a significant impact on the future of healthcare and patient well-being. Why Truveta?Be a part of building something special. Now is the perfect time to join Truveta. We have strong, established leadership with decades of success. We are well-funded. We are building a culture that prioritizes people and their passions across personal, professional, and everything in between. Join us as we build an amazing company together.We Offer:
      • Interesting and meaningful work for every career stage
      • Great benefits package
      • Comprehensive benefits with strong medical, dental and vision insurance plans
      • 401K plan
      • Professional development & training opportunities for continuous learning
      • Work/life autonomy via flexible work hours and flexible paid time off
      • Generous parental leave
      • Regular team activities (virtual and in-person as soon as we are able)
      • The base pay for this position is $230,00 to $260,000. The pay range reflects the minimum and maximum target. Pay is based on several factors including location and may vary depending on job-related knowledge, skills, and experience. Certain roles are eligible for additional compensation such as incentive pay and stock options.If you are based in California, we encourage you to read this important information for California residentslinkedhere. Truveta is committed to creating a diverse, inclusive, and empowering workplace. We believe that having employees, interns, and contractors with diverse backgrounds enables Truveta to better meet our mission and serve patients and health communities around the world. We recognize that opportunities in technology historically excluded and continue to disproportionately exclude Black and Indigenous people, people of color, people from working class backgrounds, people with disabilities, and LGBTQIA+ people. We strongly encourage individuals with these identities to apply even if you don't meet all of the requirements.Accepted file types: pdf, doc, docx, txt, rtfAccepted file types: pdf, doc, docx, txt, rtf LinkedIn Profile * Website Are you authorized to work lawfully in the United States? * Select... Do you now or in the future require visa sponsorship to continue working in the US? * Select... Do you require H1B Sponsorship? * Select... Voluntary Self-IdentificationFor government reporting purposes, we ask candidates to respond to the below self-identification survey.Completion of the form is entirely voluntary. Whatever your decision, it will not be considered in the hiringprocess or thereafter. Any information that you do provide will be recorded and maintained in aconfidential file.As set forth in Truveta's Equal Employment Opportunity policy,we do not discriminate on the basis of any protected group status under any applicable law.If you believe you belong to any of the categories of protected veterans listed below, please indicate by making the appropriate selection.As a government contractor subject to the Vietnam Era Veterans Readjustment Assistance Act (VEVRAA), we request this information in order to measurethe effectiveness of the outreach and positive recruitment efforts we undertake pursuant to VEVRAA. Classification of protected categoriesis as follows:A "disabled veteran" is one of the following: a veteran of the U.S. military, ground, naval or air service who is entitled to compensation (or who but for the receipt of military retired pay would be entitled to compensation) under laws administered by the Secretary of Veterans Affairs; or a person who was discharged or released from active duty because of a service-connected disability.A "recently separated veteran" means any veteran during the three-year period beginning on the date of such veteran's discharge or release from active duty in the U.S. military, ground, naval, or air service.An "active duty wartime or campaign badge veteran" means a veteran who served on active duty in the U.S. military, ground, naval or air service during a war, or in a campaign or expedition for which a campaign badge has been authorized under the laws administered by the Department of Defense.An "Armed forces service medal veteran" means a veteran who, while serving on active duty in the U.S. military, ground, naval or air service, participated in a United States military operation for which an Armed Forces service medal was awarded pursuant to Executive Order 12985. Select... Voluntary Self-Identification of Disability Form CC-305 Page 1 of 1 OMB Control Number 1250-0005 Expires 04/30/2026 Voluntary Self-Identification of Disability Form CC-305 Page 1 of 1 OMB Control Number 1250-0005 Expires 04/30/2026
        Why are you being asked to complete this form?We are a federal contractor or subcontractor. The law requires us to provide equal employment opportunity to qualified people with disabilities. We have a goal of having at least 7% of our workers as people with disabilities. The law says we must measure our progress towards this goal. To do this, we must ask applicants and employees if they have a disability or have ever had one. People can become disabled, so we need to ask this question at least every five years.Completing this form is voluntary, and we hope that you will choose to do so. Your answer is confidential. No one who makes hiring decisions will see it. Your decision to complete the form and your answer will not harm you in any way. If you want to learn more about the law or this form, visit the U.S. Department of Labor's Office of Federal Contract Compliance Programs (OFCCP) website at www.dol.gov/ofccp .How do you know if you have a disability?A disability is a condition that substantially limits one or more of your "major life activities." If you have or have ever had such a condition, you are a person with a disability. Disabilities include, but are not limited to:
        • Alcohol or other substance use disorder (not currently using drugs illegally)
        • Autoimmune disorder, for example, lupus, fibromyalgia, rheumatoid arthritis, HIV/AIDS
        • Blind or low vision
        • Cancer (past or present)
        • Cardiovascular or heart disease
        • Celiac disease
        • Cerebral palsy
        • Deaf or serious difficulty hearing
        • Diabetes
        • Disfigurement, for example, disfigurement caused by burns, wounds, accidents, or congenital disorders
        • Epilepsy or other seizure disorder
        • Gastrointestinal disorders, for example, Crohn's Disease, irritable bowel syndrome
        • Intellectual or developmental disability
        • Mental health conditions, for example, depression, bipolar disorder, anxiety disorder, schizophrenia, PTSD
        • Missing limbs or partially missing limbs
        • Mobility impairment, benefiting from the use of a wheelchair, scooter, walker, leg brace(s) and/or other supports
        • Nervous system condition, for example, migraine headaches, Parkinson's disease, multiple sclerosis (MS)
        • Neurodivergence, for example, attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder, dyslexia, dyspraxia, other learning disabilities
        • Partial or complete paralysis (any cause)
        • Pulmonary or respiratory conditions, for example, tuberculosis, asthma, emphysema
        • Short stature (dwarfism)
        • Traumatic brain injury Disability Status Select... PUBLIC BURDEN STATEMENT: According to the Paperwork Reduction Act of 1995 no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. This survey should take about 5 minutes to complete.
          #J-18808-Ljbffr

Keywords: Truveta Inc, Lacey , Director of AI Model Evaluation & Quality Validation - LLMs & Generative AI Seattle, WA, Executive , Seattle, Washington