Join to apply for the Machine Learning Engineer - Training & Infrastructure role at P-1 AIJoin to apply for the Machine Learning Engineer - Training & Infrastructure role at P-1 AIAbout P-1 AIWe are building an engineering AGI. We founded P-1 AI with the conviction that the greatest impact of artificial intelligence will be on the built world—helping mankind conquer nature and bend it to our will. Our first product is Archie, an AI engineer capable of quantitative and spatial reasoning over physical product domains that performs at the level of an entry-level design engineer. We aim to put an Archie on every engineering team at every industrial company on earth.About P-1 AIWe are building an engineering AGI. We founded P-1 AI with the conviction that the greatest impact of artificial intelligence will be on the built world—helping mankind conquer nature and bend it to our will. Our first product is Archie, an AI engineer capable of quantitative and spatial reasoning over physical product domains that performs at the level of an entry-level design engineer. We aim to put an Archie on every engineering team at every industrial company on earth.Our founding team includes the top minds in deep learning, model-based engineering, and industries that are our customers. We just closed a $23 million seed round led by Radical Ventures that includes a number of other AI and industrial luminaries (from OpenAI, DeepMind, etc.).About The RoleWe’re looking for an experienced engineer to take ownership of LLM training operations across our applied research team. Your focus will be on making large-scale GPU training run reliably, efficiently, and fast on a dedicated mid-size GPU cluster and possibly on cloud platforms as well.You’ll work closely with researchers and ML engineers developing new models and agentic systems, ensuring their experiments scale smoothly across multi-node GPU clusters. From debugging NCCL deadlocks to optimizing FSDP configs, you’ll be the go-to person for training infrastructure and performance.What You’ll DoOwn the training pipeline for large-scale LLM fine-tuning and post-training workflowsConfigure, launch, monitor, and debug multi-node distributed training jobs using FSDP, DeepSpeed, or custom wrappersContribute to upstream and internal forks of training frameworks like TorchTune, TRL, and Hugging Face TransformersTune training parameters, memory footprints, and sharding strategies for optimal throughputWork closely with infra and systems teams to maintain the health and utilization of our GPU clusters (e.g., Infiniband, NCCL, Slurm, Kubernetes)Implement features or fixes to unblock novel use cases in our LLM training stackAbout You3+ years working with large-scale ML systems or training pipelinesDeep familiarity with PyTorch, especially distributed training via FSDP, DeepSpeed, or DDPComfortable navigating training libraries like TorchTune, Accelerate, or Trainer APIsPractical experience with multi-node GPU training, including profiling, debugging, and optimizing jobsUnderstanding of low-level components like NCCL, Infiniband, CUDA memory, and model partitioning strategiesYou enjoy bridging research and engineering—making messy ideas actually run on hardwareNice To HaveExperience maintaining Slurm, Ray, or Kubernetes clustersPast contributions to open-source ML training frameworksExposure to model scaling laws, checkpointing formats (e.g., HF sharded safetensors vs. distcp), or mixed precision trainingFamiliarity with on-policy reinforcement learning setups with inference (policy rollouts) as part of the training loop, such as GRPO, PPO, or A2CExperience working at a startupInterview ProcessInitial screening - Head of Talent (30 mins)Hiring manager interview - Head of AI (45 mins)Technical Interview - AI Chief Scientist and/or Head of AI (45 mins)Culture fit / Q&A (maybe in person) - with co-founder & CEO (45 mins)Seniority levelSeniority levelMid-Senior levelEmployment typeEmployment typeFull-timeJob functionJob functionEngineering and Information TechnologyIndustriesSoftware DevelopmentReferrals increase your chances of interviewing at P-1 AI by 2xGet notified about new Machine Learning Engineer jobs in San Francisco, CA.Software Engineer, Machine Learning (Multiple Levels) - SlackSan Francisco, CA $167,300.00-$334,600.00 2 days agoSan Francisco, CA $180,000.00-$240,000.00 4 days agoSan Francisco, CA $140,000.00-$180,000.00 6 months agoMachine Learning Engineer (I, II, or Sr.)Machine Learning Scientist, NLP (All Levels)San Francisco, CA $200,000.00-$300,000.00 5 months agoMachine Learning Engineer (I, II, or Sr.)Redwood City, CA $167,200.00-$250,800.00 3 days agoSan Francisco, CA $140,000.00-$215,000.00 1 month agoSan Francisco, CA $150,000.00-$260,000.00 4 months agoMachine Learning Scientist, NLP (All Levels)San Francisco, CA $200,000.00-$300,000.00 4 months agoSan Francisco, CA $140,000.00-$160,000.00 5 months agoSan Mateo, CA $163,700.00-$245,500.00 2 weeks agoSan Francisco, CA $160,000.00-$185,000.00 5 days agoSan Francisco, CA $128,000.00-$240,000.00 4 days agoSan Francisco, CA $100,000.00-$180,000.00 1 year agoSan Francisco, CA $88,000.00-$140,000.00 1 month agoSan Francisco, CA $100,000.00-$300,000.00 2 weeks agoSan Francisco, CA $140,000.00-$290,000.00 8 months agoSan Francisco, CA $150,000.00-$250,000.00 1 month agoSan Francisco, CA $190,000.00-$355,000.00 6 days agoSan Francisco, CA $160,000.00-$180,000.00 3 days agoSan Francisco, CA $175,000.00-$225,000.00 8 months agoSan Mateo, CA $140,000.00-$210,000.00 1 month agoSan Francisco, CA $175,000.00-$250,000.00 4 weeks agoMachine Learning Engineer, AI (FULLY REMOTE)San Francisco, CA $176,600.00-$225,900.00 2 weeks agoMachine Learning Engineer, GenAI Applied MLSan Francisco, CA $176,000.00-$220,000.00 1 month agoWe’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI. #J-18808-Ljbffr