Staff Software Engineer(MLOps)

Toast creates technology to help restaurants and local businesses succeed in a
digital world, helping business owners operate, increase sales, engage
customers, and keep employees happy.

Now, more than ever, the Toast team is committed to our customers. We’re taking
steps to help restaurants navigate these unprecedented times with technology,
resources, and community. Our focus is on building the restaurant platform that
helps restaurants adapt, take control, and get back to what they do best:
building the businesses they love. And because our technology is purpose-built
for restaurants, by restaurant people, restaurants can trust that we’ll deliver
on their needs for today while investing in experiences that will power their
restaurant of the future. 

Toast is looking for a Staff Machine Learning Engineer to serve as a technical
linchpin for our AI Platform team. At the P4 level, you aren't just deploying
models; you are designing the fundamental infrastructure that enables dozens of
teams to build, deploy, and monitor AI at scale. You will act as a force
multiplier, mentoring senior engineers and setting the architectural standards
for our MLOps lifecycle—from feature stores and automated retraining to
high-performance inference at the edge.

A day in the life (Responsibilities)

* Architectural Leadership: Design and lead the evolution of a unified MLOps
platform that supports diverse needs across Toast, ensuring high
availability, scalability, and security of ML services.
* Engineering Excellence: Champion and institutionalize best practices for
CI/CD for ML (MLOps), automated testing, and infrastructure-as-code
(Terraform).
* Cross-Functional Synergy: Lead collaborative efforts across Data Engineering,
DevOps, and Product teams to bridge the gap between model prototyping and
production-grade reliability.
* Strategic Roadmapping: Partner with leadership and Product Managers to define
the 1-2 year technical vision for AI infrastructure, prioritizing long-term
stability over short-term fixes.
* Operational Ownership: Set the standard for observability and incident
response for ML systems, driving root-cause analysis for complex system
failures.
* Mentorship: Actively mentor P2 and P3 engineers, fostering a culture of
technical rigor and continuous learning.

What you'll need to thrive (Requirements)

* Education: Bachelor’s or Master’s degree in Computer Science, AI, or a
related technical field.
* Experience: A minimum of 10-12+ years of professional software engineering
experience, with at least 6-7 years specifically focused on productionizing
and scaling ML systems at the enterprise level.
* Core Tech Stack: Expert-level proficiency in Python, Scala, or Java/Kotlin.
Extensive experience with PySpark and high-performance computing.
* Generative AI & LLMOps: Proven track record of taking LLM applications from
research to production, including experience with Vector Databases,
LangChain/LangGraph, and A2A protocols.
* System Design: Superior ability to design distributed systems that handle
millions of requests with sub-second latency.
* Experience with microservice based architecture, preferably with AWS tooling
(SageMaker, DynamoDB, Athena, Glue, etc.)
* Experience in software engineering best practices and tools including
object-oriented programming, test-driven development, CI/CD, git, shell
scripting, task orchestration, MLflow
* Profound knowledge of model deployment, orchestration (Apache airflow,
Prefect), scaling, and managing CPU/GPU resources efficiently. 
* Exceptional problem-solving, analytical skills and the ability to tackle
complex problems with a critical thinking approach. 
* Outstanding communication and interpersonal skills, coupled with a
demonstrated ability to work collaboratively within a team environment.
* Foundational knowledge in statistical concepts (e.g. classification,
regression, etc) and deep learning algorithms (e.g. CNN, RNN) is desirable

WHAT WILL HELP YOU STAND OUT

* Data Strategy: Experience implementing enterprise-grade Feature Stores (e.g.,
Tecton, Feast) and real-time streaming frameworks like Apache Flink or Ray.
* Full-Stack Visibility: Ability to dive into the UI/UX layer (React) or deep
into the kernel/networking layer to debug performance bottlenecks.

Open Source/Community: Contributions to relevant open-source projects (MLflow,
Kubeflow, etc.) or a history of speaking at industry conferences.

AI at Toast

At Toast, one of our company values is that we're hungry to build and learn. We
believe learning new AI tools empowers us to build for our customers faster,
more independently, and with higher quality. We provide these tools across all
disciplines, from Engineering and Product to Sales and Support, and are inspired
by how our Toasters are already driving real value with them. The people who
thrive here are those who embrace changes that let us build more for our
customers; it’s a core part of our culture.

Our Total Rewards Philosophy 
We strive to provide competitive compensation and benefits programs that help to
attract, retain, and motivate the best and brightest people in our industry. Our
total rewards package goes beyond great earnings potential and provides the
means to a healthy lifestyle with the flexibility to meet Toasters’ changing
needs. Learn more about our benefits
at https://careers.toasttab.com/toast-benefits
[https://careers.toasttab.com/toast-benefits].

How Toast Uses AI in its Hiring Process

Throughout the hiring process, our goal is to get to know you. We use AI tools
to support our recruiters and interviewers with tasks like note-taking,
summarization, and documentation of interviews to ensure they can be fully
focused on your conversation. All hiring decisions are made by people. To learn
more: https://careers.toasttab.com/ai-in-hiring
[https://careers.toasttab.com/ai-in-hiring]

Diversity, Equity, and Inclusion is Baked into our Recipe for Success

At Toast, our employees are our secret ingredient—when they thrive, we
thrive. The restaurant industry is one of the most diverse, and we embrace that
diversity with authenticity, inclusivity, respect, and humility. By embedding
these principles into our culture and design, we create equitable opportunities
for all and raise the bar in delivering exceptional experiences.

We Thrive Together

We embrace a hybrid work model that fosters in-person collaboration while
valuing individual needs. Our goal is to build a strong culture of connection as
we work together to empower the restaurant community. To learn more about how we
work globally and regionally, check out:
https://careers.toasttab.com/locations-toast
[https://careers.toasttab.com/locations-toast].

Apply today!

Toast is committed to creating an accessible and inclusive hiring process. As
part of this commitment, we strive to provide reasonable accommodations for
persons with disabilities to enable them to access the hiring process. If you
need an accommodation to access the job application or interview process, please
contact candidateaccommodations@toasttab.com
[candidateaccommodations@toasttab.com].

------

For roles in the United States, it is unlawful in Massachusetts to require or
administer a lie detector test as a condition of employment or continued
employment. An employer who violates this law shall be subject to criminal
penalties and civil liability.