Software Engineer, Collective Communication

About the Team

The Workload Networking team is responsible for the collective communication stack used in our largest training jobs. Using a combination of C++ and CUDA we work on novel collective communication techniques that enable efficient training of our flagship models on our largest custom built supercomputers.

The models we train are key ingredients to the AI research progress at OpenAI and the field as a whole, and we continually incorporate learnings from our entire research org into our training platform.

About the Role

As a Software Engineer, Networking you will design and implement custom networking collectives that are tightly integrated into our training stack.

We’re looking for people who have a background in low level performance critical software. Experience with collective communication is a bonus.

This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees.

In this role, you will

• Collaborate closely with ML researchers to design and implement efficient collective operations in C++ and CUDA.
• Ensure that our largest training jobs take full advantage of the different network transports used in our supercomputers.
• Work on simulations to inform our future supercomputer network designs.

You might thrive in this role if you

• Have written distributed algorithms using RDMA in the past.
• Are comfortable writing low level performance sensitive CPU and/or GPU code.
• Are familiar with network simulation techniques.