Skip to content

fix(docker): Fix CPU/TPU nightly Docker build dependencies #490

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Toshi-31 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix(docker): Fix CPU/TPU nightly Docker build dependencies #490

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docker_build_dependency_image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ if [[ ${DEVICE} == "gpu" ]]; then
docker build --network host --build-arg MODE=${MODE} --build-arg JAX_VERSION=$JAX_VERSION --build-arg DEVICE=$DEVICE --build-arg BASEIMAGE=$BASEIMAGE -f ./maxdiffusion_gpu_dependencies.Dockerfile -t ${LOCAL_IMAGE_NAME} .
else
# Default to maxdiffusion_dependencies.Dockerfile for non-GPU builds
export BASEIMAGE=${BASEIMAGE:-python:3.12-slim-bullseye}
export BASEIMAGE=${BASEIMAGE:-python:3.12-slim-bookworm}
docker build --no-cache \
--network=host \
--build-arg MODE=${MODE} \
Expand Down
23 changes: 11 additions & 12 deletions maxdiffusion_dependencies.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Use Python 3.12-slim-bullseye as the base image unless overridden
ARG BASEIMAGE=python:3.12-slim-bullseye
# Use Python 3.12-slim-bookworm as the base image unless overridden
ARG BASEIMAGE=python:3.12-slim-bookworm
FROM $BASEIMAGE

# Environment variable for no-cache-dir and pip root user warning
ENV PIP_NO_CACHE_DIR=1
ENV PIP_ROOT_USER_ACTION=ignore

# Set environment variables for Google Cloud SDK and Python 3.10
# Set environment variables for Google Cloud SDK and Python 3.12
ENV PYTHON_VERSION=3.12
ENV CLOUD_SDK_VERSION=latest

Expand All @@ -16,15 +16,14 @@ ENV DEBIAN_FRONTEND=noninteractive
# Upgrade pip to the latest version and install uv
RUN python -m pip install --upgrade pip uv --no-warn-script-location

# Install system dependencies
RUN apt-get update && apt-get install -y apt-utils git curl gnupg procps iproute2 ethtool g++ && rm -rf /var/lib/apt/lists/*

# Add the Google Cloud SDK package repository
RUN curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee /etc/apt/sources.list.d/google-cloud-sdk.list

# Install the Google Cloud SDK
RUN apt-get update && apt-get install -y google-cloud-sdk && rm -rf /var/lib/apt/lists/*
# Install system dependencies and the Google Cloud CLI
RUN apt-get update && \
apt-get install -y apt-utils git curl gnupg procps iproute2 ethtool g++ && \
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee /etc/apt/sources.list.d/google-cloud-sdk.list && \
apt-get update && \
apt-get install -y google-cloud-cli && \
rm -rf /var/lib/apt/lists/*

# Install diagnostic and storage dependencies using uv
RUN python -m uv pip install --system \
Expand Down
20 changes: 8 additions & 12 deletions maxdiffusion_gpu_dependencies.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,14 @@ FROM $BASEIMAGE
# Stopgaps measure to circumvent gpg key setup issue.
RUN echo "deb [trusted=yes] https://developer.download.nvidia.com/devtools/repos/ubuntu2204/amd64/ /" > /etc/apt/sources.list.d/devtools-ubuntu2204-amd64.list

# Install dependencies for adjusting network rto
RUN apt-get update && apt-get install -y iproute2 ethtool lsof

# Install DNS util dependencies
RUN apt-get install -y dnsutils

# Add the Google Cloud SDK package repository
RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -

# Install the Google Cloud SDK
RUN apt-get update && apt-get install -y google-cloud-sdk
# Install system dependencies, network/DNS utilities, and Google Cloud CLI
RUN apt-get update && \
apt-get install -y iproute2 ethtool lsof dnsutils curl gnupg && \
curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
apt-get update && \
apt-get install -y google-cloud-cli && \
rm -rf /var/lib/apt/lists/*

# Set environment variables for Google Cloud SDK
ENV PATH="/usr/local/google-cloud-sdk/bin:${PATH}"
Expand Down
Loading