Bring Your Own Open-Weight Model (BYOM) to Vertex AI Model Garden — Managed Serving & Serverless Tuning (No GKE)

Disclaimer: Independent open-source reference project—not affiliated with, endorsed by, or supported by Google LLC, Alibaba Cloud, Hugging Face, or any third-party vendor; provided "AS IS" and you are responsible for all cloud billing charges (DISCLAIMER.md).

Bring your own open-weight Hugging Face checkpoints (such as Qwen 3, Llama 3.1 / 3.3, Gemma 3, Mistral, or DeepSeek) into Vertex AI Model Garden, customize them with non-default hyperparameters via Serverless Managed Open Model Tuning, and serve both baseline and fine-tuned weights on fully managed Vertex AI Dedicated Endpoints—with zero Google Kubernetes Engine (GKE) clusters to operate. Run the entire end-to-end pipeline automatically from Google Cloud Shell via ./deploy/run_cloud_shell.sh, or follow the complete manual Console UI and CLI/SDK walkthrough below.

Documentation & LLM Context: Live GitHub Pages Site · LLM Context (docs/llms.txt) · Release Notes · Disclaimer

flowchart TD
    HF["Hugging Face Hub (Qwen/Qwen3-4B)"] -->|"Stage weights & JSONL"| GCS_BASE["Cloud Storage: gs://.../hf-weights/Qwen--Qwen3-4B"]
    GCS_BASE -->|"1. CustomModel(gcs_uri=...)"| EP1["Managed Dedicated Endpoint (As-Is Base Model · vLLM)"]
    EP1 -->|"2. OpenAI SDK & endpoint.predict"| EVAL1["Baseline Domain Evaluation (0 / 8 Exact Match)"]
    GCS_BASE -->|"3a. custom_base_model + train/val JSONL"| TUNE["Serverless Managed Open Model Tuning (FULL · epochs=5 · lr=5e-5)"]
    TUNE -->|"Exports .safetensors"| GCS_FT["Cloud Storage: .../postprocess/node-0/checkpoints/final"]
    GCS_FT -->|"3b. CustomModel(gcs_uri=...)"| EP2["Managed Dedicated Endpoint (Fine-Tuned Model · vLLM)"]
    EP2 -->|"4. OpenAI SDK & endpoint.predict"| EVAL2["Post-Tuning Domain Evaluation & Comparison Report"]

Loading

---

Table of Contents

• Core Capabilities & Repository Overview
• Before You Begin & Costs
• Quickstart
  • Option A — Automated Execution in Google Cloud Shell
  • Option B — Manual Step-by-Step Guide (Console UI + gcloud & Python SDK)
    • Step 0 - Set Environment Variables
    • Step 1 - Configure a Google Cloud Project & Enable APIs
    • Step 2 - Check Enterprise Organization Policies (Optional)
    • Step 3 - Create Cloud Storage Bucket & Grant Least-Privilege Service Agent IAM
    • Step 4 - Clone the Repository & Install Python Dependencies
    • Step 5 - Stage Hugging Face Weights & Training Data to Cloud Storage
    • Step 6 - Deploy the Open-Weight Model As-Is (Managed Vertex AI — No GKE)
    • Step 7 - Test the As-Is Base Model on Domain Data
    • Step 8 - Fine-Tune the Model with New Hyperparameters (Serverless)
    • Step 9 - Deploy the Fine-Tuned Checkpoint & Compare Results
• Security, Least-Privilege Permissions & Cost Controls
• Troubleshooting & Quota Reference
• Cleaning Up
• Testing & Development (100% Offline)
• Key Terms, Limitations & Additional Resources

---

Core Capabilities & Repository Overview

• 100% Managed Serving (Zero GKE): Imports Hugging Face .safetensors directories directly from Cloud Storage via vertexai.preview.model_garden.CustomModel, pairing weights with Google-hardened vLLM or SGLang containers on managed Dedicated Endpoints (*.prediction.vertexai.goog).
• Live GPU Stock Auto-Selection: Queries CustomModel.list_deploy_options(available_machines=True, filter_by_user_quota=True) to automatically pick GPU SKUs with live regional capacity and project quota (for example, g4-standard-48 with 1x NVIDIA_RTX_PRO_6000 deploys in ~9 minutes).
• Serverless Open Model Tuning on Custom Weights: Launches FULL or PEFT_ADAPTER (LoRA) supervised fine-tuning directly from custom GCS base weights with overridden hyperparameters (epochs=5, learning_rate=5e-5 vs. default epochs=3, learning_rate=7.5e-5), exporting standard .safetensors weights back to GCS with zero vendor lock-in.
• Dual Inference Compatibility: Evaluates endpoints simultaneously via the OpenAI Python SDK (/v1beta1/.../chat/completions) and native Vertex AI endpoint.predict(..., use_dedicated_endpoint=True).
• One-Command Cloud Shell Automation + Dry-Run IAM: ./deploy/run_cloud_shell.sh provides read-only --preflight, non-mutating --emit-iam dry-run output, SHA-256 cached virtualenv setup, automatic lazy service-agent IAM retry, end-to-end --deploy, and one-command --cleanup.

Feature / Module	Purpose	Output
deploy/run_cloud_shell.sh	Automated Cloud Shell orchestrator (--preflight, --emit-iam, --deploy, --cleanup)	Provisioned resources, results/*.json, results/comparison_report.md
deploy/preflight_check.sh	Read-only check for CLIs, active gcloud auth, project access, enabled APIs, and bucket state	Human-readable preflight status report
src/generate_dataset.py	Deterministic enterprise incident triage JSONL builder (<=25% validation byte ratio enforced)	data/train.jsonl (120), data/val.jsonl (12), data/test_prompts.jsonl (8)
src/stage_hf_weights.py	Downloads Hugging Face .safetensors + tokenizer files and uploads weights & datasets to GCS	gs://<BUCKET>/hf-weights/Qwen--Qwen3-4B & datasets/*.jsonl
src/deploy_custom_weights.py	Queries live GPU stock/quota and deploys GCS weights to a Managed Dedicated Endpoint	Active Vertex AI Endpoint + results/*_deploy.json
src/evaluate_endpoint.py	Evaluates Dedicated Endpoints via OpenAI SDK (chat.completions) and endpoint.predict()	Accuracy & latency JSON (results/*_eval.json)
src/finetune_open_model.py	Submits & polls Serverless Managed Open Model Tuning (vertexai_sft or google_genai SDK)	Exported .safetensors in GCS + results/tuning_job_completed.json
src/compare_results.py	Builds side-by-side accuracy, latency, and per-ticket triage comparison report	results/comparison_report.md
src/cleanup.py	Undeploys models and deletes Dedicated GPU Endpoints to stop hourly GPU billing	Terminated endpoints ($0/hr GPU compute)
docs/ & scripts/sync_docs.sh	Zero-build-step GitHub Pages site, JSON-LD metadata, llms.txt, and screenshot sync check	Static documentation site & mirrored docs/assets/screenshots/
tests/test_offline_suite.py	100% offline unit, CLI, dry-run, asset-parity, and repository hygiene test suite	Zero-cloud-secret CI verification

---

Before You Begin & Costs

Prerequisites

• A billing-enabled Google Cloud project (Free Trial accounts generally lack GPU serving and Managed Open Model Tuning quotas).
• Vertex AI Administrator (roles/aiplatform.admin) and Storage Admin (roles/storage.admin) on the target project.
• Google Cloud Shell (recommended—gcloud, python3, and git are pre-installed) or a local terminal with gcloud and Python 3.10+.
• (Optional) A Hugging Face token (HF_TOKEN) only if substituting a gated model such as Llama 3.1 or Gemma 3. Qwen/Qwen3-4B used here is Apache 2.0 licensed and ungated.

Cost Drivers

• Vertex AI Dedicated GPU Endpoint: Billed per replica-hour while deployed (for example, g4-standard-48 with 1x NVIDIA_RTX_PRO_6000 or g2-standard-12 with 1x NVIDIA_L4). Always run --cleanup when finished testing, or enable Scale-to-Zero (min_replica_count=0, Preview).
• Serverless Managed Open Model Tuning: Billed only during active serverless job execution; zero idle GPU cost once JOB_STATE_SUCCEEDED is reached.
• Cloud Storage: Regional object storage for staged base weights (~8.06 GB), JSONL splits (~58 KB), and exported fine-tuned .safetensors weights (~8.06 GB).

---

Quickstart

Choose Option A to execute the complete workflow automatically in Google Cloud Shell, or Option B to perform every step manually via the Google Cloud Console UI, gcloud, and Python SDK.

Option A — Automated Execution in Google Cloud Shell

Open Google Cloud Shell, clone this repository, inspect the non-mutating IAM dry-run, and launch the automated 4-stage pipeline:

git clone https://github.com/mbettan/AgentForge.git
cd AgentForge

export PROJECT_ID="<YOUR_PROJECT_ID>"
export REGION="us-central1"

# 1. Inspect exact API, bucket, and least-privilege IAM commands (read-only dry-run)
./deploy/run_cloud_shell.sh --emit-iam

# 2. Run read-only environment & project preflight checks
./deploy/run_cloud_shell.sh --preflight

# 3. Execute all 4 stages end-to-end (automatically tears down temporary base endpoint during tuning,
#    and tears down the fine-tuned endpoint at the end when --auto-cleanup is passed)
./deploy/run_cloud_shell.sh --deploy --auto-cleanup

Tip

Omit --auto-cleanup if you want to keep the fine-tuned Dedicated Endpoint live for interactive testing in the Console or via the OpenAI SDK. When finished, run ./deploy/run_cloud_shell.sh --cleanup to delete any active GPU endpoints.

---

Option B — Manual Step-by-Step Guide (Console UI + gcloud & Python SDK)

Step 0 - Set Environment Variables

Replace <YOUR_PROJECT_ID> with your Google Cloud Project ID and <YOUR_GCP_ACCOUNT_EMAIL> with your Google Cloud user email:

export PROJECT_ID="<YOUR_PROJECT_ID>"
export GCP_USER_ACCOUNT="<YOUR_GCP_ACCOUNT_EMAIL>"
export REGION="us-central1"
export BUCKET_NAME="${PROJECT_ID}-mg-byom-demo"
export BUCKET_URI="gs://${BUCKET_NAME}"
export HF_REPO_ID="Qwen/Qwen3-4B"

Note

Keep these environment variables set across all manual steps. Re-export them if your Cloud Shell session restarts after an idle timeout.

Step 1 - Configure a Google Cloud Project & Enable APIs

• (Optional) If creating a brand-new Google Cloud project, create it and link your billing account (skip if using an existing project):

export ORG_ID="$(gcloud organizations list --format='value(ID)' | head -n 1)"
export BILLING_ID="<YOUR_BILLING_ACCOUNT_ID>"

gcloud projects create "${PROJECT_ID}" --organization="${ORG_ID}"
gcloud billing projects link "${PROJECT_ID}" --billing-account="${BILLING_ID}"

• Set your active gcloud configuration and Application Default Credentials (ADC), then enable the required APIs:

gcloud config set project "${PROJECT_ID}"
gcloud config set compute/region "${REGION}"
gcloud auth login "${GCP_USER_ACCOUNT}"
gcloud auth application-default login
gcloud auth application-default set-quota-project "${PROJECT_ID}"

gcloud services enable \
    aiplatform.googleapis.com \
    compute.googleapis.com \
    storage.googleapis.com \
    logging.googleapis.com \
    --project="${PROJECT_ID}"

Step 2 - Check Enterprise Organization Policies (Optional)

Tip

Unlike self-managed GKE clusters or Compute Engine VMs, Vertex AI Managed Dedicated Endpoints and Serverless Open Model Tuning run inside Google-managed tenant projects. Strict enterprise VM organization policies (constraints/compute.vmExternalIpAccess, constraints/compute.requireOsLogin, constraints/compute.requireShieldedVm, constraints/compute.restrictVpcPeering) do not block managed Model Garden serving or tuning.

Only verify constraints/gcp.resourceLocations if your organization restricts regional placement and excludes us-central1:

Description	Constraint Identifier	Required Setting
Google Cloud Resource Locations	constraints/gcp.resourceLocations	Allow in:us-central1-locations (or choose another supported region)

Step 3 - Create Cloud Storage Bucket & Grant Least-Privilege Service Agent IAM

Managed Custom Weights Serving and Serverless Open Model Tuning access your staging bucket using two Google-managed service agents:

1. Vertex AI Service Agent: service-${PROJECT_NUMBER}@gcp-sa-aiplatform.iam.gserviceaccount.com
2. Vertex AI Managed OSS Fine-Tuning Service Agent: service-${PROJECT_NUMBER}@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com

Create the regional Cloud Storage bucket with Uniform Bucket-Level Access and grant bucket-scoped roles/storage.objectUser (least privilege—read/write objects in this bucket only, no IAM admin permissions):

export PROJECT_NUMBER="$(gcloud projects describe "${PROJECT_ID}" --format='value(projectNumber)')"

# 1. Create regional bucket with uniform bucket-level access
gcloud storage buckets create "${BUCKET_URI}" \
    --project="${PROJECT_ID}" \
    --location="${REGION}" \
    --uniform-bucket-level-access

# 2. Provision the Vertex AI service identity (if never used in this project)
gcloud beta services identity create \
    --service=aiplatform.googleapis.com \
    --project="${PROJECT_ID}"

# 3. Grant bucket-scoped Storage Object User to the Vertex AI Service Agent
gcloud storage buckets add-iam-policy-binding "${BUCKET_URI}" \
    --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-aiplatform.iam.gserviceaccount.com" \
    --role="roles/storage.objectUser"

# 4. Grant bucket-scoped Storage Object User to the Managed OSS Fine-Tuning Service Agent
gcloud storage buckets add-iam-policy-binding "${BUCKET_URI}" \
    --member="serviceAccount:service-${PROJECT_NUMBER}@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com" \
    --role="roles/storage.objectUser"

Important

In a project that has never submitted an Open Model Tuning job before, service-${PROJECT_NUMBER}@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com is lazily provisioned on the first tuning API call. If command #4 reports that the service account does not exist yet, proceed to Step 8, submit the tuning job once to trigger automatic service-agent creation, re-run command #4, and re-submit Step 8 (automated seamlessly in ./deploy/run_cloud_shell.sh).

Step 4 - Clone the Repository & Install Python Dependencies

Clone this repository, create an isolated Python virtual environment, and install requirements.txt:

git clone https://github.com/mbettan/AgentForge.git
cd AgentForge
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Step 5 - Stage Hugging Face Weights & Training Data to Cloud Storage

To make fine-tuning impact unambiguous, we evaluate and customize Qwen/Qwen3-4B on a synthetic enterprise incident triage schema ([CYMBAL-TRIAGE] queue=CYM-... | severity=... | sla=... | runbook=RB-... | action=...). Because these proprietary queue codes and runbook IDs do not exist on the public web, an untuned model scores 0% exact match out of the box.

python3 src/generate_dataset.py

python3 src/stage_hf_weights.py \
    --project="${PROJECT_ID}" \
    --bucket="${BUCKET_URI}" \
    --hf-repo="${HF_REPO_ID}" \
    --cleanup-local

Note

Managed Open Model Tuning requires val.jsonl byte size to be at most 25% of train.jsonl byte size (12 validation examples vs. 120 training examples keeps the ratio at 10.4%). For gated Hugging Face models, append --hf-token="${HF_TOKEN}".

Verify in Cloud Storage -> Buckets that all 11 Qwen/Qwen3-4B .safetensors shards, config.json, and tokenizer files are staged in ${BUCKET_URI}/hf-weights/Qwen--Qwen3-4B:

Step 6 - Deploy the Open-Weight Model As-Is (Managed Vertex AI — No GKE)

You can deploy custom weights from Cloud Storage either via the Model Garden Console UI or via the Vertex AI Python SDK.

Via Google Cloud Console UI

1. Open Vertex AI -> Model Garden and click Deploy model with custom weights in the top action bar:

2. Browse to ${BUCKET_URI}/hf-weights/Qwen--Qwen3-4B, select us-central1, pick an available GPU machine configuration, leave Public (Dedicated endpoint) selected, and click Deploy:

Via CLI / Python SDK (src/deploy_custom_weights.py)

Note

Three critical SDK rules verified against google-cloud-aiplatform:

1. Import CustomModel from vertexai.preview.model_garden (not vertexai.model_garden).
2. machine_type, accelerator_type, and accelerator_count follow an all-or-none rule: either pass all three or omit all three. Passing --machine-type=auto calls list_deploy_options(available_machines=True, filter_by_user_quota=True) to pick live available GPU stock automatically.
3. CustomModel.deploy() always creates a Dedicated Endpoint—do not pass use_dedicated_endpoint=True to .deploy(), only to endpoint.predict(..., use_dedicated_endpoint=True).

python3 src/deploy_custom_weights.py \
    --project="${PROJECT_ID}" \
    --region="${REGION}" \
    --gcs-uri="${BUCKET_URI}/hf-weights/Qwen--Qwen3-4B" \
    --endpoint-display-name="qwen3-4b-byom-base-ep" \
    --model-display-name="qwen3-4b-byom-base" \
    --machine-type="auto" \
    --output-json="results/base_deploy.json"

Standalone Python SDK equivalent:

import vertexai
from vertexai.preview import model_garden

vertexai.init(project=PROJECT_ID, location=REGION)
base_model = model_garden.CustomModel(gcs_uri=f"{BUCKET_URI}/hf-weights/Qwen--Qwen3-4B")

# Inspect live GPU availability and quota in the target region
print(base_model.list_deploy_options(available_machines=True, filter_by_user_quota=True))

# Deploy to a Managed Vertex AI Dedicated Endpoint (vLLM managed by Google)
base_endpoint = base_model.deploy(
    machine_type="g4-standard-48",
    accelerator_type="NVIDIA_RTX_PRO_6000",
    accelerator_count=1,
    model_display_name="qwen3-4b-byom-base",
    endpoint_display_name="qwen3-4b-byom-base-ep",
)

Open Vertex AI -> Model Registry to verify the imported model (DEPLOY_SOURCE=CUSTOM_WEIGHTS, BASE_MODEL_ID=Qwen/Qwen3-4B, prebuilt pytorch-vllm-serve container—zero GKE clusters created):

Step 7 - Test the As-Is Base Model on Domain Data

Evaluate the as-is base endpoint across data/test_prompts.jsonl using both endpoint.predict(use_dedicated_endpoint=True) and the OpenAI Chat Completions API:

export BASE_EP_ID="$(python3 -c 'import json; print(json.load(open("results/base_deploy.json"))["endpoint_id"])')"

python3 src/evaluate_endpoint.py \
    --project="${PROJECT_ID}" \
    --region="${REGION}" \
    --endpoint-id="${BASE_EP_ID}" \
    --label="base_qwen3_4b_as_is" \
    --output-json="results/base_model_eval.json"

On the 8-prompt evaluation suite, untuned Qwen/Qwen3-4B responds cleanly (200 OK, 17.5s avg latency) with 0 / 8 (0.0%) exact schema & queue/runbook match, confirming it has no prior knowledge of the proprietary triage taxonomy.

Open Cloud Logging (resource.type="aiplatform.googleapis.com/Endpoint") to inspect the managed vLLM container startup (Application startup complete) and /generate + /v1/chat/completions 200 OK logs:

(Optional) If your project has limited regional GPU quota, tear down the base endpoint now before launching fine-tuning:

python3 src/cleanup.py --project="${PROJECT_ID}" --region="${REGION}" --endpoint-ids "${BASE_EP_ID}"

Step 8 - Fine-Tune the Model with New Hyperparameters (Serverless)

In Vertex AI Model Garden, Qwen3 supports both one-click deployment and serverless supervised fine-tuning (publishers/qwen/models/qwen3):

For qwen/qwen3@qwen3-4b, Model Garden's default hyperparameters are epochs=3 and learning_rate=7.5e-5 (FULL tuning mode). Below, we override both hyperparameters (--epochs=5, --learning-rate=5e-5) while pointing --custom-base-model directly at our staged Hugging Face weights in Cloud Storage:

python3 src/finetune_open_model.py \
    --project="${PROJECT_ID}" \
    --location="${REGION}" \
    --sdk="vertexai_sft" \
    --base-model="qwen/qwen3@qwen3-4b" \
    --custom-base-model="${BUCKET_URI}/hf-weights/Qwen--Qwen3-4B" \
    --train-dataset="${BUCKET_URI}/datasets/train.jsonl" \
    --val-dataset="${BUCKET_URI}/datasets/val.jsonl" \
    --output-uri="${BUCKET_URI}/tuning-runs/qwen3-4b-e5-lr5e5" \
    --display-name="qwen3-4b-cymbal-ft-e5-lr5e5" \
    --tuning-mode="FULL" \
    --epochs=5 \
    --learning-rate=5e-5 \
    --output-json="results/tuning_job_completed.json"

Standalone Python SDK equivalent:

import vertexai
from vertexai.preview.tuning import sft
from vertexai.tuning import SourceModel

vertexai.init(project=PROJECT_ID, location=REGION)

sft_job = sft.train(
    source_model=SourceModel(
        base_model="qwen/qwen3@qwen3-4b",
        custom_base_model=f"{BUCKET_URI}/hf-weights/Qwen--Qwen3-4B",
    ),
    tuning_mode="FULL",
    epochs=5,            # Overridden from default 3
    learning_rate=5e-5,  # Overridden from default 7.5e-5
    train_dataset=f"{BUCKET_URI}/datasets/train.jsonl",
    validation_dataset=f"{BUCKET_URI}/datasets/val.jsonl",
    output_uri=f"{BUCKET_URI}/tuning-runs/qwen3-4b-e5-lr5e5",
    tuned_model_display_name="qwen3-4b-cymbal-ft-e5-lr5e5",
)

Open Vertex AI -> Tuning in the Google Cloud Console to verify the serverless job configuration (Base model: qwen/qwen3@qwen3-4b, Tuning mode: Full tuning, Custom model weights, Number of epochs: 5, Learning rate: 0.00005):

Click the Dataset tab on the tuning job to inspect the validated chat splits (132 total examples across train.jsonl and val.jsonl) and conversation preview:

Step 9 - Deploy the Fine-Tuned Checkpoint & Compare Results

When src/finetune_open_model.py reaches JOB_STATE_SUCCEEDED, Vertex AI exports standard Hugging Face .safetensors weights and tokenizer files to ${BUCKET_URI}/tuning-runs/qwen3-4b-e5-lr5e5/postprocess/node-0/checkpoints/final.

1. Deploy the exported fine-tuned checkpoint to a Managed Dedicated Endpoint (CustomModel, no GKE):

python3 src/deploy_custom_weights.py \
    --project="${PROJECT_ID}" \
    --region="${REGION}" \
    --gcs-uri="${BUCKET_URI}/tuning-runs/qwen3-4b-e5-lr5e5/postprocess/node-0/checkpoints/final" \
    --endpoint-display-name="qwen3-4b-byom-tuned-ep" \
    --model-display-name="qwen3-4b-cymbal-ft-e5-lr5e5" \
    --machine-type="auto" \
    --output-json="results/finetuned_deploy.json"

2. Evaluate the fine-tuned endpoint on the exact same 8 test prompts and generate results/comparison_report.md:

export TUNED_EP_ID="$(python3 -c 'import json; print(json.load(open("results/finetuned_deploy.json"))["endpoint_id"])')"

python3 src/evaluate_endpoint.py \
    --project="${PROJECT_ID}" \
    --region="${REGION}" \
    --endpoint-id="${TUNED_EP_ID}" \
    --label="finetuned_qwen3_4b_e5_lr5e5" \
    --output-json="results/finetuned_model_eval.json"

python3 src/compare_results.py \
    --base-json="results/base_model_eval.json" \
    --tuned-json="results/finetuned_model_eval.json" \
    --output-md="results/comparison_report.md"

Stage	Serving Runtime	Hyperparameters	Domain Exact Match
Step 6–7: Base Qwen/Qwen3-4B (As-Is)	Managed Dedicated Endpoint (vLLM)	Pretrained HF base weights	0 / 8 (0.0%)
Step 8–9: Fine-Tuned Qwen/Qwen3-4B	Managed Dedicated Endpoint (vLLM)	FULL, epochs=5, lr=5e-5	Populated in results/comparison_report.md

---

Security, Least-Privilege Permissions & Cost Controls

• Bucket-Scoped Service Agent IAM (roles/storage.objectUser): Neither gcp-sa-aiplatform nor gcp-sa-vertex-moss-ft requires project-wide Storage Admin access. Binding roles/storage.objectUser strictly on ${BUCKET_URI} scopes read/write object access to that single bucket with Uniform Bucket-Level Access enabled.
• Authenticated Inference by Default: Vertex AI Dedicated Endpoints require valid Google Cloud OAuth 2.0 / IAM bearer tokens (aiplatform.endpoints.predict) on every request—no unauthenticated public endpoints are exposed. For private VPC-only traffic, pass enable_private_service_connect=True and psc_project_allow_list=[...] to CustomModel.deploy().
• Non-Mutating Dry-Run Mode: Always run ./deploy/run_cloud_shell.sh --emit-iam and ./deploy/run_cloud_shell.sh --preflight before mutating a shared or enterprise Google Cloud project.
• GPU Billing Safety Caps: ./deploy/run_cloud_shell.sh --deploy automatically undeploys the baseline GPU endpoint before starting multi-hour fine-tuning (unless --keep-base-endpoint is explicitly passed) and supports --auto-cleanup so GPU replicas are deleted immediately after evaluation finishes.

---

Troubleshooting & Quota Reference

• FAILED_PRECONDITION: Service account service-<PROJECT_NUMBER>@gcp-sa-vertex-moss-ft.iam.gserviceaccount.com does not have [storage.objects.get, storage.objects.list] gcp-sa-vertex-moss-ft is lazily created on a project's very first Open Model Tuning API call. Grant roles/storage.objectUser on ${BUCKET_URI} (command #4 in Step 3) and re-run Step 8.
• ImportError: cannot import name 'CustomModel' from 'vertexai.model_garden' Import from vertexai.preview.model_garden (from vertexai.preview import model_garden).
• TypeError: Failed to set field 'base_model' on TuningJob in sft.train() Import SourceModel from vertexai.tuning (from vertexai.tuning import SourceModel), not vertexai.preview.tuning.
• ValueError on CustomModel.deploy() hardware parameters machine_type, accelerator_type, and accelerator_count must either all be provided together or all be omitted (None).
• INVALID_ARGUMENT on validation split size Ensure val.jsonl byte size is <= 25% of train.jsonl byte size.
• Quota Increases: Check Google Cloud Quotas if deployment or tuning reports RESOURCE_EXHAUSTED:

Service	Quota Name	Quota Metric	Location	Recommended Minimum
Vertex AI API	Custom model serving Nvidia L4 GPUs	aiplatform.googleapis.com/custom_model_serving_nvidia_l4_gpus	us-central1	2
Vertex AI API	Custom model serving Nvidia RTX PRO 6000 GPUs	aiplatform.googleapis.com/custom_model_serving_nvidia_rtx_pro_6000_gpus	us-central1	2
Vertex AI API	Global concurrent managed OSS model tuning jobs	aiplatform.googleapis.com/global_concurrent_managed_oss_model_tuning_jobs	global	2

---

Cleaning Up

Dedicated GPU endpoints accrue hourly charges while deployed, even when idle.

# Option 1: Automated cleanup of any endpoints recorded in results/*.json
./deploy/run_cloud_shell.sh --cleanup

# Option 2: Explicit cleanup by Endpoint ID
python3 src/cleanup.py \
    --project="${PROJECT_ID}" \
    --region="${REGION}" \
    --endpoint-ids "${BASE_EP_ID}" "${TUNED_EP_ID}"

# Option 3 (Optional): Delete staged weights and checkpoints in Cloud Storage
gcloud storage rm --recursive "${BUCKET_URI}"

Note

Deleting a Vertex AI Endpoint does not delete your imported models in Vertex AI Model Registry or your exported .safetensors checkpoints in Cloud Storage, so you can re-deploy later without re-running fine-tuning.

---

Testing & Development (100% Offline)

All unit tests, CLI smoke checks, --emit-iam dry-run checks, screenshot sync checks, and repository hygiene checks run 100% offline with zero cloud credentials or live API calls required:

# 1. Run the offline unit, CLI, and repository hygiene test suite
python3 -m unittest discover -s tests -v

# 2. Verify docs/assets/screenshots/ parity with images/
./scripts/sync_docs.sh --check

# 3. Optional: Run ruff & shellcheck locally (enforced in GitHub Actions CI)
ruff check src/ tests/
shellcheck deploy/*.sh scripts/*.sh

---

Key Terms, Limitations & Additional Resources

• BYOM (Bring Your Own Model): Importing custom or modified open-weight checkpoints (.safetensors + tokenizer/config files) from Cloud Storage into Vertex AI Model Garden.
• Dedicated Endpoint: Isolated, low-latency Vertex AI serving endpoint (*.prediction.vertexai.goog) supporting OpenAI-compatible /chat/completions and streaming responses.
• Managed Open Model Tuning: Serverless Vertex AI supervised fine-tuning pipeline for open-weight families (Qwen, Llama, Gemma, DeepSeek) that exports standard Hugging Face .safetensors checkpoints back to GCS.
• Limitations:
  • CustomModel verifies config.json against Model Garden's supported open-model families (Qwen, Llama, Gemma, Mistral, DeepSeek, Phi, etc.). Arbitrary custom PyTorch nn.Module classes unsupported by vLLM/SGLang require a custom serving container in Vertex AI Model Registry.
  • For qwen/qwen3@qwen3-4b, Managed Open Model Tuning supports FULL tuning (max_seq_length=8192). Parameter-efficient PEFT_ADAPTER (LoRA) tuning is supported on larger variants (qwen/qwen3@qwen3-32b, google/gemma3@gemma-3-27b-it, meta/llama3_1@llama-3.1-8b-instruct).
• Official Documentation:
  • Deploy models with custom weights in Model Garden
  • Tune an open model on Vertex AI
  • Use prebuilt vLLM and SGLang containers for open models

---

Disclaimer: Independent personal open-source project, not affiliated with or endorsed by Google LLC or any third-party vendor; provided "AS IS" without warranty under Apache License 2.0—see DISCLAIMER.md before provisioning billable cloud resources.