feat: add HTTP catalog for Python packages and repository metrics

.gitignore

@@ -8,6 +8,7 @@ __pycache__/
 # Distribution / packaging
 .Python
 env/
+.venv/
 build/
 develop-eggs/
 dist/
@@ -61,3 +62,6 @@ target/
 
 # PyCharm
 .idea
+
+# VS Code / Cursor
+.vscode/

CHANGES/1358.feature

@@ -0,0 +1 @@
+Added repository package catalog and metrics endpoints, plus `collapse_builds` and `base_version` on the Python package content API.

docs/index.md

@@ -12,6 +12,7 @@ The REST API documentation for `pulp_python` is available [here](site:pulp_pytho
 
 - [Create local mirrors of PyPI](site:pulp_python/docs/user/guides/sync/) that you have full control over
 - [Upload your own Python packages](site:pulp_python/docs/user/guides/upload/)
+- [Browse the package catalog](site:pulp_python/docs/user/guides/catalog/) over the REST API
 - [Perform pip install](site:pulp_python/docs/user/guides/host/) from your Pulp Python repositories
 - Download packages on-demand to reduce disk usage
 - Every operation creates a restorable snapshot with Versioned Repositories
@@ -34,5 +35,4 @@ Users may also find pulpcore’s conceptual docs useful.
 This documentation falls into two main categories:
 
 1. `How-to Guides` shows the **major features** of the Python plugin, with links to reference docs.
-2. The [REST API Docs](site:pulp_python/restapi/) are automatically generated and provide more detailed information for each 
-minor feature, including all fields and options.
+2. The [REST API Docs](site:pulp_python/restapi/) are automatically generated and provide more detailed information for each minor feature, including all fields and options.

docs/user/guides/_SUMMARY.md

@@ -1,6 +1,7 @@
 * [Set up your own PyPI](pypi.md)
 * [Sync from Remote Repositories](sync.md)
 * [Upload and Manage Content](upload.md)
+* [Browse the package catalog](catalog.md)
 * [Host Python Content](host.md)
 * [Vulnerability Report](vulnerability_report.md)
 * [Attestation Hosting](attestation.md)

docs/user/guides/catalog.md

@@ -0,0 +1,113 @@
+# Browse the package catalog
+
+The content API lists **one row per file** (wheel, sdist, and so on). Use the repository
+package catalog when you want **one row per package name**, for example in a UI that
+shows Django once with its versions underneath.
+
+Both catalog endpoints default to the repository's latest complete version. Pass
+`repository_version` (HREF or PRN) to read an older snapshot. `{pulp_id}` is the
+repository UUID.
+
+## List packages
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/?limit=10"
+```
+
+`count` is the number of distinct packages, not files. Each row looks like:
+
+```json
+{
+  "name": "shelf-reader",
+  "name_normalized": "shelf-reader",
+  "last_updated": "2026-08-10T10:45:08.099362Z",
+  "versions": ["0.1"],
+  "latest_releases": [
+    {
+      "version": "0.1",
+      "release": "",
+      "created_at": "2026-08-10T10:45:08.099362Z"
+    }
+  ]
+}
+```
+
+- `versions` is the list of version numbers, newest first (PEP 440, so `1.10` before `1.9`).
+- `latest_releases` is the same versions with extra metadata. `release` is filled when
+  that version has a rebuild (for example `5.3.17+test.1` is shown as version
+  `5.3.17` with `release` `test.1`); otherwise it is empty.
+- `created_at` is when that version was added to the repository.
+- `last_updated` is when **any** file for the package last changed in this repository
+  version, including a rebuild of an older version.
+
+### Ordering
+
+Default order is `name`. Allowed fields: `name`, `name_normalized`, `last_updated`.
+Prefix with `-` for descending.
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/" \
+  ordering==-last_updated
+```
+
+### Name search
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/" \
+  name_normalized__istartswith==shelf
+http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/packages/" \
+  name_normalized__icontains==http
+```
+
+`name_normalized__istartswith` and `name_normalized__icontains` match the PEP 503
+normalized name and require at least 3 characters. `name__istartswith` matches the
+original project name and has no minimum length.
+
+## Repository metrics
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/repositories/python/python/${REPO_PK}/metrics/"
+```
+
+```json
+{
+  "package_count": 3,
+  "version_count": 9,
+  "build_count": 9
+}
+```
+
+| Field | Meaning |
+|-------|---------|
+| `package_count` | Distinct packages |
+| `version_count` | Distinct packages × versions (rebuilds of the same version count as one) |
+| `build_count` | Distinct packages × stored version strings (each rebuild counted) |
+
+Until a repository contains rebuilds, `version_count` equals `build_count`.
+
+## List files for a package
+
+Use the content API. `packagetype=sdist` returns one sdist per version (retry with
+`packagetype=bdist_wheel` if a release is wheel-only). `collapse_builds=true` keeps
+the newest rebuild per version so you do not have to page through every rebuild.
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/content/python/packages/" \
+  name==shelf-reader \
+  packagetype==sdist \
+  collapse_builds==true \
+  repository_version=="${LATEST_VERSION_HREF}"
+```
+
+Each content row includes `base_version`: the version without a PEP 440 local
+version (equal to `version` when there is none).
+
+To fetch a single version, omit `collapse_builds` and filter by `name`, `version`,
+and `packagetype`:
+
+```bash
+http GET "${BASE_ADDR}/pulp/api/v3/content/python/packages/" \
+  name==shelf-reader \
+  version==0.1 \
+  packagetype==sdist
+```

pulp_python/app/catalog.py

@@ -0,0 +1,202 @@
+"""Helpers for repository package catalog, metrics, and rebuild collapse."""
+
+from collections import defaultdict
+
+from django.db.models import CharField, Func, Max, Min, Q, Value
+from django.db.models.functions import Coalesce
+
+from pulp_python.app.models import PythonPackageContent
+from pulp_python.app.versions import (
+    BUILD_SUFFIX_PATTERN,
+    normalize_package_index_ordering,
+    rebuild_release,
+    version_sort_key,
+)
+
+
+def base_version_annotation(field_name="version"):
+    """SQL expression that strips a PEP 440 local version from ``version``.
+
+    Uses ``versions.BUILD_SUFFIX_PATTERN`` (POSIX) so Python ``strip_build_suffix``
+    and this ``REGEXP_REPLACE`` stay aligned. Implemented with ``REGEXP_REPLACE``
+    so it does not depend on Django's ``RegexpReplace`` (not present in every
+    Django 4.2/5.2 packaging Pulp uses).
+    """
+    return Func(
+        field_name,
+        Value(BUILD_SUFFIX_PATTERN),
+        Value(""),
+        function="REGEXP_REPLACE",
+        output_field=CharField(),
+    )
+
+
+def collapse_python_builds(queryset):
+    """Keep one content unit per ``(name_normalized, base_version)``.
+
+    ``base_version`` is ``version`` with a PEP 440 local version stripped.
+    The unit with the latest ``pulp_created`` is kept. Callers that want one
+    row per logical version (not per wheel/sdist) should also filter
+    ``packagetype``.
+    """
+    # DISTINCT ON cannot reuse pulpcore's list prefetches (cloned lookups /
+    # JOINs). Drop them, collapse, then prefetch artifacts for the reduced set.
+    return (
+        queryset.prefetch_related(None)
+        .annotate(_collapse_base_version=base_version_annotation())
+        .order_by("name_normalized", "_collapse_base_version", "-pulp_created")
+        .distinct("name_normalized", "_collapse_base_version")
+        .prefetch_related("contentartifact_set")
+    )
+
+
+def python_packages_in_version(repository_version):
+    """Python package content contained in ``repository_version``."""
+    if repository_version is None:
+        return PythonPackageContent.objects.none()
+    return PythonPackageContent.objects.filter(pk__in=repository_version.content)
+
+
+def membership_in_version_q(repository, repository_version):
+    """Q-object matching RepositoryContent rows present in ``repository_version``."""
+    return Q(
+        version_memberships__repository=repository,
+        version_memberships__version_added__number__lte=repository_version.number,
+    ) & (
+        Q(version_memberships__version_removed__isnull=True)
+        | Q(version_memberships__version_removed__number__gt=repository_version.number)
+    )
+
+
+def last_updated_annotation(repository, repository_version):
+    """Newest repository-membership time among all package units for a name.
+
+    Uses ``RepositoryContent.pulp_created`` (any rebuild/build), falling back to
+    the content unit's ``pulp_created``.
+    """
+    return Coalesce(
+        Max(
+            "version_memberships__pulp_created",
+            filter=membership_in_version_q(repository, repository_version),
+        ),
+        Max("pulp_created"),
+    )
+
+
+def distinct_package_names_qs(content_qs, repository, repository_version, ordering=None):
+    """One row per distinct ``name_normalized``, ordered for stable pagination."""
+    if ordering is None:
+        ordering = normalize_package_index_ordering([])
+    qs = content_qs.order_by().values("name_normalized").annotate(name=Max("name"))
+    if repository_version is None:
+        qs = qs.annotate(last_updated=Max("pulp_created"))
+    else:
+        qs = qs.annotate(last_updated=last_updated_annotation(repository, repository_version))
+    return qs.order_by(*ordering)
+
+
+def assemble_package_index(content_qs, name_rows, repository, repository_version):
+    """Build package-index dicts for ``name_rows``.
+
+    ``versions`` are distinct logical versions, newest first (PEP 440).
+    ``latest_releases`` keeps the newest rebuild (latest ``pulp_created``)
+    per base version in the same order. ``created_at`` is that unit's
+    repository-membership time (``RepositoryContent.pulp_created``), falling
+    back to the content unit's ``pulp_created``. ``last_updated`` is the newest
+    membership among all units for the package (any rebuild), taken from
+    ``name_rows`` when annotated.
+    """
+    if not name_rows or repository_version is None:
+        return []
+
+    names = [row["name_normalized"] for row in name_rows]
+    name_by_normalized = {row["name_normalized"]: row["name"] for row in name_rows}
+
+    in_this_version = membership_in_version_q(repository, repository_version)
+
+    newest_units = list(
+        content_qs.filter(name_normalized__in=names)
+        .prefetch_related(None)  # DISTINCT ON; see collapse_python_builds
+        .annotate(_base_version=base_version_annotation())
+        .order_by("name_normalized", "_base_version", "-pulp_created")
+        .distinct("name_normalized", "_base_version")
+    )
+    newest = [
+        {
+            "pk": unit.pk,
+            "name_normalized": unit.name_normalized,
+            "version": unit.version,
+            "_base_version": unit._base_version,
+            "pulp_created": unit.pulp_created,
+        }
+        for unit in newest_units
+    ]
+
+    memberships = {}
+    if newest:
+        memberships = dict(
+            PythonPackageContent.objects.filter(pk__in=[row["pk"] for row in newest])
+            .annotate(
+                membership_created=Min(
+                    "version_memberships__pulp_created",
+                    filter=in_this_version,
+                )
+            )
+            .values_list("pk", "membership_created")
+        )
+
+    releases_by_name = defaultdict(list)
+    for rel in newest:
+        releases_by_name[rel["name_normalized"]].append(rel)
+
+    result = []
+    for row in name_rows:
+        normalized = row["name_normalized"]
+        rels = sorted(
+            releases_by_name.get(normalized, []),
+            key=lambda item: version_sort_key(item["_base_version"]),
+            reverse=True,
+        )
+        versions = [item["_base_version"] for item in rels]
+        latest_releases = [
+            {
+                "version": item["_base_version"],
+                "release": rebuild_release(item["version"]),
+                "created_at": memberships.get(item["pk"]) or item["pulp_created"],
+            }
+            for item in rels
+        ]
+        result.append(
+            {
+                "name": name_by_normalized[normalized],
+                "name_normalized": normalized,
+                "last_updated": row.get("last_updated"),
+                "versions": versions,
+                "latest_releases": latest_releases,
+            }
+        )
+    return result
+
+
+def repository_metrics(content_qs):
+    """Distinct package / logical-version / build counts for package content.
+
+    Identity is always ``PythonPackageContent`` (not filtered by packagetype):
+
+    * ``package_count``: distinct ``name_normalized``
+    * ``version_count``: distinct ``(name_normalized, base_version)``
+    * ``build_count``: distinct ``(name_normalized, version)``
+
+    Until rebuild suffixes exist, ``version_count`` equals ``build_count``.
+    """
+    content_qs = content_qs.order_by()
+    return {
+        "package_count": content_qs.values("name_normalized").distinct().count(),
+        "version_count": (
+            content_qs.annotate(_base_version=base_version_annotation())
+            .values("name_normalized", "_base_version")
+            .distinct()
+            .count()
+        ),
+        "build_count": content_qs.values("name_normalized", "version").distinct().count(),
+    }

pulp_python/app/migrations/0025_pythonpackagecontent_name_normalized_trgm.py

@@ -0,0 +1,23 @@
+from django.contrib.postgres.indexes import GinIndex
+from django.contrib.postgres.operations import AddIndexConcurrently, TrigramExtension
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+    atomic = False  # required for CONCURRENTLY
+
+    dependencies = [
+        ("python", "0024_pythonrepository_error_on_reject"),
+    ]
+
+    operations = [
+        TrigramExtension(),
+        AddIndexConcurrently(
+            model_name="pythonpackagecontent",
+            index=GinIndex(
+                fields=["name_normalized"],
+                name="python_name_normalized_trgm",
+                opclasses=["gin_trgm_ops"],
+            ),
+        ),
+    ]