# Rodmena Container Registry > A private OCI/Docker-compatible container registry. It speaks the Docker > Registry HTTP API V2 so that **standard `docker push` / `docker pull` work > with no custom client tooling.** Backed by PostgreSQL (metadata), OVH S3 > (blobs, via 307 redirects to presigned URLs), Redis (chunk buffer), auth.rbac > (authorization), and TokenGate (metering/rate limits). > > This file is the ONE document a coding agent should read first. It tells you > exactly how to USE the service and how to WORK on this repo. If something is > not here, check `README.md` and `SPECS/0001-container-registry.md` (the EARS > contract). Do not guess — the details below are verified against the live code. - **Public endpoint:** `https://containers.rodmena.co.uk` - **Live service:** uvicorn on `0.0.0.0:8230` (NOT 8200 — that is HashiCorp Vault), behind nginx 443→127.0.0.1:8230, Let's Encrypt TLS. - **Stack:** Python 3.12+ / FastAPI / asyncpg / aioboto3 / Redis. Migrations: **migretti (`mg`) only — never Alembic.** - **Status:** deploys public push/pull + TokenGate 429 verified end-to-end against the live TLS path. (See `audit-verifications/`.) --- ## 1. How to push and pull images (the #1 thing) Use the **standard docker client**. Authentication is HTTP Basic where the **password is a per-user API key** (`rak_...`) issued by auth.rodmena.app — NOT a static shared password, and never a bare username (those are spoofable). ```bash # 1. Log in. The password is a per-user rak_... key from auth.rodmena.app. docker login containers.rodmena.co.uk -u -p # 2. Tag an image with the registry hostname + /:. docker tag myimage:latest containers.rodmena.co.uk/rodmena/myimage:latest # 3. Push. Repositories use a mandatory org prefix: /. docker push containers.rodmena.co.uk/rodmena/myimage:latest # 4. Pull. docker pull containers.rodmena.co.uk/rodmena/myimage:latest ``` ### Auth rules (do not get these wrong) - `GET /v2/` **without** credentials returns **401 + `WWW-Authenticate: Basic realm="containers.rodmena.co.uk"`**. This is **required, intentional, and correct** — Docker only sends stored credentials after a 401 challenge. A `/v2/` that returns 200 makes Docker treat the registry as anonymous and never send creds, which **breaks push/pull**. Do not "fix" the 401 into a 200. This is how Docker Hub and GHCR behave. - The password is authenticated via `check_api_key_permission` against auth.rodmena.app, which validates the key **and** the required permission **in one round trip**. - **Permission names use underscores, not dots:** `registry_push` and `registry_pull`. (Dots are illegal in auth permission names.) - A valid key lacking the permission → **403 `DENIED`**. A bad key → **401**. If auth.rodbena.app is down → **503** (fail-closed, never unauthenticated access — REQ-RESIL-04). - Push needs `registry_push`; pull needs `registry_pull`. Service accounts live in auth.rodmena.app, not in this repo. There are NO local users/roles/permissions tables (REQ-AUTH-05) — never add any. ### Metering / rate limits - Each pull and push is metered through TokenGate. **Resources use dots:** `registry.pulls` and `registry.pushes`. The subject is `registry:container-registry` (the `registry:` prefix is **mandatory** — a bare subject returns 403 `subject_not_permitted` from TokenGate). - On TokenGate quota/rate denial the registry returns **HTTP 429 + `Retry-After`** (OCI code `TOOMANYREQUESTS`). Metering is a **no-op while `TOKENGATE_API_KEY` is unset** (REQ-METER-01), so tests/CI that stub TokenGate still run. - Do **not** hand-roll Redis counters, usage tables, or a limiter (REQ-METER-03). All metering is delegated to TokenGate. --- ## 2. Configuration All config is environment variables loaded from `.env` (12-factor, **gitignored — never committed**). `.env` is the secret store; this file and the repo never contain keys. | Var | Required | Default | Purpose | |---|---|---|---| | `REGISTRY_DB_URL` | yes | `postgresql://registry_app:...@127.0.0.1:5432/registry` | PostgreSQL metadata DSN. Split by `Settings.db` for migretti/asyncpg. | | `REGISTRY_REDIS_URL` | yes | `redis://127.0.0.1:6379/0` | Redis for chunk buffer + upload state. | | `REGISTRY_S3_ENDPOINT` | yes (OVH) | — | `https://s3.eu-west-par.io.cloud.ovh.net` | | `REGISTRY_S3_REGION` | yes | `eu-west-par` | S3 region. | | `REGISTRY_S3_BUCKET` | yes | `rodmena-registry` | Bucket name. | | `REGISTRY_S3_ACCESS_KEY` | yes | — | S3 access key. | | `REGISTRY_S3_SECRET_KEY` | yes | — | S3 secret key. | | `REGISTRY_PRESIGN_TTL` | no | `900` | Presigned URL TTL, seconds. | | `REGISTRY_AUTH_KEY` | yes | — | UUID4 for the auth.rodmena.app client. | | `AUTH_BASE_URL` | no | `https://auth.rodmena.app` | auth-rbac base URL. | | `TOKENGATE_BASE_URL` | no | `https://tokengate.rodmena.co.uk` | TokenGate base URL. | | `TOKENGATE_API_KEY` | no | unset | TokenGate consume key. Absent → metering no-op. The literal `__PENDING__` is treated as unset. | | `TOKENGATE_SUBJECT` | no | unset | e.g. `registry:container-registry`. Prefix `registry:` is enforced. | | `REGISTRY_HOST` | no | `0.0.0.0` | Bind address. | | `REGISTRY_PORT` | no | `8230` | Bind port. **8230, not 8200** (Vault owns 8200). | | `REGISTRY_GC_SCHEDULE` | no | `0 3 * * *` | GC cron (daily 03:00). | Resilience knobs are also configurable (defaults shown): `bulkhead_max_concurrent=20`, `s3_retries=3`, `auth_breaker_failure_limit=0.30`, `auth_breaker_cooldown_seconds=30`. --- ## 3. Storage layout (S3 + PostgreSQL) - **S3 blobs** are content-addressable and path-style on OVH: `registry/v2/blobs/sha256///data` The 2-hex prefix optimizes S3 request partitioning. Uploads in flight live at `registry/v2/uploads/`. - **Blob downloads are NEVER streamed through the app.** `GET /v2//blobs/` returns **HTTP 307 Temporary Redirect** to a presigned S3 URL (NOT 301/302 — Docker must preserve the method, REQ-PULL-04). `HEAD /v2//blobs/` returns 200 + `Docker-Content-Digest` + `Content-Length`, no body. - **PostgreSQL** stores metadata in the `registry` schema: `repositories`, `blobs`, `repo_blobs`, `manifests`, `tags`, `upload_sessions`. Blobs are content-addressable by digest and shared across repos via `repo_blobs`. Tag re-points are ACID via `SELECT ... FOR UPDATE` (concurrent `latest` pushes serialize). - **Redis** buffers PATCH chunks below S3's 5 MB minimum part size: key `upload::buffer` (TTL 24h). A running SHA-256 is updated per chunk and finalized at `PUT`. --- ## 4. API surface Docker Registry HTTP API V2, all under `/v2/`: - `GET /v2/` — version handshake (200 + `Docker-Distribution-API-Version: registry/2.0` with creds; 401 challenge without). - Push: `POST /v2//blobs/uploads/` (202, start S3 multipart), `PATCH .../uploads/` (202, buffer/UploadPart), `PUT .../uploads/?digest=sha256:...` (201, complete + verify digest), `PUT /v2//manifests/` (201, verifies all layer/config digests exist). - Pull: `GET/HEAD /v2//manifests/` (by tag or digest), `GET /v2//blobs/` (307→S3). - Listing: `GET /v2/_catalog`, `GET /v2//tags/list`. - Deletion: `DELETE /v2//manifests/`, `DELETE /v2//blobs/` (removes repo link; S3 object kept if other repos reference it). - Cross-repo mount: `POST /v2//blobs/uploads/?mount=&from=` (201, no re-upload). - **Health** (outside `/v2/`): `GET /health` (liveness), `GET /ready` (checks PostgreSQL + S3; 200 or 503). Error mapping per OCI Distribution Spec: `BLOB_UNKNOWN` 404, `DIGEST_INVALID` 400, `MANIFEST_INVALID` 400, `UNAUTHORIZED` 401, `DENIED` 403, `TOOMANYREQUESTS` 429. S3 multipart is aborted on digest mismatch. --- ## 5. Working on this repo ### Layout ``` registry/ application package (api/ v2+health+errors, auth/ middleware, db/, storage/ s3+buffer, tasks/ gc, config.py, app.py, services.py, metering.py) migrations/ migretti SQL migrations (hand-written SQL) scripts/migrate.sh sources .env, splits REGISTRY_DB_URL, runs `mg apply --yes` tests/ pytest, real PostgreSQL + real OVH S3 (auth + TokenGate stubbed) deploy/ nginx site config (reference; the live nginx is managed by the auditor side) supervice.ini process supervision config mg.yaml migretti config (interpolates REGISTRY_DB_*) ``` ### Run, test, deploy ```bash # Apply migrations (idempotent; safe at startup — REQ-OPS-05). bash scripts/migrate.sh # or: make migrate # Run the API (reads .env). make run # uvicorn registry.main:app --host 0.0.0.0 --port 8230 # Tests (real PostgreSQL + real OVH S3; auth + TokenGate stubbed). make test # Supervise (auto-restart + TCP health check on 127.0.0.1:8230). supervice -c supervice.ini ``` ### Testing discipline (important — a real gotcha) The suite runs against a **dedicated `registry_test` database** (default `postgresql://registry_app:...@127.0.0.1:5432/registry_test`), created + migrated once per session, NOT the live `registry` DB. `conftest.py` overrides `REGISTRY_DB_URL` to point at it and restores afterward; override with `REGISTRY_TEST_DB_URL`. **Never point the suite at the live `registry` DB** — live docker activity pollutes it and produces spurious FK failures. S3/Redis/PG are exercised against the REAL services (a mocked-only S3 test cannot catch signing/endpoint mistakes). auth.rbac and TokenGate are stubbed in tests; their response mappings are asserted separately in `tests/test_auth.py` and `tests/test_metering.py`. Do not weaken a control to make a test pass. ### Migrations **migretti (`mg`) only.** Alembic/Flyway/etc. are forbidden (REQ-OPS-04). `mg apply --yes` runs at startup. Migrations are hand-written SQL. `mg.yaml` interpolates `REGISTRY_DB_HOST/PORT/USER/PASSWORD/NAME` from `REGISTRY_DB_URL`. ### Process supervision `supervice.ini` runs uvicorn on `0.0.0.0:8230`, auto-restart, TCP health check (127.0.0.1:8230, 10s interval, 15s start period). The `directory`/`env_file` point at the deployed checkout — update them after a deploy/move. --- ## 6. Resilience model (do not bypass) - **S3 calls** go through a **bulkman bulkhead** with `circuit_breaker_enabled=False` (breaker ALWAYS off), max 20 concurrent (REQ-RESIL-01), wrapped by **resilient-circuit** retry: 3 retries, exponential backoff 1s–10s, factor 2, jitter 0.1, on `ConnectionError`/`TimeoutError`/botocore transient exceptions (REQ-RESIL-02). - **auth.rodbena.app calls** go through a resilient-circuit **circuit breaker**: trip at 30% failure, 30s cooldown (REQ-RESIL-03). On auth unavailability the registry returns **503**, never unauthenticated access (REQ-RESIL-04). - **Metering** is fail-open on TokenGate transport error (`TokenGateError` → warn + allow) — no EARS requirement mandates metering fail-closed. If you want fail-closed metering that is a policy change, not a bug; record it as an assumption. --- ## 7. Port/infra facts (avoid surprises) - **8230 is the registry port. 8200 is taken by HashiCorp Vault.** Do not "reclaim" 8200. - nginx reverse-proxies 443 → 127.0.0.1:8230 with `client_max_body_size 0` and `proxy_request_buffering off` (Docker layers are large; uploads stream straight through to S3). The live nginx config is managed outside this repo (auditor side); `deploy/containers.rodmena.co.uk.conf` is a reference copy. - DNS `containers.rodmena.co.uk` → the host's public IP. TLS via Let's Encrypt (certbot, webroot). - `tokengate.rodmena.co.uk` resolves to a **separate host** (.253) — the registry calls it over HTTPS; it is not local. --- ## 8. What to do / not do - **Use the standard docker client** for verification — no registry-specific tooling (REQ-CI-01). - **Verify through the product's own interface** (HTTP/docker), never by reading/writing the database directly. - For any limit/quota/cap, test **both directions**: blocks when exceeded AND resumes when it should. - Repositories **must** use an org prefix (`/`, e.g. `rodmena/ci-python`). - Garbage collection is a **separate offline job** (default daily 03:00); blobs are NOT deleted during push/pull (REQ-GC-02). - Do not commit `.env` or any secret. Keys (S3, TokenGate, RAK, auth) live in `.env` or house secrets files, never in the repo. ## 9. Where to look next - `SPECS/0001-container-registry.md` — the full EARS spec (the contract; schema, endpoints, requirements, error mapping). - `SPECS/0002-deploy-containers-rodmena.md` — deploy requirements (DNS, nginx, SSL, S3, supervice). - `README.md` — quickstart + architecture summary. - `audit-verifications/` — the auditor's live end-to-end verification record (reusable baseline for the next audit round). - `registry/metering.py`, `registry/auth/middleware.py` — the TokenGate + auth-rbac integrations (the two external dependencies).