Identity operations

Operator runbook for the gateway side of identity: identity environments, platform admins, the identity service API and troubleshooting. The identity service itself (identity-api / identity-web, its IDENTITY_ENVIRONMENTS config, schemas and migrations) is documented in identity-api/README.md.

Model in one paragraph

An identity environment is one OIDC issuer of identity-api (<publicUrl>/api/auth, e.g. PROD https://identity.iesf.gg/api/auth, DEV https://dev.identity.iesf.gg/api/auth) plus the gateway's OIDC client in it and an optional post-logout redirect URI. Each platform is bound to one environment. Users are keyed by (issuer, subject), so the same e-mail address in DEV and PROD is two unrelated users. Only the deployment environment PROD can carry gateway permissions.

Gateway configuration

Variable	Purpose
`IDENTITY_ISSUER`	Required. PROD issuer (HTTPS; HTTP only for localhost)
`IDENTITY_OIDC_CLIENT_ID` / `IDENTITY_OIDC_CLIENT_SECRET`	The gateway's OIDC client in PROD (identity-api `gatewayClient` of `PROD`)
`IDENTITY_POST_LOGOUT_REDIRECT_URI`	Where PROD logout returns when the session has no platform; empty = `ACCOUNT_CENTER_URL`
`IDENTITY_ACCESS_TOKEN_AUDIENCE`	Required `aud` of Bearer access tokens (default `http://localhost:8080`); must equal identity-api `GATEWAY_AUDIENCE`
`IDENTITY_ENVIRONMENT_SECRET_KEY`	≥ 32 chars; encrypts client secrets of database environments
`IDENTITY_SERVICE_URL` / `IDENTITY_SERVICE_KEY`	Internal service API (e.g. `http://identity-api:8090`); key ≥ 32 chars, equal to identity-api `GATEWAY_SERVICE_KEY`
`PLATFORM_ADMIN_BOOTSTRAP_EMAILS`	Comma-separated, case-insensitive e-mails that become `platform_admin` on PROD login
`AUTH_CALLBACK_URL`	Gateway callback; the redirect URI of every environment's gateway client

On the identity side, identity-api's GATEWAY_PUBLIC_URL must be the gateway's public origin (https://auth.iesf.gg): identity-web's central onboarding page calls the gateway there. The gateway trusts the identity hosts back automatically: the host of IDENTITY_ISSUER and of every active environment's issuer are in its CORS and cookie-write allowlist.

Registering identity environments

PROD (deployment)

The PROD row (source: deployment) is seeded by the migration and always active. Its issuer, client and post-logout URI come from the IDENTITY_* variables above; the API shows it but refuses changes (409). Change it by editing the deployment configuration and restarting.

DEV, TEST, …

In identity-api, add the environment to IDENTITY_ENVIRONMENTS with a gatewayClient:
- redirectUris must contain the gateway's AUTH_CALLBACK_URL exactly (e.g. https://auth.iesf.gg/v1/auth/callback)
- postLogoutRedirectUris must contain exactly the gateway's /v1/auth/logged-out next to AUTH_CALLBACK_URL (e.g. https://auth.iesf.gg/v1/auth/logged-out). The gateway sends no other post-logout URI: the final destination (a platform return URL, or the environment's postLogoutRedirectUri) travels as an encrypted state, so platform URLs never go here
- the environment key must equal the gateway's environmentKey (the service API addresses users by that key)

http

POST /v1/identity-environments
Content-Type: application/json

{
  "environmentKey": "DEV",
  "displayName": "Development",
  "tier": "development",
  "issuer": "https://dev.identity.iesf.gg/api/auth",
  "oidcClientId": "iesf-gateway-dev",
  "oidcClientSecret": "<gatewayClient.clientSecret>",
  "postLogoutRedirectUri": "https://card-dev.iesf.gg/"
}

The secret is write-only (oidcClientSecretConfigured: true in responses) and encrypted with IDENTITY_ENVIRONMENT_SECRET_KEY; without that key the call fails with 503 IDENTITY_ENVIRONMENT_SECRET_KEY_MISSING. The issuer must differ from the PROD issuer.

Run POST /v1/identity-environments/DEV/validate and fix every failed check (below).
Register a separate platform for that environment (identityEnvironmentKey: "DEV", own credentials and redirect pairs).

Change fields later with PATCH (oidcClientSecret: null removes the secret). DELETE deactivates (refused while active platforms use it); POST …/activate re-enables. The tier cannot change while platforms are bound.

Validate checklist

POST /v1/identity-environments/{environmentKey}/validate is read-only and returns { environmentKey, ok, checks[] }:

Check	Passes when	Typical fix
`discovery`	`<issuer>/.well-known/openid-configuration` is reachable and its `issuer` matches exactly	Fix the issuer (scheme, host, `/api/auth` suffix, no trailing slash); check DNS / ingress
`endpoints`	`authorization_endpoint`, `token_endpoint`, `end_session_endpoint` are advertised	Upgrade / fix identity-api
`jwks`	`jwks_uri` is reachable and has at least one key	Run identity-api migrations / start the service
`client`	Client ID and secret are configured	Set `oidcClientId` / `oidcClientSecret`
`secret`	The stored secret decrypts (database environments with a secret)	Re-enter the secret (see key rotation)

Validate does not prove that the client ID/secret or redirect URIs are accepted by identity-api; finish with a real login through a platform of that environment.

Secret key rotation

Stored client secrets are bound to IDENTITY_ENVIRONMENT_SECRET_KEY. Changing the key makes them unreadable: logins of those environments fail and validate reports secret as failed. After a key change, re-enter every database environment's secret with PATCH { "oidcClientSecret": "…" }. Keep the key out of Git; PROD's secret is not affected (it comes from IDENTITY_OIDC_CLIENT_SECRET).

Platform admins

Admin permissions (platforms:admin, onboarding:schema:read, onboarding:schema:write) come from the local role platform_admin, and only for users of the PROD environment.

Bootstrapping the first admin

Set PLATFORM_ADMIN_BOOTSTRAP_EMAILS=ops@iesf.gg and deploy.
Sign in through a PROD platform (e.g. the Account Center) with that address. The e-mail must be verified in identity-api.
The gateway grants platform_admin (audit user_role.bootstrapped) and the session carries the permissions immediately.

Bootstrap never applies to DEV/TEST logins or unverified e-mails. Once real admins exist, remove addresses you no longer want auto-promoted; the list grants, it never revokes.

Managing roles and sessions

http

GET /v1/admin/users?search=jane&environmentKey=PROD&page=1&pageSize=25
PUT /v1/admin/users/{userId}/roles/platform_admin
DELETE /v1/admin/users/{userId}/roles/platform_admin
POST /v1/admin/users/{userId}/sessions/revoke

Granting is idempotent and only allowed for PROD users (409 ROLE_REQUIRES_DEPLOYMENT_ENVIRONMENT). It takes effect at the user's next login; existing sessions keep their snapshot.
Removing strips the permissions from the user's active sessions immediately. The last remaining admin cannot be removed (409 LAST_PLATFORM_ADMIN).
Session revoke ends all gateway sessions first, then calls identity-api revoke-sessions, and returns { "revokedGatewaySessions": 2, "identitySessionsRevoked": true }. If identity-api fails you get 503 with details.revokedGatewaySessions; the gateway part is already done, retry later for the identity part.

Service API (gateway → identity-api)

Account deletion (suspend) and admin session revoke (revoke-sessions) call identity-api's internal API at IDENTITY_SERVICE_URL under /internal/v1/environments/{key}/users/{sub}, with an HS256 JWT signed by IDENTITY_SERVICE_KEY (iss=iesf-gateway, aud=iesf-identity, env=<key>, act=users.*, 60 s lifetime, 5 s timeout).

IDENTITY_SERVICE_KEY must equal identity-api GATEWAY_SERVICE_KEY.
Gateway environment keys must equal identity-api environment keys, otherwise calls hit the wrong or no environment.
The internal API is not exposed by the ingress; use the cluster-internal URL.
Missing configuration → 503 IDENTITY_SERVICE_NOT_CONFIGURED; failures → 503 IDENTITY_UNAVAILABLE; unknown user → IDENTITY_USER_NOT_FOUND.

Machine clients

Platform backends that call admin or onboarding-schema APIs use a client_credentials client:

Add it to identity-api serviceClients of the PROD environment with the needed scopes (platforms:admin, onboarding:schema:read, onboarding:schema:write).

Request a token at the PROD token endpoint with resource=<IDENTITY_ACCESS_TOKEN_AUDIENCE>; without resource identity-api returns an opaque token the gateway rejects.

http

POST https://identity.iesf.gg/api/auth/oauth2/token
Authorization: Basic base64(clientId:clientSecret)
Content-Type: application/x-www-form-urlencoded

grant_type=client_credentials&scope=onboarding:schema:read&resource=https://auth.iesf.gg

Send it as Authorization: Bearer <token>.

Scopes are honoured only from the PROD issuer; machine tokens from other environments authenticate but carry no permissions. Machine tokens cannot call user endpoints (403). An iesf_platform / platform_key claim limits a caller to one platform's onboarding schema.

Production deployment (DOKS)

Current state (2026-10-09)

Component	Version	Where
identity-api	`iesf-identity-api:0.1.6`	PROD `identity.iesf.gg/api/auth`, DEV `…/api/auth/env/dev`, TEST `…/api/auth/env/test`
identity-web	`iesf-identity-web:0.1.6`	`identity.iesf.gg` (DEV/TEST entered through `/env/dev`, `/env/test`)
Gateway	`iesf-central-identity:2.1.0`	`auth.iesf.gg`
IESF Card	`iesf-card:1.3.0`	`card.iesf.gg`

Database. Identity data lives in the identity database on the sandbox-backend cluster (schemas auth_prod, auth_dev, auth_test), next to the gateway's profile-api database. The k8 user has CONNECT, CREATE, TEMPORARY on identity.
Gateway purge. The gateway database was purged for 2.0.0, and only IESF_CARD was registered again, keeping its previous client ID. A dump of the old database is in backups/ (ignored by Git).
Environments on one host. PROD, DEV and TEST all run on identity.iesf.gg, so no DNS records are needed for DEV or TEST:
- API by path. PROD is /api/auth and is the default. DEV and TEST are addressed by pathKey (/api/auth/env/dev, /api/auth/env/test). Each path is its environment's issuer, so the gateway's cookieless discovery and token calls always reach the right environment.
- Web UI by cookie. /env/<key>/… links (the login redirect, e-mail links) set the iesf_env cookie and continue on the plain path. Without the cookie the UI uses PROD, and /env/prod/… clears the cookie.
- Isolation. Sessions stay separate: separate schemas, secrets and cookie names (iesf_dev.session_token and so on).
- Own hosts later. The iesf-identity-preproduction ingress (deploy-identity-digitalocean.ps1 -Preproduction) is only needed if DEV and TEST ever move to their own hosts.
DEV platforms. IESF_COMPETITIONS, IESF_TEST, IESF_GG_LOCAL and IESF_GG_LOCAL2 are registered on DEV with their previous client IDs and redirect pairs. IESF_TEST also has its 17 onboarding fields back. Their new client secrets are in k8s/platform-secrets.local.yaml (ignored by Git); give each team only its own entry.
Google sign-in in DEV and TEST. Register these redirect URIs on the Google client as well: https://identity.iesf.gg/api/auth/env/dev/callback/google and https://identity.iesf.gg/api/auth/env/test/callback/google.
ZITADEL. It still runs in namespace zitadel; only its two ingresses were removed. They are saved in backups/zitadel-ingress-*.yaml, and kubectl apply of that file brings them back.
No Auth0 users yet. The Auth0 import has not been run; see identity-api/docs/auth0-import.md (run it from a one-off pod in the cluster; it is idempotent and merges with accounts registered in the meantime by e-mail address).

Secrets

Every secret lives in a local file that is ignored by Git, and the deploy scripts apply these files. None of them is in a manifest.

File	Secret	Contents
`k8s/identity-secrets.local.yaml`	`iesf-identity-secrets`	`DATABASE_URL`, Brevo API and SMTP keys, `GATEWAY_SERVICE_KEY`, `IDENTITY_ENVIRONMENTS` (per-environment Better Auth secret, Google client, gateway client, service clients)
`k8s/gateway-secrets.local.yaml`	`iesf-central-identity-secrets`	`DATABASE_URL`, `AUTH_TRANSACTION_SECRET`, `IDENTITY_OIDC_CLIENT_SECRET`, `IDENTITY_ENVIRONMENT_SECRET_KEY`, `IDENTITY_SERVICE_KEY`, `PLATFORM_ADMIN_BOOTSTRAP_EMAILS`, S3 keys
`k8s/frontend-secrets.local.yaml`	`iesf-card-secrets`	Card session secret, `IESF_CARD` client ID and secret

Values that must match between files:

IDENTITY_SERVICE_KEY = GATEWAY_SERVICE_KEY.
IDENTITY_OIDC_CLIENT_SECRET = PROD gatewayClient.clientSecret.

S3 is not configured yet (not-configured), so avatar upload is disabled.

The PROD environment also has a service client, iesf-ops-admin, with platforms:admin. Operators use it for API access when no admin can sign in, for example to register the first platform or grant the first role (see Machine clients).

Deploying

powershell

.\k8s\deploy-identity-digitalocean.ps1 -Version 0.1.x            # identity-api + identity-web
.\k8s\deploy-identity-digitalocean.ps1 -Version 0.1.x -Cutover   # also (re)applies the identity.iesf.gg ingress
.\k8s\deploy-digitalocean.ps1 -Version 2.0.x                     # gateway
.\k8s\deploy-frontend-digitalocean.ps1 -Version 1.3.x            # IESF Card (frontend/ is its own repository)

Keep image tags in the manifests current

kubectl apply -f k8s/<manifest>.yaml resets each Deployment to the image written in the manifest. Applying a manifest after a deploy therefore silently rolls back to the tag in the file. This once put an old Auth0 build of the Card back online. After every release, update the image: tag in the manifest. The deploy scripts set the image after applying, so they are safe.

The identity migration Job (iesf-identity-migrate-0001) is idempotent and its pod template is immutable. For a release with schema changes, add a new Job name and image tag to the manifest and pass -MigrationJob.

E-mail (Brevo)

identity-api sends through the Brevo HTTP API (BREVO_API_KEY). The SMTP key is stored only for reference.
The sender is currently no_reply@eonex.hu in every environment (mail.fromEmail). It must be a verified sender (or an authenticated domain) in Brevo. Otherwise verification and reset mails are rejected, and nobody can finish registration.
Without Brevo template IDs, the built-in English copy is used. Non-production subjects get a [DEV] or [TEST] prefix.

One OAuth client serves all environments. Every environment's redirect URI must be registered on it: https://identity.iesf.gg/api/auth/callback/google (and later https://dev.… and https://test.…).
The consent screen must be External and In production:
- Error 403: org_internal means it is set to Internal, so only Workspace accounts can sign in.
- Testing status limits sign-in to listed test users.

Configuration. Each environment gets an optional discord: { clientId, clientSecret }. One Discord application currently serves PROD, DEV and TEST.
Redirect URIs. Register all three in the Discord developer portal (OAuth2 → Redirects):
- https://identity.iesf.gg/api/auth/callback/discord
- https://identity.iesf.gg/api/auth/env/dev/callback/discord
- https://identity.iesf.gg/api/auth/env/test/callback/discord
Scopes. identify email. The public key is not needed; it is only used for bot interactions.
Verified address only. Unlike Google, Discord is not a trusted provider. An address Discord has not verified never creates an IESF user (audit sign_up.rejected, error page unable_to_create_user) and never links to an existing one. A verified address links automatically to the account with the same address, which gets an "accountLinked" e-mail.

Breach check exemptions

HIBP_SKIP_EMAIL_PREFIXES in the identity ConfigMap lists address prefixes that skip the Have I Been Pwned check at registration, sign-in, reset, change and add-password. It is currently donatdobos85, for test accounts. The password rules still apply to these addresses. Keep the list short: every entry weakens protection for those accounts.

Granting admin rights

An address in PLATFORM_ADMIN_BOOTSTRAP_EMAILS gets platform_admin on its first verified PROD login.
Anyone else can be promoted in the Card (Admin → Users), or with the ops service client via PUT /v1/admin/users/{id}/roles/platform_admin.
The user must have signed in once and must sign in again afterwards, because the session keeps the permissions it was created with.

ZITADEL decommissioning (after go-live is confirmed)

Confirm that nothing uses zitadel.iesf.gg or the ZITADEL APIs any more. Then remove the Helm release: helm uninstall zitadel -n zitadel. Until then, a helm upgrade would recreate the identity.iesf.gg ingress and take the host back.
Delete the namespace zitadel, its database zitadel on sandbox-backend, and the masterkey secret.
Remove k8s/zitadel-*, k8s/restore-zitadel-database.ps1 and infrastructure/zitadel from the repository.

Troubleshooting

Code	Status	Meaning / action
`IDENTITY_UNAVAILABLE`	503	Discovery, token endpoint, JWKS or service API unreachable. Check identity-api health, DNS, `IDENTITY_SERVICE_URL`
`IDENTITY_DISCOVERY_INVALID`	502	Discovery reports another issuer or lacks an endpoint. Fix the environment's `issuer`
`OIDC_CLIENT_NOT_CONFIGURED`	503	Client ID or secret missing for the environment
`IDENTITY_ENVIRONMENT_INACTIVE`	409	Platform's environment is deactivated; activate it or rebind the platform
`IDENTITY_ENVIRONMENT_NOT_FOUND`	400	Unknown `identityEnvironmentKey` on a platform request
`IDENTITY_ENVIRONMENT_NOT_CONFIGURED`	503	PROD row missing (run migrations) or the transaction's environment no longer exists
`IDENTITY_ENVIRONMENT_SECRET_KEY_MISSING`	503	Set `IDENTITY_ENVIRONMENT_SECRET_KEY` before storing or using secrets
`AUTHORIZATION_CODE_EXCHANGE_FAILED`	401	Token endpoint refused the code: wrong client secret, redirect URI not equal to `AUTH_CALLBACK_URL`, or code reused
`TOKEN_RESPONSE_INVALID`	401	Token response without ID token (check `openid` scope / client config)
`ID_TOKEN_INVALID`	401	Issuer, audience (client ID), algorithm or signature mismatch
`NONCE_MISMATCH`	401	Replayed or mixed-up login; restart from the platform
`ACCOUNT_DELETED` / `ACCOUNT_INACTIVE`	403	Local user is soft-deleted or inactive
`IDENTITY_SERVICE_NOT_CONFIGURED`	503	Set `IDENTITY_SERVICE_URL` and `IDENTITY_SERVICE_KEY`
`IDENTITY_USER_NOT_FOUND`	404	identity-api has no such user in that environment (check environment keys)
`ROLE_REQUIRES_DEPLOYMENT_ENVIRONMENT`	409	Tried to grant `platform_admin` to a non-PROD user
`LAST_PLATFORM_ADMIN`	409	Grant another admin first
`UNAUTHORIZED`	401	Missing/invalid Bearer token or cookie: wrong audience (`resource` missing), unknown or inactive issuer, expired token, ID token sent
Google `Error 403: org_internal`	—	OAuth consent screen is Internal; switch it to External and publish it (see Google sign-in)
"We sent you an email" but nothing arrives	—	Brevo sender `no_reply@eonex.hu` not verified, or the API key is wrong; check identity-api logs for `E-mail delivery failed`
`FORBIDDEN`	403	Missing permission (non-PROD token, role not yet in the session — sign in again), machine token on a user endpoint, or cookie write from a non-allowlisted `Origin`

Identity operations ​

Model in one paragraph ​

Gateway configuration ​

Registering identity environments ​

PROD (deployment) ​

DEV, TEST, … ​

Validate checklist ​

Secret key rotation ​

Platform admins ​

Bootstrapping the first admin ​

Managing roles and sessions ​

Service API (gateway → identity-api) ​

Machine clients ​

Production deployment (DOKS) ​

Current state (2026-10-09) ​

Secrets ​

Deploying ​

E-mail (Brevo) ​

Google sign-in ​

Discord sign-in ​

Breach check exemptions ​

Granting admin rights ​

ZITADEL decommissioning (after go-live is confirmed) ​

Troubleshooting ​

Related pages ​

Identity operations

Model in one paragraph

Gateway configuration

Registering identity environments

PROD (deployment)

DEV, TEST, …

Validate checklist

Secret key rotation

Platform admins

Bootstrapping the first admin

Managing roles and sessions

Service API (gateway → identity-api)

Machine clients

Production deployment (DOKS)

Current state (2026-10-09)

Secrets

Deploying

E-mail (Brevo)

Google sign-in

Discord sign-in

Breach check exemptions

Granting admin rights

ZITADEL decommissioning (after go-live is confirmed)

Troubleshooting

Related pages