Troubleshooting
Audience: Technical administration
Overview
This chapter sorts typical problems by symptom. For each symptom you find the likely cause and the fix. At the end there is a table of important error messages. Messages of the server follow the interface language; this chapter quotes them as shown in the English interface. Messages of the shell scripts and container logs are German and are quoted with an English translation.
First places to look
| Source | Content |
|---|---|
| System status page | status of database, Redis, object storage, worker and opsi server, see System status |
| Jobs page | complete log of every job with error message, see Jobs |
docker compose ps -a | which containers are running and which have exited (with exit code) |
docker logs opbweb-api | startup, migrations, sign-in, LDAP |
docker logs opbweb-worker | jobs, opsi connection |
docker logs opbweb-garage-init | object storage setup |
| Status bar of the user interface | “Backend not connected” if the API does not respond; orange notice for problems with the opsi server |
Stack does not start
| Symptom | Cause | Fix |
|---|---|---|
start.sh reports „.env fehlt – bitte ./setup-opbweb.sh ausführen (im Entwickler-Checkout: ./setup-opbweb.sh –dev).“ (.env is missing, run the setup script) | no .env | ./setup-opbweb.sh (or cp .env.example .env) |
| API does not start, log: “Secret file not found: …” | secret file missing, for example because the stack was started with docker compose instead of ./start.sh | use ./start.sh or run ./scripts/init-secrets.sh |
| API exits after about 60 seconds, log: “Database not reachable: …” | MariaDB not ready, or the password does not match the volume | check docker logs opbweb-db; if db_password.txt was changed, see Configuration |
| api, worker and frontend do not start, garage-init has a non-zero exit code | object storage setup failed | docker logs opbweb-garage-init; “ERROR: No admin token found” means secrets/garage_admin_token.txt is missing |
| garage-init keeps reporting “Not ready yet, retrying in 2s…” | Garage does not start or the admin token does not match | docker logs opbweb-garage; do not change the token in secrets/ afterwards |
| API log: “No users and no ADMIN_PASSWORD configured - no initial admin created” | secrets/admin_password.txt is empty or missing while the user table is empty | fill the file or let init-secrets.sh create it, restart the stack |
Sign-in
“Wrong user name or password.” For local accounts, every failed attempt counts. After 5 failed attempts from the same address the account is locked for that address for 15 minutes: “Account locked until HH:MM.” The user can still sign in from other addresses; the account is locked for everyone only after 50 failed attempts. After 20 failed attempts from one address (across all accounts) oPBWeb answers “Too many failed sign-ins from this address.” An administrator removes the lock under Users and roles with Unlock; Set password also unlocks the account. See Users and roles .
Password of the only administrator forgotten. There is no tool for resetting it without signing in. Another administrator (local or via an LDAP group) can set the password. So create a second administrator account early on. secrets/admin_password.txt only applies on the very first start.
“Account is disabled.” The account is deactivated in oPBWeb. An administrator reactivates it under Users and roles.
LDAP messages (“Directory service not reachable.”, “No permission for oPBWeb (no matching group).”): see LDAP Sign-in .
Signed out again after reloading the page or after about 15 minutes. The session is renewed via the cookie opbweb_refresh. If COOKIE_SECURE=true is set while oPBWeb is only reachable via HTTP, the browser does not store the cookie. Set COOKIE_SECURE to match the protocol and restart with ./start.sh.
“Session expired.” The token is invalid or was revoked, for example after signing out, after 7 days without renewal, or after secrets/jwt_secret_key.txt was changed. Sign in again.
Connection to the opsi server
The opsi server row on the system status page shows the status, with one entry per config server; in addition, there is the colored dot in front of each config server in the Depots view. The worker checks all config servers every 5 minutes; after a change you can check immediately under Depots → config server with Test connection. The setup is described in Config servers .
| Symptom | Cause | Fix |
|---|---|---|
| “not checked yet” | worker has just started | wait for the next check (default 5 minutes) or use Test connection |
| “No config server configured (Depots → add config server).” | no config server stored | add a config server under Depots with + |
| “No CA certificate is stored for the opsi server (Depots → config server).” | TLS verification without CA | Fetch from server, compare the fingerprint, Trust, Save |
| TLS or certificate error despite CA | URL with an IP address instead of a host name; the opsiconfd certificate is issued for the FQDN | enter the URL with the FQDN, e.g. https://opsi.example.local:4447 |
| “The server delivers a different CA certificate than the stored one (rotation?)”, on the system status page “Server delivers a new CA certificate, please accept it under Depots” | opsi has renewed its CA | Accept new certificate for the affected config server under Depots; until then, connections fail as soon as the server uses the new CA |
| “CA certificate expires in N days” | warning 60 days before expiry | have the CA renewed on the opsi server, then accept it |
| “… has no opsi admin rights”, on the system status page “Login without admin rights” | technical user is not an opsi admin | add the user to the opsi admin group |
| HTTP 403 on calls despite correct password | admin-networks in opsiconfd excludes the worker’s address | see admin-networks |
| “Config server not reachable: …” | network, DNS or firewall between worker and port 4447 | check reachability from the worker (see below) |
| “A config server with this URL is already configured: …” | the same URL is to be added a second time | select the existing config server in the list |
| in the job log “Depot … already belongs to config server … and is skipped” | the same opsi server is configured under two URLs | remove the duplicate config server |
| Sign-in to the opsi server fails after a restore | opsi_key_private.pem does not match the database | enter the password of the technical user again |
Check reachability from the worker container (example; -k only for this test, because the opsi CA is not known inside the container):
docker exec opbweb-worker curl -sk -o /dev/null -w '%{http_code}\n' https://opsi.example.local:4447/
An HTTP status number shows that name resolution and connection work. An error such as “Could not resolve host” or a timeout points to DNS or the firewall.
Building
Build errors are shown in the log of the Build job under Jobs.
| Message | Fix |
|---|---|
| “Script file (action) is missing in CLIENT_DATA.” | The script entered in the product data is missing from the package. Upload the file or correct the entry. |
| “A build is already running for this version.” | Wait for the running build to finish. |
| “The version was changed during the build. Please build again.” | Something was saved during the build. Click Build again. |
| “Version X is released; use “Back to draft” to rebuild.” | Released versions are read-only. Use Back to draft or New version. |
| “Archive could not be created: …” | Check the details in the message; often there is no free space on the host or the object storage is not reachable. |
| “The package cannot be built like this.” | The check before building found errors; each error is listed separately in the job log. Correct them and click Build again. |
Installing and uninstalling
| Symptom or message | Cause | Fix |
|---|---|---|
| Install … is grayed out, or “The version has not been built yet or was changed since the last build. Please build it first.” | no current build | Build again |
| “The depot depot (stage) requires the role Releaser.” | production depot, Packager role | ask a Releaser, or check the depot stage, see Depot management |
| “Depot depot is disabled or no longer exists.” | depot deactivated in oPBWeb or no longer reported by the configserver | Sync under Depots, activate the depot if needed |
| “Installation failed on: …” | opsiconfd error during upload or depot_installPackage | check the job log and the opsiconfd log |
| Product marked as locked on the depot | an aborted installation left the lock behind | under Depots → Installed products, select it and click Unlock (n), or in the install dialog use Force installation, even if the product is locked |
| “[depot] Depot has been busy with another job for too long.” | another job has held the depot for more than 6 hours | check the running job under Jobs; if the worker hangs, docker restart opbweb-worker |
| Upload of large packages aborts | time limit for upload and installation reached | increase OPSI_INSTALL_TIMEOUT_SECONDS; with many depots also JOB_TIMEOUT_SECONDS |
| Job ends with “Cancelled: job time limit exceeded or worker stopped.” | job ran longer than JOB_TIMEOUT_SECONDS or the worker was restarted | raise the limit or start the job again; a version released by it and installed nowhere is a draft again |
| Version is released although the installation failed | oPBWeb releases the version when the installation job starts | intended; for changes use Back to draft or New version |
Worker not reachable
Symptoms: On the system status page, the Worker row reports “no sign of life” or “last sign of life N s ago”; jobs stay in the waiting state; the file manager under Depots reports “Worker not reachable.” or “The worker does not respond.”
Causes and fix:
- Check whether the container is running:
docker compose ps worker. - Read the log:
docker logs --tail=200 opbweb-worker. Common causes are missing secrets, no connection to Redis, or an error at startup. - Restart the worker:
docker restart opbweb-worker. - “The worker does not respond.” can also mean that the worker is busy. It runs at most 10 jobs at the same time; file manager calls wait at most 60 seconds. Wait for running jobs to finish and try again.
Object storage
| Symptom | Cause | Fix |
|---|---|---|
| System status, object storage: “Bucket ‘packages’ is missing” | garage-init did not run or did not succeed | docker logs opbweb-garage-init, then ./start.sh |
| Access to the object storage denied, for example after a restore | S3 credentials in the volume garage-credentials do not match the key in Garage | ./stop.sh, docker volume rm opbweb_garage-credentials, ./start.sh |
| Files cannot be opened, system status reports missing blobs | database and object storage come from different points in time | restore a consistent backup, see Backup and Restore ; rebuild affected versions |
| Disk space on the host is running low | old blobs and build results of deleted packages | Clean up … on the System status
page; check volume usage with docker system df -v |
Uploads and time limits
| Symptom | Cause | Fix |
|---|---|---|
| Upload aborts with HTTP 413 | file larger than 20 GB (client_max_body_size) | make the file smaller or get it into the package another way, for example via the workbench |
| “Network error while uploading” | connection between browser and server dropped, or a proxy in between has its own limit | check upstream proxies and firewalls for size and time limits |
| Long requests abort after about 11 minutes | nginx timeout of 660 seconds without data transfer to the API | long operations run as jobs; set similar timeouts on your own proxies in front |
The worker deletes uploaded files that were not processed after 24 hours.
HTTPS and certificates
| Symptom | Cause | Fix |
|---|---|---|
| HTTPS does not respond, only HTTP | certs/fullchain.pem or certs/key.pem is missing or misnamed | check the file names; the opbweb-frontend log shows „Kein TLS-Zertifikat – HTTP auf Port 80“ (no TLS certificate – HTTP on port 80) |
| Frontend does not start after a certificate change | certificate and key do not match or are damaged | docker logs opbweb-frontend; check the pair |
| New certificate is not served | nginx reads certificates only at startup | docker restart opbweb-frontend |
| HTTP no longer works in the browser after switching back from HTTPS | HSTS header (one year) stored in the browser | stay with HTTPS or delete the HSTS entry in the browser |
| Signed out after reload | COOKIE_SECURE does not match the protocol | see Sign-in |
Important error messages
| Message | Meaning | Fix |
|---|---|---|
| Not signed in. | no token sent | sign in again |
| Session expired. | token invalid or revoked | sign in again |
| Account not available. | account deleted or deactivated | ask an administrator |
| No permission for this action. | role is insufficient (HTTP 403) | check the role |
| Wrong user name or password. | wrong credentials or unknown user | check the input; lock after 5 failed attempts from the same address |
| Too many failed sign-ins from this address. | 20 failed attempts from one address within 15 minutes | wait |
| Account locked until HH:MM. | local lock after failed attempts | wait or Unlock |
| Account is disabled. | account deactivated in oPBWeb | activate the account |
| Directory service not reachable. | LDAP connection or search failed | LDAP Sign-in |
| No permission for oPBWeb (no matching group). | LDAP sign-in without a matching group | check the group mapping |
| The last active administrator cannot be removed. / … cannot be deleted. | protection of the last administrator | create another administrator first |
| Version X is released (installed) and read-only. … | editing a released version | Back to draft or New version |
| Script file (action) is missing in CLIENT_DATA. | script missing during build | add the file |
| The version has not been built yet or was changed since the last build. Please build it first. | installing without a current build | Build |
| The depot depot (stage) requires the role Releaser/Packager. | depot stage rule | check the role or the depot stage |
| [depot] Depot has been busy with another job for too long. | depot lock longer than 6 h | check for a hanging job or worker |
| Installation failed on: … | opsiconfd error | job log, opsiconfd log |
| Config server not reachable: … | network to the opsi server | check reachability from the worker |
| No config server configured (Depots → add config server). | no config server configured | add a config server under Depots |
| No CA certificate is stored for the opsi server (Depots → config server). | CA missing | Fetch from server |
| A config server with this URL is already configured: name | duplicate URL | use the existing config server |
| The server delivers a different CA certificate than the stored one (rotation?) | CA rotation | Accept new certificate |
| user has no opsi admin rights | technical user without admin rights | grant the rights in opsi |
| {method}: HTTP {code} … | opsiconfd refuses (text from opsiconfd, not translated); for 403 usually admin-networks | admin-networks |
| Worker not reachable. / The worker does not respond. | worker stopped or busy | Worker not reachable |
| Download link invalid or expired. | link older than 2 minutes | start the download again |
| Deleting requires a dry run. / The dry run is missing or older than one hour. Please check again. | cleanup without a valid dry run | start a new dry run |
| Imports or builds are running. Please start again after they have finished. | cleanup blocked | start again later |
| Unexpected error: … | unhandled error in the job | check the job log and the worker log |
| Secret file not found: path (API log) | secret file missing at startup | ./start.sh or init-secrets.sh |
| Database not reachable: … (API log) | database not reachable after 60 s | opbweb-db log |