Troubleshooting
Start with the two commands Netos support will ask for:
sudo ./netos-install status --site /root/netos/site.yml # what is installed, last result
sudo ./netos-install support-bundle --site /root/netos/site.yml # one file to send Netos
Every run writes /var/log/netos/<command>-<timestamp>.log (every command it ran, its exit code and duration; for a
failed command, the last 200 lines of its output). A failing command also prints the last 20 lines of its error
output on the console.
Bundle signature checks
| Message | Action |
|---|---|
unsigned |
Ask Netos for a signed bundle. --require-signature makes this a failure. |
site has a pinned signing key; unsigned bundle rejected |
Obtain a signed bundle; use --allow-unsigned only for an explicit unsigned override, which is logged as a warning. |
signature valid, key NOT independently trusted |
Verify again using --trusted-key with the public key obtained separately from Netos support. |
no trusted Ed25519 public key is available |
Obtain the public key from support; a key supplied only inside the bundle is not an independently trusted key; the release key and its ID are in Install. |
SHA256SUMS signature verification failed |
Stop. The checksum list or signature does not match the chosen key; request a new bundle and key from support. |
invalid Ed25519 public key or malformed SHA256SUMS signature |
Stop. Replace the bad key or bundle using copies from support. |
SHA256SUMS signature required but bundle is unsigned |
Obtain a signed bundle, or omit --require-signature while unsigned bundles are allowed. |
cryptography is unavailable and OpenSSL pkeyutl -rawin is unavailable |
Run the installer in the NetBox venv with cryptography, or provide an OpenSSL version that supports pkeyutl -rawin. |
Health check: doctor
./netos-install doctor --site /root/netos/site.yml
Read-only. On a healthy test host it printed:
PASS manage_check
PASS plugin_versions {'netos': '4.14.304', 'netos_reporting': '4.9.97', 'netos-model-builder': '4.5.90'}
SKIP http_landing_pages authenticated HTTP probe requires temporary user writes
PASS netos_platform_checks exit 0
SKIP pipeline_run_health command writes data or opens outbound HTTP
PASS netos_detect_stale_pipeline_runs exit 0
PASS netos_job_effective_config exit 0
SKIP validate_doc_links command writes data or opens outbound HTTP
PASS pip_check
PASS licence
PASS rq {'high': {'depth': 0, 'oldest_job': None, 'workers': 1}, 'default': {...}, 'low': {...}}
SKIP lines are checks it deliberately does not run because they would write or reach the network. In rq, a
queue whose depth keeps growing with workers: 0 means that worker service is down
(systemctl status netbox-rq netbox-rq-low).
Recipes: troubleshooting commands Netos may ask you to run
./netos-install run --list --site /root/netos/site.yml
| Recipe | What it does | Writes? |
|---|---|---|
showmigrations |
Lists every database migration and whether it is applied | No |
platform-checks |
Netos's own platform findings | No |
stale-pipeline-runs |
Reports pipeline runs stuck in "running" | No |
reconcile-insights-permissions |
Checks the Insights read permission against installed models | No (check mode) |
collectstatic |
Rebuilds NetBox's static files | Yes: snapshots first |
remove-stale-contenttypes |
Removes content types for models that no longer exist | Yes: snapshots first |
Run one with ./netos-install run <recipe> --site /root/netos/site.yml. A recipe that writes stops NetBox and
takes a snapshot first, like an install. Netos may also send you a new recipe file with its SHA-256; run it with
./netos-install run /path/to/recipe.yml --sha256 <value> --site /root/netos/site.yml, which refuses if the
file does not match.
Site report: collect
./netos-install collect --site /root/netos/site.yml --out /root/netos/site-report.json
Read-only and safe to run any time, including before Netos is installed. It records the OS, NetBox and
PostgreSQL versions and settings, installed packages, services and the preflight results. Every value not on a
fixed list of safe fields is written as present or absent, so the file is safe to send; you can read it first.
Netos uses it to prepare a site.yml for you.
Preflight FAIL lines
| Line | Meaning | What to do |
|---|---|---|
netbox_version 4.5.9 |
This bundle is built for a different NetBox version | Get the bundle for your NetBox version from Netos |
python_abi |
The NetBox venv is not Python 3.12 | Ask Netos for a bundle for your Python version |
licence missing or invalid |
No licence_file in site.yml, the file is not where it says, or the token does not decrypt with its key |
Check the path; ask Netos to re-send the licence |
rehearsal copied-venv pip check failed: <package> has requirement <x>, but you have <y> |
Another package in your NetBox venv conflicts with a Netos dependency. Your live venv was not changed | Send Netos the line. The fix is in the conflicting plugin or in Netos, never a forced install |
database_tools |
pg_dump/pg_restore missing, or no way to restore the database for rollback |
Install the PostgreSQL client for your server's major version; for a remote database, see Roll back |
disk |
Not enough room for the snapshot | Free space under /var/backups/netos or set snapshot_root in site.yml |
configure requires installed Netos plugins in the NetBox venv |
configure ran before the plugin wheels were installed |
The supplied licence_file and external_url were saved to site.yml; run install first, then configure if needed |
Tested on the offline test host: a wrong NetBox version, a missing licence and a conflicting third-party package
each produced exactly one FAIL and left the live venv byte-identical (checked by hashing pip freeze).
Install or upgrade FAIL
The last lines name the failing step and print the rollback command. Read the log for that step first:
sudo less "$(ls -t /var/log/netos/install-*.log | head -1)"
Search for the last RC line that is not RC 0; the STDERR lines under it are the cause. Then either fix the
cause and run install/upgrade again, or roll back (Roll back).
The install was interrupted
An SSH drop, a reboot or a killed process during install/upgrade leaves NetBox stopped and possibly
half-installed. Run rollback --latest. That case was tested twice (killed during the package step, and during
migrations) and both times rollback alone restored the box exactly.
NetBox is slow to come back after install
The NetBox workers import every plugin at start. On the 2 vCPU test host the first page answered about a minute
after the services started. The post-check waits up to 5 minutes (http_wait_seconds in site.yml).
The NetBox application
These checks apply to any NetBox with Netos installed. Run them as root on the NetBox host.
Check that NetBox and its workers are running:
systemctl status netbox netbox-rq netbox-rq-low
Restart them:
systemctl restart netbox netbox-rq netbox-rq-low
Read the last 1000 lines of each service's log:
journalctl --no-pager -n 1000 -u netbox
journalctl --no-pager -n 1000 -u netbox-rq
journalctl --no-pager -n 1000 -u netbox-rq-low
Read the web server logs. With the standard NetBox NGINX setup they are:
tail -n 200 /var/log/nginx/netbox.access.log
tail -n 200 /var/log/nginx/netbox.error.log
If your web server or log names differ, use yours. Send the output with ./netos-install support-bundle when you
contact Netos support.
Offline behaviour
The bundle never reaches the network during install, upgrade or rollback: the test host's firewall logged zero outbound packets from NetBox across all of them, and across a runtime check of 39 plugin pages and a PDF render with a chart. Known places where the plugins still refer to Netos-hosted addresses (none were contacted in the test):
- a bundled sample data pipeline definition points at a Netos-hosted address;
- in-app documentation links point at
docs.netos.io(11 files in the Netos package), which an offline site cannot reach.
Connectors and data pipelines that you configure to reach other systems will fail if those systems are not reachable from the NetBox host. That is expected; they were not exercised in the offline test.