Troubleshooting

Start with the two commands Netos support will ask for:

sudo ./netos-install status --site /root/netos/site.yml          # what is installed, last result
sudo ./netos-install support-bundle --site /root/netos/site.yml  # one file to send Netos

Every run writes /var/log/netos/<command>-<timestamp>.log (every command it ran, its exit code and duration; for a failed command, the last 200 lines of its output). A failing command also prints the last 20 lines of its error output on the console.

Bundle signature checks

Message Action
unsigned Ask Netos for a signed bundle. --require-signature makes this a failure.
site has a pinned signing key; unsigned bundle rejected Obtain a signed bundle; use --allow-unsigned only for an explicit unsigned override, which is logged as a warning.
signature valid, key NOT independently trusted Verify again using --trusted-key with the public key obtained separately from Netos support.
no trusted Ed25519 public key is available Obtain the public key from support; a key supplied only inside the bundle is not an independently trusted key; the release key and its ID are in Install.
SHA256SUMS signature verification failed Stop. The checksum list or signature does not match the chosen key; request a new bundle and key from support.
invalid Ed25519 public key or malformed SHA256SUMS signature Stop. Replace the bad key or bundle using copies from support.
SHA256SUMS signature required but bundle is unsigned Obtain a signed bundle, or omit --require-signature while unsigned bundles are allowed.
cryptography is unavailable and OpenSSL pkeyutl -rawin is unavailable Run the installer in the NetBox venv with cryptography, or provide an OpenSSL version that supports pkeyutl -rawin.

Health check: doctor

./netos-install doctor --site /root/netos/site.yml

Read-only. On a healthy test host it printed:

PASS  manage_check
PASS  plugin_versions          {'netos': '4.14.304', 'netos_reporting': '4.9.97', 'netos-model-builder': '4.5.90'}
SKIP  http_landing_pages       authenticated HTTP probe requires temporary user writes
PASS  netos_platform_checks    exit 0
SKIP  pipeline_run_health      command writes data or opens outbound HTTP
PASS  netos_detect_stale_pipeline_runs exit 0
PASS  netos_job_effective_config exit 0
SKIP  validate_doc_links       command writes data or opens outbound HTTP
PASS  pip_check
PASS  licence
PASS  rq                       {'high': {'depth': 0, 'oldest_job': None, 'workers': 1}, 'default': {...}, 'low': {...}}

SKIP lines are checks it deliberately does not run because they would write or reach the network. In rq, a queue whose depth keeps growing with workers: 0 means that worker service is down (systemctl status netbox-rq netbox-rq-low).

Recipes: troubleshooting commands Netos may ask you to run

./netos-install run --list --site /root/netos/site.yml
Recipe What it does Writes?
showmigrations Lists every database migration and whether it is applied No
platform-checks Netos's own platform findings No
stale-pipeline-runs Reports pipeline runs stuck in "running" No
reconcile-insights-permissions Checks the Insights read permission against installed models No (check mode)
collectstatic Rebuilds NetBox's static files Yes: snapshots first
remove-stale-contenttypes Removes content types for models that no longer exist Yes: snapshots first

Run one with ./netos-install run <recipe> --site /root/netos/site.yml. A recipe that writes stops NetBox and takes a snapshot first, like an install. Netos may also send you a new recipe file with its SHA-256; run it with ./netos-install run /path/to/recipe.yml --sha256 <value> --site /root/netos/site.yml, which refuses if the file does not match.

Site report: collect

./netos-install collect --site /root/netos/site.yml --out /root/netos/site-report.json

Read-only and safe to run any time, including before Netos is installed. It records the OS, NetBox and PostgreSQL versions and settings, installed packages, services and the preflight results. Every value not on a fixed list of safe fields is written as present or absent, so the file is safe to send; you can read it first. Netos uses it to prepare a site.yml for you.

Preflight FAIL lines

Line Meaning What to do
netbox_version 4.5.9 This bundle is built for a different NetBox version Get the bundle for your NetBox version from Netos
python_abi The NetBox venv is not Python 3.12 Ask Netos for a bundle for your Python version
licence missing or invalid No licence_file in site.yml, the file is not where it says, or the token does not decrypt with its key Check the path; ask Netos to re-send the licence
rehearsal copied-venv pip check failed: <package> has requirement <x>, but you have <y> Another package in your NetBox venv conflicts with a Netos dependency. Your live venv was not changed Send Netos the line. The fix is in the conflicting plugin or in Netos, never a forced install
database_tools pg_dump/pg_restore missing, or no way to restore the database for rollback Install the PostgreSQL client for your server's major version; for a remote database, see Roll back
disk Not enough room for the snapshot Free space under /var/backups/netos or set snapshot_root in site.yml
configure requires installed Netos plugins in the NetBox venv configure ran before the plugin wheels were installed The supplied licence_file and external_url were saved to site.yml; run install first, then configure if needed

Tested on the offline test host: a wrong NetBox version, a missing licence and a conflicting third-party package each produced exactly one FAIL and left the live venv byte-identical (checked by hashing pip freeze).

Install or upgrade FAIL

The last lines name the failing step and print the rollback command. Read the log for that step first:

sudo less "$(ls -t /var/log/netos/install-*.log | head -1)"

Search for the last RC line that is not RC 0; the STDERR lines under it are the cause. Then either fix the cause and run install/upgrade again, or roll back (Roll back).

The install was interrupted

An SSH drop, a reboot or a killed process during install/upgrade leaves NetBox stopped and possibly half-installed. Run rollback --latest. That case was tested twice (killed during the package step, and during migrations) and both times rollback alone restored the box exactly.

NetBox is slow to come back after install

The NetBox workers import every plugin at start. On the 2 vCPU test host the first page answered about a minute after the services started. The post-check waits up to 5 minutes (http_wait_seconds in site.yml).

The NetBox application

These checks apply to any NetBox with Netos installed. Run them as root on the NetBox host.

Check that NetBox and its workers are running:

systemctl status netbox netbox-rq netbox-rq-low

Restart them:

systemctl restart netbox netbox-rq netbox-rq-low

Read the last 1000 lines of each service's log:

journalctl --no-pager -n 1000 -u netbox
journalctl --no-pager -n 1000 -u netbox-rq
journalctl --no-pager -n 1000 -u netbox-rq-low

Read the web server logs. With the standard NetBox NGINX setup they are:

tail -n 200 /var/log/nginx/netbox.access.log
tail -n 200 /var/log/nginx/netbox.error.log

If your web server or log names differ, use yours. Send the output with ./netos-install support-bundle when you contact Netos support.

Offline behaviour

The bundle never reaches the network during install, upgrade or rollback: the test host's firewall logged zero outbound packets from NetBox across all of them, and across a runtime check of 39 plugin pages and a PDF render with a chart. Known places where the plugins still refer to Netos-hosted addresses (none were contacted in the test):

  • a bundled sample data pipeline definition points at a Netos-hosted address;
  • in-app documentation links point at docs.netos.io (11 files in the Netos package), which an offline site cannot reach.

Connectors and data pipelines that you configure to reach other systems will fail if those systems are not reachable from the NetBox host. That is expected; they were not exercised in the offline test.