Quick Thoughts: TrueNAS 25.04 Restoring Default Route

Updated on
TrueNAS Community Edition webGUI shell.

This guide provides a clear, step-by-step procedure to restore a missing default route on TrueNAS 25.04 using the native midclt API. You may encounter this issue when adding new interface cards to your TrueNAS server.

This issue can disable your Docker containers and ability to update the OS. Unfortunately, there's no easily accessible documentation on how to fix it, so here's a tutorial on how to do it.

Prerequisites:

  • SSH, web GUI, or text console access to the TrueNAS shell.
  • Root or equivalent privileges.
  • TrueNAS 25.04 with the interface and route APIs available.

Important:

  • Run all commands from the TrueNAS shell.
  • Keep the Web UI open in another window for reference.
  • Do not interrupt or close the session while a transaction is pending.

Table of Contents

High-Level Overview

The issue: TrueNAS has a gateway configured, but the kernel is missing the default route, causing “Network is unreachable” errors (e.g., when contacting update.ixsystems.com).

The fix: Use TrueNAS’s native interface API to:

  • Create a small, safe interface update.
  • Explicitly attach the configured gateway to the pending transaction.
  • Commit with automatic rollback protection.
  • Verify and finalize.

Phase 1: Pre-Flight Diagnostics

Before changing anything, confirm that TrueNAS recognizes the mismatch and that no stale transaction is blocking us.

Step 1: Check for pending changes

Ensures no previous interface operation is stuck.

midclt call interface.has_pending_changes

Expected: False

Step 2: Check if the default route would be removed

Uses a built-in diagnostic to see if TrueNAS detects the gateway/kernel mismatch.

midclt call interface.default_route_will_be_removed

  • If true: TrueNAS recognizes the issue. Proceed with the repair.
  • If false: The middleware is not detecting the missing route via this path. Stop here and investigate further before modifying the interface.

Phase 2: Confirm Current State

Verify that the gateway is configured but missing from the kernel, and confirm the interface is up.

Step 3: Inspect configuration, interface, and routes

Run the following block as one script. It prints key values in a readable format.

echo "=== CONFIGURATION ==="
midclt call network.configuration.config | python3 -c '
import sys, json
d = json.load(sys.stdin)
print("configured gateway:", repr(d["ipv4gateway"]))
print("runtime gateway:", repr(d["state"]["ipv4gateway"]))
print("DNS1:", repr(d["nameserver1"]))
'

echo
echo "=== INTERFACE ==="
ip -4 addr show dev <INTERFACE NAME>

echo
echo "=== ROUTE ==="
ip -4 route

echo
echo "=== TRUE NAS ROUTES ==="
midclt call route.system_routes | python3 -m json.tool

echo
echo "=== PENDING CHANGES ==="
midclt call interface.has_pending_changes

You should currently see:

configured gateway: '[GATEWAY]'
runtime gateway: ''

And you should not see:

default via [GATEWAY] dev <INTERFACE NAME>

Phase 3: Create a Safe Interface Update

TrueNAS requires an actual interface update before save_default_route can attach the gateway to the pending transaction. We will change only the interface description to avoid affecting networking.

Step 4: Get current interface configuration

Use this to confirm current values (name, IP, MTU, description).

midclt call interface.query '[["name","=","enp32s0"]]' | python3 -m json.tool

Key values should be approximately:

name = <INTERFACE NAME>
ipv4_dhcp= false
ipv6_auto= false
aliases= <SERVER IP>/22
mtu= 1500
description = "<INTERFACE NAME>"

Step 5: Apply a temporary description change

This creates a pending interface transaction without altering IP or routing.

midclt call interface.update "enp32s0" '{"description":"enp32s0-route-repair"}'

Verify that a pending change now exists:

midclt call interface.has_pending_changes

Expected: true

If the existing description is already exactly enp32s0-route-repair, choose a different temporary value.

Phase 4: Inject Gateway and Commit (with Rollback Protection)

Now we attach the configured gateway to the pending transaction and commit it safely.

Step 6: Capture the configured gateway

GATEWAY="$(midclt call network.configuration.config | python3 -c '
import sys, json
print(json.load(sys.stdin)["ipv4gateway"])
')"
echo "GATEWAY=${GATEWAY}"

Confirm it matches your expected gateway, e.g.:

GATEWAY=[GATEWAY]

Step 7: Save the default route into the pending transaction

This is the core step: it tells TrueNAS to include the gateway when the interface changes are applied.

midclt call interface.save_default_route "${GATEWAY}"

Step 8: Verify pending transaction still exists

midclt call interface.has_pending_changes

Expected: true

At this point, the kernel route is still missing. The pending transaction now contains:

database configuration
 │
 ▼
interface.update
 │
 ▼
PENDING interface transaction
 │
 ├── description change
 │
 └── default gateway = [GATEWAY]

Step 9: Commit with rollback protection

This applies the pending changes. If something goes wrong, TrueNAS will automatically roll back.

midclt call interface.commit '{"rollback":true,"checkin_timeout":60}'

Behavior:

  • Applies the pending interface changes.
  • Automatically rolls back if application fails.
  • Waits 60 seconds for confirmation (checkin).

Phase 5: Verify Route and Connectivity

Immediately after commit, confirm the route is restored and external connectivity works.

Step 10: Check the default route

ip -4 route show default

Desired result:

default via [GATEWAY] dev <INTERFACE NAME>

Step 11: Confirm TrueNAS route view

midclt call route.system_routes | python3 -m json.tool

You should see an entry similar to:

{
"network": "0.0.0.0",
"netmask": "0.0.0.0",
"gateway": "[GATEWAY]",
"interface": "<INTERFACE NAME>",
"table_id": 254
}

Step 12: Confirm runtime gateway matches configuration

midclt call network.configuration.config | python3 -c 'import sys, json d = json.load(sys.stdin) print("configured gateway:", repr(d["ipv4gateway"])) print("runtime gateway:", repr(d["state"]["ipv4gateway"]))'

Desired result:

configured gateway: '[GATEWAY]'
runtime gateway: '[GATEWAY]'

This confirms the configuration and kernel states are now aligned.

Step 13: Test external connectivity

Run these before the 60-second checkin timeout expires.

ping -c 4 "${GATEWAY}"
ping -c 4 1.1.1.1
getent hosts update.ixsystems.com
curl -4 -I --connect-timeout 10 https://update.ixsystems.com/

If the default route is correct, the original error:

Cannot connect to host update.ixsystems.com:443
[Network is unreachable]

should no longer appear.

Phase 6: Finalize and Clean Up

Once verification succeeds, confirm the changes and restore the original description.

Step 14: Confirm the committed changes (checkin)

Only run this if the route and connectivity tests pass.

midclt call interface.checkin

Then verify no pending changes remain:

midclt call interface.has_pending_changes

Expected: false

Step 15: Restore original description (optional but recommended)

Perform this as a second, clean transaction.

midclt call interface.update "<INTERFACE NAME>" '{"description":"<INTERFACE NAME>"}'
midclt call interface.commit '{"rollback":true,"checkin_timeout":120}'

Verify:

midclt call interface.query '[["name","=","enp32s0"]]' | python3 -c 'import sys, json d = json.load(sys.stdin)[0] print("description:", d["description"]) print("aliases:", d["aliases"])'

Then finalize:

midclt call interface.checkin

Why This Is the Strongest Repair Path

This procedure:

  • Uses only native TrueNAS APIs (no manual ip route hacks).
  • Preserves existing interface configuration.
  • Explicitly re-attaches the configured gateway via save_default_route.
  • Protects against lockouts with rollback=true and checkin_timeout.
  • Aligns with TrueNAS 25.04 behavior where the default route can be removed if the configured gateway does not match the kernel’s installed gateway.

Recommended Immediate Command

Before doing anything else, run:

midclt call interface.default_route_will_be_removed

  • If true: Proceed with this procedure.
  • If false: Stop before interface.update; the middleware is not recognizing the missing route via the expected mechanism, and further investigation is required.

Script Based Solution

This isn't immediately usable unless you're able to SSH into the TrueNAS server and write the script to a drive in the server, or directly connect a USB drive and copy it onto the server.

#!/usr/bin/env bash
set -euo pipefail

# ==========================
# TrueNAS Gateway Repair Script
# ==========================
# Role: Expert Linux/TrueNAS Shell Script
# Purpose: 
#   - Compare configured vs runtime IPv4 gateway
#   - If runtime is blank/null or mismatched, trigger a safe interface transaction
#     to permanently sync the runtime gateway to the configured gateway.
# Usage:
#   1. Replace <INTERFACE_NAME> with your actual interface (e.g., enp32s0)
#   2. Run as root or via sudo in the TrueNAS shell
# ==========================

INTERFACE="<INTERFACE_NAME>"
TEMP_DESC="route-repair-$(date +%s)"
ORIGINAL_DESC=""

# Helper: run midclt and return JSON (suppress noisy stderr)
midclt_json() {
    midclt call "$@" 2>/dev/null | python3 -c "
    import sys, json
    try:
        print(json.dumps(json.load(sys.stdin), indent=2))
    except:
        print('')
    "
}

# Helper: extract a nested field from JSON using dot notation
get_field() {
    local json_data="$1"
    local field="$2"
    echo "$json_data" | python3 -c "
    import sys, json
    try:
        data = json.load(sys.stdin)
        keys = '$field'.split('.')
        val = data
        for k in keys:
            if isinstance(val, dict):
                val = val.get(k)
            else:
                val = None
                break
        print(val if val is not None else '')
    except:
        print('')
    "
}

echo "=== TrueNAS Gateway Repair Script ==="
echo "Target Interface: $INTERFACE"

# 1. Fetch network configuration
NET_CONFIG_JSON=$(midclt_json network.configuration.config)
CONFIGURED_GW=$(get_field "$NET_CONFIG_JSON" "ipv4gateway")
RUNTIME_GW=$(get_field "$NET_CONFIG_JSON" "state.ipv4gateway")

echo "Configured Gateway: '$CONFIGURED_GW'"
echo "Runtime Gateway:    '$RUNTIME_GW'"

# 2. Compare gateways
if [[ -n "$CONFIGURED_GW" && "$CONFIGURED_GW" == "$RUNTIME_GW" ]]; then
    echo "Gateways match. No repair needed."
    exit 0
fi

if [[ -z "$CONFIGURED_GW" ]]; then
    echo "ERROR: No configured gateway found. Aborting."
    exit 1
fi

echo "Runtime gateway is missing/blank or mismatched. Initiating repair sequence..."

# 3. Diagnostic (informational)
WILL_REMOVE=$(midclt call interface.default_route_will_be_removed 2>/dev/null || echo "false")
echo "default_route_will_be_removed: $WILL_REMOVE"

# 4. Preserve current interface metadata
IFACE_JSON=$(midclt_json interface.query "[[\"name\",\"=\",\"$INTERFACE\"]]")
IFACE_DATA=$(echo "$IFACE_JSON" | python3 -c "
import sys, json
try:
    print(json.dumps(json.load(sys.stdin)[0]))
except:
    print('')
")

if [[ -z "$IFACE_DATA" ]]; then
    echo "ERROR: Could not retrieve configuration for $INTERFACE. Aborting."
    exit 1
fi

ORIGINAL_DESC=$(get_field "$IFACE_DATA" "description")
echo "Original Description: '$ORIGINAL_DESC'"

# 5. Trigger pending changes with a harmless update
echo "Updating interface description to create pending transaction..."
midclt call interface.update "$INTERFACE" "{\"description\":\"$TEMP_DESC\"}"

PENDING=$(midclt call interface.has_pending_changes 2>/dev/null)
if [[ "$PENDING" != "true" ]]; then
    echo "ERROR: interface.update did not create pending changes. Aborting."
    exit 1
fi
echo "Pending changes: $PENDING"

# 6. Inject configured gateway into pending transaction
echo "Saving default route to pending transaction..."
midclt call interface.save_default_route "$CONFIGURED_GW"

# 7. Commit with automatic rollback protection
echo "Committing interface changes (rollback=true, checkin_timeout=60)..."
midclt call interface.commit '{"rollback":true,"checkin_timeout":60}'

# 8. Verify kernel route
echo "Verifying default route..."
sleep 2
ROUTE_OUTPUT=$(ip -4 route show default 2>/dev/null || true)
if ! echo "$ROUTE_OUTPUT" | grep -q "via $CONFIGURED_GW dev $INTERFACE"; then
    echo "WARNING: Default route not found immediately. Dumping system routes for inspection:"
    midclt call route.system_routes | python3 -m json.tool
else
    echo "Default route confirmed: $ROUTE_OUTPUT"
fi

# 9. Confirm changes (cancel rollback)
echo "Confirming changes (checkin)..."
midclt call interface.checkin

# 10. Restore original description (second transaction)
echo "Restoring original description..."
midclt call interface.update "$INTERFACE" "{\"description\":\"$ORIGINAL_DESC\"}"
midclt call interface.commit '{"rollback":true,"checkin_timeout":60}'
midclt call interface.checkin

# 11. Final verification
echo "=== Final Verification ==="
FINAL_NET_JSON=$(midclt_json network.configuration.config)
FINAL_CONFIG_GW=$(get_field "$FINAL_NET_JSON" "ipv4gateway")
FINAL_RUNTIME_GW=$(get_field "$FINAL_NET_JSON" "state.ipv4gateway")

echo "Configured Gateway: '$FINAL_CONFIG_GW'"
echo "Runtime Gateway:    '$FINAL_RUNTIME_GW'"

if [[ "$FINAL_CONFIG_GW" == "$FINAL_RUNTIME_GW" && -n "$FINAL_RUNTIME_GW" ]]; then
    echo "SUCCESS: Runtime gateway now matches configured gateway."
    exit 0
else
    echo "FAILURE: Gateways still do not match. Manual intervention required."
    exit 1
fi

Related posts