Clear System VM IP from NICs for PublicNetworkGuru

[Pearl1594]

Description

Fixes: #10872
This PR cleanup the IP details for the public NIC associated to the systemVM to follow consistency with how control & management NICs are cleaned up

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

Before Fix:

On deleting SSVM id 3: (both public & control nic)

*************************** 1. row ***************************
            id: 7
          uuid: 77430159-8fd2-41c3-adbc-b999ed9d4ed5
   instance_id: 3
   mac_address: 1e:00:48:00:00:02
   ip4_address: 10.0.52.82
       netmask: 255.255.240.0
       gateway: 10.0.48.1
       ip_type: Ip4
 broadcast_uri: vlan://51
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: vlan://51
   ip6_address: NULL
   default_nic: 1
       vm_type: SecondaryStorageVm
       created: 2025-11-05 11:40:27
       removed: 2025-11-05 12:27:51
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
*************************** 2. row ***************************
            id: 8
          uuid: 476f87b8-c5fb-4f00-a9db-e6fc237f8450
   instance_id: 3
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 202
          mode: Static
         state: Deallocating
      strategy: Start
 reserver_name: ControlNetworkGuru
reservation_id: 8d17740d-353e-419f-941c-3cd4d818e49d
     device_id: 0
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 0
       vm_type: SecondaryStorageVm
       created: 2025-11-05 11:40:27
       removed: 2025-11-05 12:27:51
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL

How Has This Been Tested?

After fix:

On deleting SSVM id 3: (both public & control nic)

mysql> select * from  nics where instance_id = 10\G
*************************** 1. row ***************************
            id: 26
          uuid: f51c03aa-09c4-46e2-8d93-9fe5c413c3a0
   instance_id: 10
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 1
       vm_type: SecondaryStorageVm
       created: 2025-11-05 16:31:22
       removed: 2025-11-05 17:18:25
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
*************************** 2. row ***************************
            id: 27
          uuid: 93eb165c-a055-4eca-9c6c-185b3f6ab341
   instance_id: 10
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 202
          mode: Static
         state: Deallocating
      strategy: Start
 reserver_name: ControlNetworkGuru
reservation_id: 9156f5bb-2c3f-410a-8b1d-6c6059ca4ac0
     device_id: 0
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 0
       vm_type: SecondaryStorageVm
       created: 2025-11-05 16:31:22
       removed: 2025-11-05 17:18:25
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

❌ Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.56%. Comparing base (d160731) to head (17ba242).
⚠️ Report is 243 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...tack/engine/orchestration/NetworkOrchestrator.java	0.00%	4 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #11992      +/-   ##
============================================
- Coverage     17.56%   17.56%   -0.01%     
+ Complexity    15545    15543       -2     
============================================
  Files          5909     5909              
  Lines        529056   529620     +564     
  Branches      64617    64920     +303     
============================================
+ Hits          92947    93009      +62     
- Misses       425654   426156     +502     
  Partials      10455    10455              

Flag	Coverage Δ
uitests	3.53% <ø> (-0.06%)	⬇️
unittests	18.63% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[weizhouapache]

@Pearl1594
When system vms are destroyed, will the new system vms always be allocated with the old public IP ?

I see the benefit of keeping the public IPs, from users' perspective.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
26.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 15657

[DaanHoogland]

@blueorangutan test keepEnv

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-14778)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 48969 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11992-t14778-kvm-ol8.zip
Smoke tests completed. 149 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@Pearl1594 When system vms are destroyed, will the new system vms always be allocated with the old public IP ?

I see the benefit of keeping the public IPs, from users' perspective.

when a new nic/ssvm is created this is a leakages as the old one does not get released. Do you have an alternative to releasing it this way @weizhouapache ?

tested and works as expected.

[weizhouapache]

when a new nic/ssvm is created this is a leakages as the old one does not get released. Do you have an alternative to releasing it this way @weizhouapache ?

I do not know if the ip needs to be released, probably reserve the IP as what ACS does on public IP of VR ?

is the public IP of the system VM (CPVM,SSVM) changed in your testing ?
If so, it might cause issues for users who use a reverse proxy in front of CloudStack.

[shwstppr]

sysvm-clearpublicip.mp4

LGTM. Tried recereating system VMs multiple times and IPs for nics with PublicNetworkGuru were cleared while I didn't see public IPs change for new system VMs.

mysql> SELECT * FROM nics WHERE instance_id IN (3,4) AND reserver_name='PublicNetworkGuru'\G
*************************** 1. row ***************************
            id: 7
          uuid: a0f0cc87-bfc0-4d3b-b0ef-849692905fa5
   instance_id: 3
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 1
       vm_type: SecondaryStorageVm
       created: 2026-01-30 06:06:49
       removed: 2026-01-30 06:08:08
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
*************************** 2. row ***************************
            id: 10
          uuid: 23804b39-ef02-485e-9792-5a068deb8017
   instance_id: 4
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 1
       vm_type: ConsoleProxy
       created: 2026-01-30 06:06:49
       removed: 2026-01-30 06:08:08
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
2 rows in set (0.00 sec)

[shwstppr]

@weizhouapache can you please check?

[weizhouapache]

@weizhouapache can you please check?

@shwstppr
I think there is a potential issue when

• destroy a system vm
• ACS scans the system vms every 30 seconds.
• Before new system vm is created, assign the public IP to a network
• the new system vm might use another public IP

for testing, we may disable the zone, then destroy a systemvm, assign the public Ip to network, enable the zone, etc

[kiranchavala]

LGTM

Tested manually

mysql> SELECT * FROM nics WHERE instance_id IN (1,2) AND reserver_name='PublicNetworkGuru'\G
*************************** 1. row ***************************
            id: 1
          uuid: b5d76a34-63c3-413d-8417-56b92335ed3d
   instance_id: 1
   mac_address: 1e:00:6d:00:00:01
   ip4_address: 10.0.57.201
       netmask: 255.255.240.0
       gateway: 10.0.48.1
       ip_type: Ip4
 broadcast_uri: vlan://51
    network_id: 200
          mode: Static
         state: Reserved
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: vlan://51
   ip6_address: NULL
   default_nic: 1
       vm_type: ConsoleProxy
       created: 2026-03-10 11:10:58
       removed: NULL
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
*************************** 2. row ***************************
            id: 4
          uuid: dfbbc168-bab9-43e1-8e75-e8e812c26601
   instance_id: 2
   mac_address: 1e:00:ad:00:00:02
   ip4_address: 10.0.57.202
       netmask: 255.255.240.0
       gateway: 10.0.48.1
       ip_type: Ip4
 broadcast_uri: vlan://51
    network_id: 200
          mode: Static
         state: Reserved
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: vlan://51
   ip6_address: NULL
   default_nic: 1
       vm_type: SecondaryStorageVm
       created: 2026-03-10 11:10:58
       removed: NULL
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
2 rows in set (0.00 sec)

After destroying

mysql> SELECT * FROM nics WHERE instance_id IN (1,2) AND reserver_name='PublicNetworkGuru'\G
*************************** 1. row ***************************
            id: 1
          uuid: b5d76a34-63c3-413d-8417-56b92335ed3d
   instance_id: 1
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 1
       vm_type: ConsoleProxy
       created: 2026-03-10 11:10:58
       removed: 2026-03-11 07:44:49
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
*************************** 2. row ***************************
            id: 4
          uuid: dfbbc168-bab9-43e1-8e75-e8e812c26601
   instance_id: 2
   mac_address: NULL
   ip4_address: NULL
       netmask: NULL
       gateway: NULL
       ip_type: NULL
 broadcast_uri: NULL
    network_id: 200
          mode: Static
         state: Deallocating
      strategy: Create
 reserver_name: PublicNetworkGuru
reservation_id: NULL
     device_id: 2
   update_time: NULL
 isolation_uri: NULL
   ip6_address: NULL
   default_nic: 1
       vm_type: SecondaryStorageVm
       created: 2026-03-10 11:10:58
       removed: 2026-03-11 07:44:49
   ip6_gateway: NULL
      ip6_cidr: NULL
  secondary_ip: 0
   display_nic: 1
           mtu: NULL
2 rows in set (0.00 sec)

Also tested by disable the zone, then destroy a systemvm, assign the public Ip to network, enable the zone,

The new systemvm got a new public ip assigned

[weizhouapache]

I will test too

please wait before merging

[weizhouapache]

cc @DaanHoogland @kiranchavala @shwstppr

My testing results

• Public IPs of System VMs

• Acquire the SSVM IP, failed as expected

(localcloud) 🐱 > associate ipaddress networkid=9e6e855f-7bb7-4911-9445-f1c1a46a11d1 ipaddress=10.0.55.101
🙈 Error: (HTTP 533, error code 9999) Insufficient address capacity

• Destroy the SSVM, acquire the IP for an isolated network again

(localcloud) 🐱 > associate ipaddress networkid=9e6e855f-7bb7-4911-9445-f1c1a46a11d1 ipaddress=10.0.55.101
{
  "ipaddress": {
    "account": "admin",
    "allocated": "2026-03-11T08:02:19+0000",
    "associatednetworkid": "9e6e855f-7bb7-4911-9445-f1c1a46a11d1",
    "associatednetworkname": "admin-001",
    "domain": "ROOT",
    "domainid": "a1d2ca63-ba1a-11f0-9974-1e00b5000454",
    "domainpath": "ROOT",
    "fordisplay": true,
    "forprovider": false,
    "forsystemvms": false,
    "forvirtualnetwork": true,
    "hasannotations": false,
    "hasrules": false,
    "id": "897dce44-31e7-4dfa-9658-2f3a3bb88af7",
    "ipaddress": "10.0.55.101",
    "isportable": false,
    "issourcenat": false,
    "isstaticnat": false,
    "issystem": false,
    "networkid": "76dc7841-9807-4993-82dd-db0cb8eb7ee7",
...
  }
}

• new SSVM has another IP

[weizhouapache]

my summary:

• the public IP of System VM is cleared when destroying it
• Public IP then can be assigned to isolated network or vpc for use
• new System VM will be allocated a new public IP
• Users might need to update DNS records (for system VMs), or reverse proxy to use the new public IP.

cc @DaanHoogland @Pearl1594
we need to assess the impact before merging

[Pearl1594]

Thanks @weizhouapache DO you suggest that we not go ahead with this and let the current behaviour be?

[weizhouapache]

Thanks @weizhouapache DO you suggest that we not go ahead with this and let the current behaviour be?

@Pearl1594
Perhaps we could discuss this with a wider group.

cc @DaanHoogland @NuxRo @rajujith @ingox

[weizhouapache]

I had a testing without this PR, and got the same result

• system VM IP cannot be associated to an isolated network
• when system VM is destroy, the IP can be associated to an isolated network (zone can be disabled so that system VM is not created automatically)
• new system VM is allcoated with another public IP (enable the zone if it is disabled)

my opinion is

• this PR does not affect the behaviour
• this PR fixes an issue with DB (@DaanHoogland what's the impact on user ?)
• the behaviour needs to be improved.
• we can merge this PR and improve later.

cc @DaanHoogland