Recently, I was helping a customer who ran into an issue during an AutoUpgrade workflow where the database shutdown phase appeared to hang indefinitely. The database is hosted on Oracle Exadata Database Service on Dedicated Infrastructure at Customer.
The shutdown process never completed, even after more than 12 hours.
To be clear, it was not an AutoUpgrade issue!
Symptoms
During the AutoUpgrade DRAIN phase, the following messages repeatedly appeared:
Process termination requested for pid 321321 (oracleNIDD0099) [source = rdbms], [info = 2] [request issued by pid: 123456, uid: 1001]
SHUTDOWN: Active sessions prevent database close operation
Shutdown did not complete in 60 minutes
Instance shutdown cancelled
At first, this looked like an AutoUpgrade issue or a session cleanup problem. However, after further investigation, we confirmed the root cause was unrelated to AutoUpgrade itself.
Root Cause
The issue was caused by an excessive number of entries in REG$.
This is a known issue in OCI environments and can significantly delay the database shutdown process, especially during the DRAIN phase executed by AutoUpgrade.
Because of the large number of REG$ entries, internal EMON cleanup and unregister operations became extremely slow, causing the shutdown to stall for hours.
Workaround
The following hidden parameters resolved the issue:
"_emon_outbound_connect_timeout"=300000
"_emon_send_timeout"=300000
"_client_enable_auto_unregister"=true
After applying these parameters, the shutdown and drain phases completed successfully.
Test Results
| Scenario | Result |
|---|---|
| Test 1 – Single PDB | ~2 hours 30 minutes |
| Test 2 – Multiple PDBs | ~2 hours 36 minutes |
The second test was using the Distributed Upgrade feature:
distributed_upgrade=true
active_nodes_limit=4
Summary
- This is not primarily an AutoUpgrade bug
- The issue is tied to excessive
REG$entries and EMON unregister behavior - AutoUpgrade simply exposes the issue during the DRAIN/shutdown phase
- If the shutdown hangs for hours during DRAIN, check for this condition early
- Applying the timeout and auto-unregister parameters can dramatically improve completion time
Hopefully, this helps anyone troubleshooting long-running shutdowns or AutoUpgrade drain phases in OCI environments.
