Hi
After moving an ASE db to a different physical server by unmount/re-mounting SAN disks (no ASE reconfiguration), one big query is slower than before.
The ASE15.7 SP62 database was moved:
from an Oracle M4000 server (no Zone/VM, no ZFS, ASE devices in UFS filesystems mounted in directio)
to a M10-4 server (in a LDOM/VM, capped ZFS for the OS filesystem only, ASE devices in UFS filesystems mounted in directio)
=> similar: same ASE config, same SAN with SSDs, same RAM amount, same number of engines (24 in threaded-mode kernel)
=> different: faster cpu cores (3.4GHz to 2.5GHz) in different server, one layer of virtualization (hypervisor), ZFS
After the move, some SPs&queries are faster but the biggest SP doubled the time on the new server => 14 mins compared to 7 mins before.
We extracted the slowest query from the SP, updated statistics, rebuild/recreated index => still 14 mins.
sp_sysmon did not show anything interesting except the abnormaly high "fully logged DML":
Fully Logged DMLs
by Full ULC 22.1 2.8 19908 52.8 %
by Single Log Record 6.8 0.9 6100 16.2 %
Then we tried to send the query result to a temporary table #test using "SELECT INTO" => 3 mins !!!
So we thought that the query plan was wrong. We compared the query plans but did not see much difference:
similar abstract time, very closed statistics IO and time excluding the SELECT INTO part.
During my tests, I had tried to use "set statement_cache off" but it did not make any difference.
But then today I rebooted ASE, increased the procedure cache from 1800000 to 3000000 (still 14 mins)
and set "set statement_cache off" before the query => 3 mins !!!
So it is not a wrong query plan because it is fast when ignoring previous query plan.
It was confirmed by:
- stopping all db activity,
- running "dbcc purge purgesqlcache" to clean the statement cache
- checking there is no row in monCachedStatement
- running the query => still 14 mins
- monCachedStatement shows a huge number of IOs:
SSQLID UseCount StatementSize MinPlanSizeKB MaxPlanSizeKB CurrentUsageCount MaxUsageCount NumRecompilesSchemaChanges NumRecompilesPlanFlushes HasAutoParams ParallelDegree QuotedIdentifier TransactionIsolationLevel TransactionMode SAAuthorization SystemCatalogUpdate MetricsCount MinPIO MaxPIO AvgPIO MinLIO MaxLIO AvgLIO MinCpuTime MaxCpuTime AvgCpuTime MinElapsedTime MaxElapsedTime AvgElapsedTime AvgScanRows MaxScanRows AvgQualifyingReadRows MaxQualifyingReadRows AvgQualifyingWriteRows MaxQualifyingWriteRows LockWaits LockWaitTime SortCount SortSpilledCount TotalSortTime MaxSortTime ParallelDegreeReduced ParallelPlanRanSerial WorkerThreadDeficit DBName CachedDate LastUsedDate LastRecompiledDate OptimizationGoal OptimizerLevel
----------- ----------- ------------- ------------- ------------- ----------------- ------------- -------------------------- ------------------------ ------------- -------------- ---------------- ------------------------- --------------- --------------- ------------------- ------------ ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- ----------- -------------- -------------- -------------- ----------- ----------- --------------------- --------------------- ---------------------- ---------------------- ----------- ------------ ----------- ---------------- ------------- ----------- --------------------- --------------------- ------------------- ------------------------------ ------------------- ------------------- ------------------- ------------------------------ ------------------------------
1775483371 1 5296 94 176 0 1 0 0 1 1 0 1 32 1 0 1 1754 1754 1754 983402953 983402953 983402953 802089 802089 802089 807229 807229 807229 3162300 3162300 83403 83403 0 0 0 0 21 7 802044 802044 0 0 0 prodOdr Mar 2 2016 5:02PM Mar 2 2016 5:16PM NULL allrows_oltp ase_current
QUESTION: why disabling statement_cache at the session level or sending the result to #temp would be faster?
(and of course what is the link with changing the server => faster cpu and memory requires cache reconfig?)
Thanks. Vincent