Problem Description:
Intermittent
server outages after the server runs out of memory due to excessive memory
usage from the OWSTIMER service. When the issue occurs the server is slow to
respond, even when attempting to access the server remotely. After restarting
the timer service, manually, the server slowly regains stability and is
accessible once again.
Scope Agreement:
This
case will be considered resolved and ready to be closed once we have identified
why your SharePoint servers are experiencing intermittent server outages after
the server runs out of memory due to excessive memory usage from the OWSTIMER
service. A permanent solution or suitable workaround for this problem will be
provided, otherwise Microsoft will document why this symptom should be
considered expected behavior for this particular scenario.
Cause:
The SharePoint Server was
executing a job named “job-diagnostics-performance-counter-wfe-provider”
(Diagnostic Data Provider: Performance Counters - Web Front Ends) and was
running for 20 minutes. While it was executing, this job was consuming a large
amount of memory. During this time a new SPWeb Request was executing on threads
~46 and ~6 and created a new function
"Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPPerformanceCounterAgent.UpdateSharePointPerfmonCounters".
This SPRequest Object become stuck in a waiting state, presumably waiting on
~33 to complete.
It seems as soon as this job
started, it began consuming large blocks of memory until the system ran out of
memory.
From Thread 33:
Thread
Time
33:211c 0 days 0:20:31.347
Job:
job-diagnostics-performance-counter-wfe-provider
Title: Diagnostic Data Provider:
Performance Counters - Web Front Ends
m_StartTime -mt 000007FEF7427F20
000000000D9E53F0 10/23/2014 8:03:00
PM
The dump was triggered at 8:23PM
(20 minutes later) and is the execution time of job "Diagnostic Data
Provider: Performance Counters - Web Front Ends".
Resolution:
To resolve this issue, we
disabled the "Diagnostic Data Provider: Performance Counters - Web Front
Ends" and the issue did not reoccur.
More Information:
At this point we know what caused
it, and how to mitigate it. However, we don’t know exactly why this job was
communing so much memory. We can see the job was generating exception “Unable
to create system performance counter. It possible that cleaning up these
exceptions may resolve the issue with executing job "Diagnostic Data
Provider: Performance Counters - Web Front Ends".
However, according to our
guidance on this job, it should be used during the planning stages of a
deployment for capacity planning and then later disabled.
Monitoring and maintaining
SharePoint Server 2010
From Article:
Below is a list of the settings that you can modify to monitor
your environment in its early stages, which will help you determine if any
changes are needed. Keep in mind that increasing your monitoring capabilities
will affect the amount of disk space that your usage database will require.
Once the environment is stable and this detailed monitoring is no longer
required, you may want to reverse the settings below to their defaults.
Dump Analysis:
User Mode Time
Thread
Time
33:211c 0 days 0:20:31.347
6:3050
0 days 0:00:14.695
45:3700
0 days 0:00:00.000
0:033>
Call Site
kernel32!GetUserDefaultLocaleName
kernel32!CheckSpecialLocales
kernel32!lstrcmpiW
advapi32!PerflibciFindOutputInstance
advapi32!PerflibciLocalQueryCounterData
advapi32!PerfQueryCounterData
pdh!PdhiGetPerflibCounterData
pdh!GetQueryPerfData
pdh!PdhiCollectQueryData
pdh!PdhCollectQueryData
mscorwks!DoNDirectCall__PatchGetThreadCall
Microsoft_SharePoint_ni!DomainBoundILStubClass.IL_STUB
Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPPerformanceCounter.NextValue
Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPPerformanceCounterMonitorInternal.UpdateValue
Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPPerformanceCounterMonitorInternal.get_NextValue
Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPSystemPerformanceCounterMonitor.get_Value
Microsoft_SharePoint_ni!Microsoft.SharePoint.Diagnostics.SPDiagnosticsPerformanceCounterProvider.CollectCounter
Microsoft_SharePoint_ni!Microsoft.SharePoint.Diagnostics.SPWebFrontEndDiagnosticsPerformanceCounterProvider.Sync
Microsoft_SharePoint_ni!Microsoft.SharePoint.Diagnostics.SPDiagnosticsProvider.Execute
Microsoft_SharePoint_ni!Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.InvokeMicrosoft_SharePoint_ni!Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invokemscorwks!CallDescrWorker
mscorwks!CallDescrWorkerWithHandler
mscorwks!ForwardCallToManagedMethod
mscorwks!COMToCLRWorkerBody
mscorwks!COMToCLRWorkerDebuggerWrapper
mscorwks!COMToCLRWorker
mscorwks!GenericComCallStub
STSWEL!VwrapTimerInvoke::Invoke
STSWEL!VtimerInvoke::HrInvokeManagedCodeHandler
STSWEL!VtimerInvoke::HrInvokeJobDefinition
STSWEL!VtimerInvoke::HrInvoke
OWSSVR!CSPRequest::InvokeTimerJob
OWSTIMER!VTimerProcessRequest
ONETUTIL!COWSThreadPool::COWSJob::Execute
ONETUTIL!COWSThreadPool::ThreadHandlerProc
msvcr90!_callthreadstartex
msvcr90!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Job:
job-diagnostics-performance-counter-wfe-provider
Title: Diagnostic Data Provider:
Performance Counters - Web Front Ends
m_StartTime -mt 000007FEF7427F20
000000000D9E53F0 10/23/2014 8:03:00 PM
SPRequst Summary:
Undisposed SPRequest objects: 1
Disposed SPRequest objects: 5
Undisposed SPWeb objects: 0
Disposed SPWeb objects: 29
Undisposed SPSite objects: 0
Disposed SPSite objects: 25
undisposed special purpose
(AllowCleanupWhenThreadEnds = false) SPRequest object found at:
0x0000000006755718.
Undisposed SPRequest Objects per
managed Thread:
Thread ID: 3700, Undisposed
SPRequest: 1
Undisposed SPRequest Objects on
unmanaged or no longer existing Threads: 0
Name:
Microsoft.SharePoint.Library.SPRequest
MethodTable: 000007feee68bd38
EEClass: 000007feeba6f018
Size: 88(0x58) bytes
GC Generation: 2
(C:\Windows\assembly\GAC_MSIL\Microsoft.SharePoint\14.0.0.0__71e9bce111e9429c\Microsoft.SharePoint.dll)
Fields:
MT
Field
Offset
Type VT
Attr Value
Name
000007fef1472748
4000009 8 ...uestInternalClass
0 instance 6755770
m_SPRequest
000007fef73d7c30
400000a
28 System.Int32 1
instance
1 m_ApartmentState
000007fef73eec50
400000b
2c System.Int32 1
instance
6 m_ManagedThreadId
000007fef73eec50
400000c
30 System.Int32 1
instance 14080 m_UnmanagedThreadId
000007fef73e22e8
400000d
40 System.Guid 1
instance 0000000006755758 m_AllocationId
000007fef73e79e0
400000e
10 System.String 0
instance 6214cb0 m_Name
000007fef73e6bb0
400000f
38 System.Boolean 1
instance
0 m_AllowCleanupWhenThreadEnds
000007fef73e6bb0
4000010
39 System.Boolean 1
instance
0 m_Disposed
000007fef73e79e0
4000011
18 System.String 0
instance
0 m_StackTrace
000007fef73e79e0
4000013
20 System.String 0 instance
0
m_StackTraceOnRelease
000007fef73eec50
4000014
34 System.Int32 1
instance
0 m_UnmanagedStackCount
000007fef73e6bb0
4000012 1d50
System.Boolean 1
static
0 s_CollectStackTraceOnCreate
Managed Thread ID = 6
Unmanaged Thread ID = 14080
/ 3700 (dec)
45 Id: 1f0c.3700
Suspend: 0 Teb: 000007ff`ffeb8000 Unfrozen
0:045>
Call Site
ntdll!ZwDelayExecution
kernel32!SleepEx
mscorwks!EESleepEx
mscorwks!Thread::UserSleep
mscorwks!ThreadNative::Sleep
Microsoft_SharePoint_ni!Microsoft.SharePoint.Utilities.SPPerformanceCounterAgent.UpdateSharePointPerfmonCounters
mscorlib_ni!System.Threading.ExecutionContext.Run
mscorlib_ni!System.Threading.ThreadHelper.ThreadStart
mscorwks!CallDescrWorker
mscorwks!CallDescrWorkerWithHandler
mscorwks!MethodDesc::CallDescr
mscorwks!ThreadNative::KickOffThread_Worker
mscorwks!ManagedThreadBase_DispatchInner
mscorwks!ManagedThreadBase_DispatchMiddle
mscorwks!ManagedThreadBase_DispatchOuter
mscorwks!ManagedThreadBase_FullTransitionWithAD
mscorwks!ThreadNative::KickOffThread
mscorwks!Thread::intermediateThreadProc
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
000007fef73eec50
System.Int32 +0054
m_ManagedThreadId
6 (0n6)
0:006>
Call Site
ntdll!ZwWaitForMultipleObjects
kernel32!WaitForMultipleObjectsEx
ole32!CoWaitForMultipleHandles
ONETUTIL!VwaitForMultipleObjects
ONETUTIL!COWSThreadPool::ThreadHandlerProc
msvcr90!_callthreadstartex
msvcr90!_threadstartex
kernel32!BaseThreadInitThunk
ntdll!RtlUserThreadStart
Managed Stack:
Address
Method Table Heap Gen Size Type
0000000006c37200
000007feee68d930 0
2 64
Microsoft.SharePoint.Utilities.SqlSession
0000000006c37710
000007fef73e79e0 0
2 42 System.String PRDAdmin
000000000f5e7708
000007fef73e79e0 0
2 48 System.String 10.32.6.180
000000000f5e7738
000007fef73e79e0 0
2 34 System.String POST
000000000705e8d0
000007feee68d338 0 2
24 Microsoft.SharePoint.Diagnostics.ULSCat
0000000006e80cf8
000007fef73e79e0 0
2 110 System.String Error collecting perfmon counter
{0}. {1}
0000000006211308
000007fef73e79e0 0
2 26 System.String
00000000069eb340 000007feee782578
0 2 288
Microsoft.SharePoint.Diagnostics.SPWebFrontEndDiagnosticsPerformanceCounterProvider
00000000069eb0f8
000007fef73e79e0 0
2 72 System.String Processes
Started Total
00000000069eae80
000007fef73e79e0 0
2 80 System.String Sandboxed
Code Process Pool
000000000705e788
000007fef73e79e0 0
2 80 System.String
StackTrace_InFileLineNumber
000000000f8e6920
000007fef73eec50 0
2 24 System.Int32
0000000006672a18
000007fef73f0360 0
2 120 System.Globalization.CultureInfo
0000000006d466f8
000007fef742b638 0
2 88
System.Collections.Generic.Dictionary`2[[System.String,
mscorlib],[System.Resources.ResourceLocator, mscorlib]]
0000000006d466b0
000007fef73ed788 0
2 72 System.Resources.RuntimeResourceSet
00000000069e66d0
000007feee782cd8 0
2 288
Microsoft.SharePoint.Diagnostics.SPDatabaseServerDiagnosticsPerformanceCounterProvider
000000000666cd50
000007fef73e79e0 0
2 42 System.String PRDAdmin
000000000674c568 000007fef73d5c00
0 2 32
Microsoft.Win32.SafeHandles.SafeWaitHandle
000000000d9b9bc0
000007fee9d59f50 0
2 248
System.Data.SqlClient.SqlInternalConnectionTds
000000000674c458
000007fee9d5b0d0 0
2 176
System.Data.ProviderBase.DbConnectionPool
000000000621d6a0
000007feee697788 0
2 384
Microsoft.SharePoint.Administration.SPConfigurationDatabase
0000000006220b38
000007fef33fbdd8 0
2 96
System.Threading.ReaderWriterLockSlim
0000000006220b08
000007feee6e4590 0 2
48 Microsoft.SharePoint.Utilities.SPReaderWriterLock
0000000006212058
000007feee68d338 0
2 24
Microsoft.SharePoint.Diagnostics.ULSCat
00000000069ed580
000007feee6fda90 0
2 232
Microsoft.SharePoint.Administration.SPConfigurationRefreshJobDefinition
0000000006c8b700
000007feee7b8a60 0
2 24
Microsoft.SharePoint.Administration.SPTimerJobInvoke
00000000066667c0
000007fef73e9cc0 0
2 40 System.RuntimeType
00000000067f6bf0
000007feee6f4808 0
2 248 Microsoft.SharePoint.Administration.SPManagedAccount
0000000006265940
000007fef73e9cc0 0
2 40 System.RuntimeType
000000000621b1a0
000007fef73e79e0 0
2 62 System.String
WSS_Administration
00000000067f6ad0
000007fef73e79e0 0
2 120 System.String
S-1-5-21-1161975898-3023619224-890137498-192915
0000000006211d28
000007feee7b82c8 0
2 24
Microsoft.SharePoint.Administration.SPNativeConfigurationProvider
0000000006795840
000007feee68ea68 0 2
40 Microsoft.SharePoint.Administration.SPServiceCollection
Details:
Job:
job-diagnostics-performance-counter-sql-provider
Title: Diagnostic Data Provider:
Performance Counters - Database Servers
No comments:
Post a Comment