You are here: > ESRI Forums > arcgis desktop discussion forums > Thread Replies

ArcGIS Desktop Discussion Forums

ArcGIS Desktop - Geoprocessing Scripting (Python, JavaScript, VB) forum

GP memory leak test   Dave Bollinger Nov 30, 2005
Re: GP memory leak test   Dan Patterson Dec 03, 2005
Re: GP memory leak test   Dave Bollinger Dec 06, 2005
Re: GP memory leak test   Dan Patterson Dec 13, 2005
Re: GP memory leak test   Dave Bollinger Dec 13, 2005
Re: GP memory leak test; keywords: Toolbox...   Bill Rose Feb 21, 2006
Re: GP memory leak test; keywords: Toolbox...   Tom colson Mar 03, 2006
Re: GP memory leak test; keywords: Toolbox...   Luke Rogers Mar 14, 2006
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject GP memory leak test 
Author Dave Bollinger 
Date Nov 30, 2005 
Message For those following the other "leak" threads...

Attached is a more-or-less ready-to-run batch processing leak test. (assuming I can attach a 2.6M file) There's a readme in the zip file describing set up.

The purpose of this test was to determine if Python's memory allocator was to blame, or if the leak was internal to the geoprocessor.

So a functionally identical script was written in each of Python, JScript and VBScript. (note that JScript and VBScript, though interpreted by same executable, each use a different memory allocation/garbage collection scheme) Each script ran the same process against the same data. The processing/sample data was chosen to be big enough to create a "noticeable" leak, but hopefully not so big that it would blow up a typical system. (expect about a 120M leak) In the end, each script leaked essentially the same amount (+/- ~1M)

This, to me, suggests that the memory leak is internal to the geoprocessor. So upgrading Python, or patching its allocator, isn't likely to make a significant difference.

If anyone else gets the chance to try this test I'd appreciate independent verification - can you confirm or deny that script environment does/does not affect the leak? Thanks.
 
  Dave Bollinger
Programmer/Analyst
San Joaquin County, CA 
  BatchProcessTest.zip (opens in new window)
 
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test 
Author Dan Patterson 
Date Dec 03, 2005 
Message Dave
I had issues as well (mostly bombing out). I noticed that most of the "leak" issues seemed to occur when people were running python scripts not associated with tools in arctoolbox. The attached toolbox contains the batch folder clip script (minimal error checking) and I ran it on 50 shapefiles (5 x 10 of your original versions) without issue. This was in version 9.0 on a low end machine. Could you try this on your larger dataset(s) if you have them. I am just trying to figure out whether python scripts associated with tools in arctoolbox run within arcmap have better control over memory management than those run outside of an arcgis environment. 
  Geomatics, Carleton University, Ottawa, Canada 
  ClipTest.zip (opens in new window)
 
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test 
Author Dave Bollinger 
Date Dec 06, 2005 
Message Dan,
I couldn't get your handy script to fail on data I have readily available. I have a fat machine, and though clip alone will reveal a leak it's not enough on its own to bomb with our data, typically needs a more elaborate multi-step batch process. So I'm not sure if my results are helpful.

However, it is interesting to note that when running the script from the toolbox it's ArcCatalog.exe, not python.exe, that leaks. (just by watching processes in Task Manager as it was running)
 
  Dave Bollinger
Programmer/Analyst
San Joaquin County, CA 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test 
Author Dan Patterson 
Date Dec 13, 2005 
Message a service pack is being issued this month that is supposed to address the batch clip leak
http://support.esri.com/index.cfm?fa=downloads.patchesServicePacks.viewPatch&PID=43&MetaID=1088 
  Geomatics, Carleton University, Ottawa, Canada 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test 
Author Dave Bollinger 
Date Dec 13, 2005 
Message This is either good news, or great news, depending on how thoroughly they fixed the problem...

>> CQ00282211 - Fixes a memory leak with the geoprocessor Clipping operation.

It concerns me that the bug is worded so specifically to the clip operation. I hope its a more general fix of leaks within the geoprocessor (or at least within the analysis tools).

FYI, the identity operation currently leaks even worse than clip. So if you put those together, Bad Things happen. For example a workflow like this:

for each input point file:
clip out those that fall within overall aoi
identity against polys, to assign a "reporting district", for example
 
  Dave Bollinger
Programmer/Analyst
San Joaquin County, CA 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test; keywords: Toolbox clip tool .net 
Author Bill Rose 
Date Feb 21, 2006 
Message I am using VB.NET to create various geoprocessing tools within stand-alone apps, intended for command-line execution. Each application uses a combination of ESRI Tools, each of which is embedded in its own class that uses a "ToolHelper" to set parameters and manage the execution. The ToolHelper class is included below.

In general, this has worked pretty well, even for looping Identity operations. However, I'm seeing issues similar to those posted here in my wrapper for the Clip Tool. Looping clip operations do complete, but result in the dreaded "The instruction at... memory could not be 'read'..." error on exit, or they simply leave the process hanging on exit.

I've tried every conceivable combination of disposal methods for all apparent AO references: CoFreeUnusedLibraries, Marshal.ReleaseComObject, AOUninitialize.Shutdown, and Utility.COMSupport.AOUninitialize.Shutdown. I'm pretty sure there are no un-disposed references, if for no other reason than the other two applications I've developed this way had no such problem (and one of these uses Identity).

The test system is 9.1 sp 1, with the Feb 16th geometry performance patch. The patch was installed today, and seems to have made no difference at all.

This seems to me to be a bug. And I'm wondering if the problems described in this thread have been resolved with the latest patch.
 
 
Public Class ToolHelper

        Implements IDisposable

#Region "        Identifiers "
        Private Const mc_moduleName As String = fc_projectName & _
                                                ".ToolboxHelper.ToolHelper"

        Protected m_toolBox As Geoprocessing.IGPToolbox
        Protected m_tool As Geoprocessing.IGPTool
        Protected m_params As esriSystem.IArray
        Protected m_paramIndex As Int32 = 0
#End Region

#Region "        Properties "
        Public ReadOnly Property ParameterCount() As Int32
            Get
                Return m_params.Count
            End Get
        End Property
#End Region

#Region "        Initialization and disposal "
        Public Sub New(ByVal toolBox As Geoprocessing.IGPToolbox, _
                       ByVal toolName As String)
            m_toolBox = toolBox
            m_tool = m_toolBox.OpenTool(toolName)
            m_params = m_tool.ParameterInfo
        End Sub
        Public Overridable Sub Dispose() Implements System.IDisposable.Dispose
            ReleaseComObject(DirectCast(m_toolBox, Object))
            ReleaseComObject(DirectCast(m_tool, Object))
            ReleaseComObject(DirectCast(m_params, Object))
        End Sub
#End Region

#Region "        SetParameter and Execute functions "
        Public Overridable Overloads Function SetParameter( _
                                    ByVal value As String) As Boolean

            Dim param As Geoprocessing.IGPParameter
            Dim paramEdit As Geoprocessing.IGPParameterEdit
            Dim paramDataType As Geodatabase.IGPDataType

            Try
                If m_paramIndex < m_tool.ParameterInfo.Count() Then

                    param = DirectCast(m_params.Element(m_paramIndex), _
                                       Geoprocessing.IGPParameter)
                    paramEdit = DirectCast(param, _
                                           Geoprocessing.IGPParameterEdit)
                    paramDataType = param.DataType
                    paramEdit.Value = paramDataType.CreateValue(value)

                    m_paramIndex += 1
                    Return True
                Else
                    'out of range...
                End If
            Finally
                ReleaseComObject(DirectCast(param, Object))
                ReleaseComObject(DirectCast(paramEdit, Object))
                ReleaseComObject(DirectCast(paramDataType, Object))
            End Try
        End Function
        Public Overridable Overloads Function SetParameter(ByVal index As Int32, _
                                               ByVal value As String) As Boolean

            Dim param As Geoprocessing.IGPParameter
            Dim paramEdit As Geoprocessing.IGPParameterEdit
            Dim paramDataType As Geodatabase.IGPDataType

            Try
                If index < m_tool.ParameterInfo.Count() Then

                    param = DirectCast(m_params.Element(index), _
                                       Geoprocessing.IGPParameter)
                    paramEdit = DirectCast(param, _
                                           Geoprocessing.IGPParameterEdit)
                    paramDataType = param.DataType
                    paramEdit.Value = paramDataType.CreateValue(value)

                    Return True
                Else
                    'out of range...
                End If
            Finally
                ReleaseComObject(DirectCast(param, Object))
                ReleaseComObject(DirectCast(paramEdit, Object))
                ReleaseComObject(DirectCast(paramDataType, Object))
            End Try
        End Function
        Public Overridable Function Execute(ByRef msg As String, _
                                Optional ByVal msgToIgnore As String = "", _
                                Optional ByVal msgDelimiter As String = ";", _
                                Optional ByVal useCancelTracker As Boolean = False, _
                                Optional ByVal errorCode As Integer = 0) _
                                As Boolean

            Dim gpMsgs As Geodatabase.IGPMessages
            Dim gpMsg As Geodatabase.IGPMessage

            Try
                gpMsgs = DirectCast(New Geodatabase.GPMessagesClass, _
                                    Geodatabase.IGPMessages)
                m_tool.Execute(m_params, Nothing, Nothing, gpMsgs)

                'get any resulting error from the messages collection.
                'this seems to provide the primary message of interest,
                'when it is an error.
                gpMsg = DirectCast(gpMsgs, Geodatabase.IGPMessage)
                If gpMsg.IsError Then
                    errorCode = gpMsg.ErrorCode
                End If

                'prepare to evaluate each message inside the collection.
                msg = ""
                Dim l As Int32
                For l = 0 To gpMsgs.Messages.Count() - 1
                    're-use the msg reference to get each individual message.
                    gpMsg = DirectCast(gpMsgs.Messages.Element(l), _
                                       Geodatabase.IGPMessage)
                    Dim str As String = gpMsg.Description
                    If Not (str Is Nothing) AndAlso str.Length > 0 AndAlso _
                        str.Trim() <> msgToIgnore Then
                        'the gp is reporting something other than the 
                        'message to ignore.
                        msg = msg & str & msgDelimiter
                    End If
                Next

                If msg.Length = 0 Then
                    Return True
                End If

            Finally
                ReleaseComObject(DirectCast(gpMsgs, Object))
                ReleaseComObject(DirectCast(gpMsg, Object))
            End Try
        End Function
#End Region

    End Class
 
  Bill Rose
rosewl@ci.richmond.va.us
City of Richmond, Virginia 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test; keywords: Toolbox clip tool .net 
Author Tom colson 
Date Mar 03, 2006 
Message amazing.... I used to be a software QA engineer. the most important industry standard (ISO) test we did was for memory leaks. It seems ESRI must be reading a different test manual 
 
Tom Colson
North Carolina State University
College of Natural Resources
Center For Earth Observation
Raleigh, North Carolina, 27695
(919) 673 8023
 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: GP memory leak test; keywords: Toolbox clip tool .net 
Author Luke Rogers 
Date Mar 14, 2006 
Message From ESRI...

This email is in re: your question about geoprocessing tools causing a memory leak when used in a loop. This is a known problem with ArcGIS and is a really hot issue with Development right know. Development is planning on having memory leaks caused by calling gp tools within loops fixed in 9.2. The bug tracking number for this is CQ00250716.

However, there is no work around until 9.2, other than increasing your RAM and limiting your calls to gp commands in loops. I.e. loop for 50 times, release the memory by shutting ArcGIS down, then loop more. 
  Luke Rogers
Research Scientist/Engineer
Rural Technology Initiative
University of Washington
http://www.ruraltech.org