You are here: > ESRI Forums > arcgis desktop discussion forums > Thread Replies

ArcGIS Desktop Discussion Forums

ArcGIS Desktop - Extension - Spatial Analyst forum

"Unable to allocate memory"- points to kern...   Richard Thurau Aug 28, 2008
Re: "Unable to allocate memory"- points to...   William Huber Sep 03, 2008
Re: "Unable to allocate memory"- points to...   Richard Thurau Sep 03, 2008
Re: "Unable to allocate memory"- points to...   William Huber Sep 03, 2008
Re: "Unable to allocate memory"- points to...   Richard Thurau Sep 29, 2008
Re: "Unable to allocate memory"- points to...   William Huber Sep 29, 2008
Re: "Unable to allocate memory"- points to...   Richard Thurau Oct 14, 2008
Re: "Unable to allocate memory"- points to...   William Huber Oct 14, 2008
Re: "Unable to allocate memory"- points to...   Richard Thurau Oct 14, 2008
Re: "Unable to allocate memory"- points to...   William Huber Oct 14, 2008
Re: "Unable to allocate memory"- points to...   Richard Thurau Oct 15, 2008
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Aug 28, 2008 
Message Hi,
I'm trying to run kernel density on a point shapefile with 54 mil.+ points. The tool barely initializes before an error is returned:

An error was encountered while executing KernelDensity.
("esriGeoAnalyst.GridEngine") Unable to allocate memory
Error in executing grid expression
Failed to execute (Kernel Density)

I've bumped up the minimum amount of records in the raster attribute table to 100 mil. Separating the dataset into smaller blocks is going to mean some painful manual polygon creation.

Does anyone know the source of the "unable..." error? It seems the Grid engine won't even look at the shapefile.

And, what is the limit? I have many areas I'd like to run in a model. Is there a size I should aim for?

Thanks

Rich 
  Be aware of your environment. 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author William Huber 
Date Sep 03, 2008 
Message Rich,

What are the dimensions of the grid you are trying to create? 
  --Bill Huber
Quantitative Decisions (http://www.quantdec.com )
More GIS Q&A at http://gis.stackexchange.com/q/3083/664 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Sep 03, 2008 
Message William,
Thanks for the reply.
I'm not sure what the new raster dimensions will be, but here's what I do know:

I'm wanting to create the new grid, a kernel density, at 1M resolution. The studay area is a chunk of public land inside Perry County, IN. The study area mask is at 50M resolution so that won't give me the rows and columns, but the entire county is 3360x4610 rxc. The new grid will be probably about 2/3 that size.

I'm still hung up on this, so any input would be great.

Thanks again.

Rich 
  Be aware of your environment. 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author William Huber 
Date Sep 03, 2008 
Message Rich,

So, it looks like you have more input points than grid cells. Do you even need to compute a density? Just compute the cell coordinates for each point, summarize the point data by cell, and convert that directly to a grid.

A density calculation amounts to a smoothing of this grid. ArcGIS can't handle it mainly because it uses an algorithm designed for a relatively small number of input points, not an entire grid of them. There is another algorithm though, using Fast Fourier Transforms, that will do the job and--amazingly--it will be faster in processing your 54 million points than Spatial Analyst will in processing just a few hundred points. (You will need a few seconds of CPU time, a few seconds of disk I/O, and about 100-200 MB RAM.) That's the good news. The bad news is that you have to use other software. Idrisi will do it and so, with some effort, will GRASS. Just about any decent image-processing software will also do it, albeit at some loss in the resolution of the grid values. 
  --Bill Huber
Quantitative Decisions (http://www.quantdec.com )
More GIS Q&A at http://gis.stackexchange.com/q/3083/664 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Sep 29, 2008 
Message William,
If you're still out there, thanks for your insight. I'm juggling many projects, and I'm back on this one for a while here.

"Any image processor"? Can Erdas Imagine perform such a calculation? I see the Fourier tools...any guidance?

Regardless of the math I'v posted, the points should be at no greater density than one point per meter, and that is the desired output raster resoultion as well. An option I've considered is to create a lower resolution grid, and aggregate points to assign my own density. But, of course, I'd like to maintain the 1m resolution.

Thanks for your time. I'd still like to know what the "allowable" file size is for Arc.

Rich 
  Be aware of your environment. 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author William Huber 
Date Sep 29, 2008 
Message Rich,

A kernel density is computed as the convolution of a kernel function with a grid that represents the original points. I explain convolution, as simply as I can, in a three-part article on the Web at

http://www.directionsmag.com/features.php?feature_id=19

http://www.directionsmag.com/features.php?feature_id=24

http://www.directionsmag.com/features.php?feature_id=26

The discussion of the Fourier analysis approach begins midway through the second part. 
  --Bill Huber
Quantitative Decisions (http://www.quantdec.com )
More GIS Q&A at http://gis.stackexchange.com/q/3083/664 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Oct 14, 2008 
Message Bill,
Thanks for the information. I'm learning a great deal thanks to you.

Unfortunately, for this project, the team I'm working with is grounded in ESRI Modelbuilder.

The best solution is going to be for me to extract the points by chunks small enough to process. Of course, the bigger the chunks the easier it will be to complete the model.

My question, then, is what is the max memory, or max number of points that Arc will process? Eventually, I'll have my own answer, but if anyone else knows, it can save me some time.

Thanks again for the help.

Rich 
  Be aware of your environment. 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author William Huber 
Date Oct 14, 2008 
Message Rich,

The memory problem almost surely occurs in the conversion of points to a grid. At most, the kernel density calculation itself will need 8 * NRows * NColumns bytes of RAM (that is, one single-point float for each cell of input and one for each cell of output).

Therefore, if you possibly can, take the point-to-grid conversion out of ModelBuilder. If you can't, you're going to have to experiment with tiling your points, converting them, and mosaicing the resulting grids. I don't know what the maximum number of points is because I avoid massive vector-raster conversions as much as possible in my own work. 
  --Bill Huber
Quantitative Decisions (http://www.quantdec.com )
More GIS Q&A at http://gis.stackexchange.com/q/3083/664 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Oct 14, 2008 
Message Bill,
Alright, after reading your articles, I think I should be able to do what I want to do (density) without going to points. I'm trying to develop a moving window in ERDAS Imagine.

I have a reclassified image with 1's and 0's. I want the 1's to have a discrete 20 meter kernel. It seems to me, the values in the moving window don't really matter. Where more cells with 1's are within the moving window, the output cell will have a greater value.

So, I made a 20x20 moving window with a value of 1/400 (0.0025) in each cell. All this is under the "convolution" tools in Imagine. But, when I run the convolution on the raster of 1's and 0's, my output raster is all 0's.

Have you used Imagine? Know what I'm talking about?

I really appreciate your time.

Rich 
  Be aware of your environment. 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author William Huber 
Date Oct 14, 2008 
Message I haven't used Imagine, Rich, but I wonder what would happen if you placed all ones in your kernel.

BTW, the values in the kernel do matter, as you will quickly see once you get this working. However, when you rescale them all by a constant, the output will be rescaled by the same constant.

Note, too, that some software is picky about how you define your convolution kernel: you might have to put the values dead center, or you might have to scatter them around the four corners. Reading the documentation is useful in such instances ;-). 
  --Bill Huber
Quantitative Decisions (http://www.quantdec.com )
More GIS Q&A at http://gis.stackexchange.com/q/3083/664 
   
Report Inappropriate Content • Top • Print • This Forum is closed for replies.    
Subject Re: "Unable to allocate memory"- points to kernel density 
Author Richard Thurau 
Date Oct 15, 2008 
Message Bill,
I'm pretty damn excited right now. My 10x10 coeficient of 1 kernel works great. I've gone from trying to process 50 million points (from not being able to process those) to using convolution to calculate density in about 8 minutes.

Thanks so much for your help. Your articles really hammered home some concepts that I sort of knew but didn't quite understand completely. Next step will be to figure out the fourier space concept, but maybe next project. I only have 8 grids to process once, so it's not necessary here.

If you're ever near Bloomington, IN, or maybe the ESRI conference next year, I'd like to buy you a beer or something.

All the best.

Rich 
  Be aware of your environment.