Shapefile Null Mask For ArcGIS

You are here: > ESRI Forums > arcgis desktop discussion forums > Thread Replies

ArcGIS Desktop Discussion Forums

ArcGIS Desktop - ArcObjects General forum

Shapefile Null Mask For ArcGIS	Ruth Bowers	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Neil Clemmons	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Neil Clemmons	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Lance Shipman	May 06, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 07, 2004
Re: Shapefile Null Mask For ArcGIS	Lance Shipman	May 07, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 07, 2004
Re: Shapefile Null Mask For ArcGIS	Lance Shipman	May 07, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 10, 2004
Re: Shapefile Null Mask For ArcGIS	Lance Shipman	May 10, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	Jun 16, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	May 07, 2004
Re: Shapefile Null Mask For ArcGIS	Roberto Cortinas	Jun 16, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	Jun 16, 2004
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	Jun 23, 2004
Re: Shapefile Null Mask For ArcGIS	Russell Egypt	Feb 10, 2005
Re: Shapefile Null Mask For ArcGIS	Ruth Bowers	Feb 10, 2005

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 06, 2004
Message	Hi everybody! My question is whether anybody has developed a method of emulating NULL values in shapefiles similar to what is available in ArcView 3.2 and Excel? I develop software for which the output is shapefiles and the data includes NULL values, and I would really like to figure this out. I'd like to be able to apply it in the ArcMap environment. Thanks in advance! Ruth

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Neil Clemmons
Date	May 06, 2004
Message	Shapefiles cannot contain null values due to the limitation imposed by the DBF file format. ArcView GIS and ArcGIS circumvent this limitation by inserting a space into text fields instead of NULL and (I believe) 0 into numeric fields. Your code will need to do the same.
	Neil Clemmons Senior GIS Developer Geographic Information Services, Inc. Birmingham, AL http://www.gis-services.com Check out our blog: http://blog.gis-services.com/

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 06, 2004
Message	Thanks for your prompt reply. I think there is more to it than that. If they were simply inserting a "0" into number fields to represent null values, then there would be no way to tell the difference between a null value and a zero. But both Excel and ArcView 3.2 can somehow do this, so I think they must be forcing in a special character somehow that they both are programmed to look for as null values. So, I'm still looking...

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Neil Clemmons
Date	May 06, 2004
Message	OK, I just created a new shapefile in ArcCatalog with one numeric field and one text field. I loaded it into ArcMap and used the Editor to add a new feature. I then wrote a macro to show me the values of the two fields for the new feature. The numeric field was given the value 0 and the text field was given the value ' ' (single space). Neither the Attribute table window or the Editor Attribute Editor recognize these values as NULL, so they are real live values, not special codes. Once the values are written, there's no way to tell if the field is 0 because that's the actual correct value or if it represents lack of data. Same for text fields containing a single space.
	Neil Clemmons Senior GIS Developer Geographic Information Services, Inc. Birmingham, AL http://www.gis-services.com Check out our blog: http://blog.gis-services.com/

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 06, 2004
Message	I understand that. But, if ArcMap is not programmed to see or to understand the special character that Excel or AV3.2 are inserting, this doesn't mean it isn't there, right? Somehow Excel and AV3.2 are looking at these same fields and seeing a NULL there instead of a 0. I'll let you know what I find out. Ruth Bowers

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Lance Shipman
Date	May 06, 2004
Message	ArcGIS does not support NULLs in dBase files. The dBase file format does not, so ArcGIS does not.
	Lance Shipman ESRI Product Development Geodatabase Product Engineer - File Geodatabase

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 07, 2004
Message	It's really not that simple. I assume that you have never used ArcView 3.2? For 3.2 ESRI developed a mask that emulated NULL values in dBASE tables. They are not true NULL values, but software can be programmed to use them as such and to let users see them as NULL values. Excel has a similar capability regarding dBASE files. This is the technology I'm trying to find out about. Because Excel and AV3.2 can read each others "NULL" values, I think they are using the same technology, but I can't find anything on the Microsoft site so far (I'm also talking to ESRI). Like I said above, I'll post what I find out when I get to the bottom of this. Ruth Bowers

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Lance Shipman
Date	May 07, 2004
Message	I have used ArcView and am familure with how it handles NULL values. In any case ArcGIS (8.x, 9.x) does NOT handle NULLs in dBase files.
	Lance Shipman ESRI Product Development Geodatabase Product Engineer - File Geodatabase

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 07, 2004
Message	Lance, I realize, of course, that ArcGIS does not handle NULL values in shapefiles out of the box. But perhaps an extension could be developed that could provide this functionality. When you say you are familiar with how AV3.2 handles NULL values, do you mean in a programming or software architecture sense? If so, can you be more specific? I must not be explaining this very well because people are having a hard time understanding what I am trying to do. Anyway, thanks! Ruth Bowers

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Lance Shipman
Date	May 07, 2004
Message	I'm a ESRI Product Specialist on the Geodatabase Team and have worked with ESRI Software for 17 years. I've supported data loading and data issues over most of that time. I'm VERY failure with issues involving NULLs in ESRI software both from a user and programming view point. The dBase specification does not include NULLs (there are two exceptions, Logical and Date types support a NULL, ? and 0 respectively). The lack of support for NULLs in ArcGIS (and in ArcInfo) is a design decision, not a technical one. If you feel that supporting NULLs in shapefiles is needed, feel free to put in an enhancement request. ArcGIS supports nulls in the Geodatabase. Not in Coverages or in Shapefiles.
	Lance Shipman ESRI Product Development Geodatabase Product Engineer - File Geodatabase

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 10, 2004
Message	Lance, I'm not doubting your expertise. I think I'm miscommunicating somehow. When I'm talking about NULLs in AV3.2, I'm not talking about true NULL, rather the simulation of such. I'm talking about how you can have "blanks" in number columns in AV3.2, which are recognized as NULL by the software. Excel has the same capability with dBase files. If you look at the link that I posted in the post right below this one, you will see what I mean. I realize that dBase does not support NULL values. This is a workaround to that that some software manufacturers have found (which I am trying to find out about). ESRI technical support think that their white paper on shapefile architecture will answer this question, so I am about to go read through that again. I don't remember ever seeing this there, but maybe because I was looking for something else then. And, yes, I very much realize that this was a design decision. That is why I am having to write this software, after all. ESRI seems very clear that they have no interest in developing this technology for ArcGIS. Ruth Bowers

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Lance Shipman
Date	May 10, 2004
Message	What you want then is the format of the ArcView 3.x NULL so you can write a workaround? Yes? Please send me you e-mail and I'll see what I can find.
	Lance Shipman ESRI Product Development Geodatabase Product Engineer - File Geodatabase

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	Jun 16, 2004
Message	Lance helped me find the AV3.2 side of this problem. Thanks again, Lance! See the June 16 post for the information. Cheers! Ruth

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	May 07, 2004
Message	Here is the link to where ESRI briefly talks about this issue. They give a one-sentence explanation of the NULL characters in AV3.2 shapefiles in the last line. http://support.esri.com/index.cfm?fa=knowledgebase.techarticles.articleShow&d=20177 Ruth Bowers

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Roberto Cortinas
Date	Jun 16, 2004
Message	Ruth, I certainly hope that you can find a way to allow the inclusion of null values in ArcGIS attribute tables (even if they are emulated). I certainly wish that ESRI had not decided to equate null values with zero in numeric fields. Zero means you have data, null means unknown - a very important distinction in statistical analyses. Keep us informed in your progress!

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	Jun 16, 2004
Message	I am moving rather slowly on this, but I'll tell you what I have found out so far. dBASE files store number column values as ASCII characters on disk. This enables developers to create functions that write non-number values to number columns in dBASE tables. Then, same developer can also write it into his software to interpret these non-number values in a number column as "NULL" values. MS Excel interprets non-number values in a number column as "blank", and AV3.2 interprets non-number values as "NULL". This is why dbf files pass back and forth between Excel and AV3.2 so nicely. AV3.2 inserts astericks as NULL numbers, and VB6 inserts spaces as NULL numbers. As a non-number character, spaces will work just fine. So, as you can see, writing NULL numbers to a dBASE file in ArcGIS is not a problem. The problem comes in the way the values are read. Instead of seeing non-number characters in a number field as NULL, they are ignored and given the default value of 0. I haven't been able to figure out exactly what's going on behind the scenes in ArcGIS that causes the non-number characters to be seen as 0's. It's possible it happens outside ArcGIS with a driver setting in Microsoft. I think it is possible that the ODBC driver or whatever is called to read dBASE files has a default value setting. I don't know yet though. Anyone, please let me know if you know the answer to this riddle! Cheers! Ruth

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	Jun 23, 2004
Message	I have finally gotten the answer to the ArcGIS side of this riddle. The following is a quote from an ESRI developer (identity unknown) that a support person was kind enough to consult with for me. I am happy to finally know what is going on here, although I guess this means that this is an integral part of ArcGIS behavior that cannot be changed. Here is what the ESRI developer said: "We do not use an ODBC driver or any other component to read dbase files. Rather, it is C++ code written by ESRI. However, we do rely on some operating system run time libraries to do some of the work. In particular, when converting the bytes representing a field value back in to a binary number, we call a C run time function "fscanf". Part of the behavior of fscanf is that it interprets an embedded blank to be the same as the digit '0'. We cannot change this behavior of the C run time library."

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Russell Egypt
Date	Feb 10, 2005
Message	Hi I would like to know concerning the above issue, is there a work around in displaying an empty field for number fields, meaning they don't default to 0 in shape files.

Report Inappropriate Content • Top • Print • This Forum is closed for replies.
Subject	Re: Shapefile Null Mask For ArcGIS
Author	Ruth Bowers
Date	Feb 10, 2005
Message	Not as far as I know. I have seen fields seem to remain blank sometimes if they are never "touched", but this behavior doesn't seem to be dependable.