||Thanks Dan--I didn't know about that one. However, it might or might not obtain the correct answers, so let me explain what the procedure is, so that anybody reading this can use the extension intelligently and correctly. (You might also choose to do the procedure yourself, as shown below: it's not hard.)
The trick is to modify the elevations of the DEM, then carry out the visibility computation on the adjusted DEM. In effect, this causes the DEM's values to be misinterpreted as heights relative to a fixed plane rather than heights relative to the surface of the spheroid. So long as the DEM does not cover too much of the earth, these heights are about the same. However, once the DEM covers more than a few degrees, the distortions can get pretty great and the trick produces unreliable results. Fortunately, it makes no sense to compute visibility for DEMs that are larger than this, so usually we're ok.
Due to inaccuracies that will become evident below, it suffices to adopt a spherical earth model with a sphere of radius about R = 6371 km. Select a point of origin "O". Choose a reference plane parallel to the sphere's surface at O, passing through the center of the earth. The height of the sphere at O therefore is R. By definition of the cosine, the apparent height of any point on the sphere at a distance 'd' from O is R times the cosine of d, provided we measure d as an angle.
Often we don't bother to compute the cosine but instead use its Maclaurin series approximation,
cos(d) = 1 - d^2/2 + ...
for small values of d (expressed in radians). This is the approach taken by the extension. The error that is made is no greater than d^4/24. For a visibility calculation made out to, say, 200 miles, d would be about three degrees or around 0.05 radians. Here, d^4/24 = 0.05^4/24 = about 0.26 parts per million. Multiplying by the earth's radius gives an error no greater than 1.6 meters, which should be just fine. This justifies the approximation.
There are two complications. One is to convert radians to your preferred units of distance and elevation. The other is to account for atmospheric refraction effects, which cause light rays to curve.
Let's deal with the distance units. Subtract the apparent height, R*cos(d), from the DEM's height, R + DEM value = R + z, say. This gives the adjusted height of R + z - R*cos(d), which--using the approximation--simplifies to
Adjusted height z' = z - R*d^2/2.
If, for instance, you are measuring heights in feet, then you *must* measure distances in feet as well. By definition, one radian is the earth radius R. In feet it's 20.91 million. Divide by this value to convert d from radians to feet.
In general, if you express the earth radius R in your elevation units (such as feet or meters), and you measure distances in the same units, then the adjustment formula is
z' = z - d^2 / (2*R).
The value used in the extension for 2*R is 12740000. I presume that's meters. It corresponds to an assumed earth radius of 6370 km, which is ok: but you need to know that ***for the extension to work, your distance units and your elevation units MUST be in meters***. (It makes no effort to identify the units or correct for non-metric units.)
To give you a sense of the magnitude of this adjustment, it equals 7.848 centimeters in one kilometer--not worth worrying about. It increases quadratically with distance though, so at ten kilometers it's 7.848 meters and at 100 km it's 784.8 meters. (See equation 9a at http://geomechanics.geol.pdx.edu/Courses/G485/TotalStation/JGE%20Paper%201%20-%20Introduction.pdf ).
Atmospheric refraction is essentially in the vertical direction. Changes in air density curve the light downward. (Light passes faster through the thinner air at the top, causing its waves to curl down.) This causes you to see further; equivalently, the earth appears less curved than it really is. The magnitude of the effect varies with atmospheric conditions. A standard rule of thumb is that refraction of visible light reduces the apparent curvature of the earth by one-seventh. We therefore subtract only six-sevenths of the amount suggested by the geometric adjustment. This gives a combined curvature-refraction adjustment of the form
z' = z - (6/7) * d^2 / (2*R).
The extension uses a value of 0.87 for 6/7, which may be close enough: 6/7 = 0.857. The link provided above gives a value of 0.86 (this is equivalent to its equation 9c, since 0.0675/0.07848 = 0.860). Take your pick. If you're doing visibility calculations for radio transmission or other non-visible wavelengths, you will need to choose a different correction factor.
These considerations lead to a relatively simple Map Calculator procedure. Here's the recipe:
(1) Make sure your elevation units and distance units are identical. Make them so by converting the elevation units in the Map Calculator, if necessary.
(2) Compute the radius of the earth, 'R', in terms of your elevation units.
(3) Choose a value for the refraction correction, 'k'. Use a value around 6/7 for visible light.
(4) Compute the Euclidean Distance grid relative to any point near the center of your DEM. Call this grid theme "d".
(5) Let "z" be the name of the DEM theme. In the Map Calculator, compute an adjusted DEM as
[z] - ([d]^2 * (6/7 / (2*R))).
If you want to neglect all refraction effects, then leave out the "6/7" term.
(6) Perform all visibility calculations on the adjusted DEM.
I actually prefer to go though these steps each time, rather than apply some "black box" extension, because I can be confident in obtaining the correct answer and they don't really take any more time to execute.