Sail Away
Cover  <<  Sail Away  <<  Celestial Navigation  <<  .

Correction Tables for Sextant Altitudes

The Altitude-Intercept method now generally used for Celestial Navigation, is based on the comparison of an "observed altitude" (Ho) with a related "computed altitude" (Hc).
The computed altitude is based on a mathematical model implying a number of conditions and assumptions some of which are:

  • The celestial bodies have no physical dimensions and observations are made from the center of the Earth.
  • Light rays from celestial bodies come from infinitely far away and hence reach the Earth parallel to one another.
  • The Earth has no atmosphere and air has no index of refraction.

These assumptions are not full-filled for observations made in the physical world. In order to put the mathematically computed altitude and the "real world" observed altitude on an alike basis of comparison, some "corrections" must be applied to the measured sextant altitude.

These corrections are performed on the measured sextant altitude according to the following scheme:

  Hs     __ ° __'_                Sextant Altitude
  IE   ±      __'_                Index Error
  dip  -      __'_                Dip Correction
  Ha     __ ° __'_                Apparent Altitude
  Refr -      __'_                Refraction
         __ ° __'_                
  Prlx + __ ° __'_     HP __°_    Parallax Correction
  SD   ±      __'_     SD __'_    Semi-Diameter Correction     
       _____________              (lower limb:+ /upper limb:-)  
  Ho     __ ° __'_                Observed Altitude

The sextant altitude (Hs) corrected for index error (IE) and dip correction (dip) is called "apparent altitude" (Ha). This value is used as entry for further corrections (refraction (Refr) and parallax (Prlx) ).

The values for "semi-diameter" (SD) and "horizontal parallax" (HP) must be obtained from the Nautical Almanac. The required values for the correction of dip, refraction and parallax can be found in the pre-compiled correction pages (125 Kbyte PDF file), or they can be obtained from the interactive tables below.
The pre-compiled PDF file can be viewed and printed e.g. with the free tools "Evince", "xpdf" or "Acrobat Reader".


Correction Table for Dip

The "dip correction" compensates for the fact that observations made by a navigator through a sextant are not performed on the "level of the horizon". The difference between the visible horizon - the line where the sky meets the sea - and the geodial horizon - the plane perpendicular to the zenith line in the location of the observer - is called "Dip of the Horizon". This "dip" (of the horizon) is not only caused by the elevation of the observer's eye above the plane of the geoidal horizon (He) but also by atmospheric refraction.

The higher the observation point (sextant telescope) is above the sea level, the further "away" the visible horizon seems to wander. Due to this effect, the measured altitude values are too high. So "dip corrections" are always negative.
It is important to use the correct value for He and not just a coarse estimation. Using a He value that is 2m off, will result in a LoP that is one mile off!

The correction for dip and the distance to the horizon can also be calculated with the following formulas:

  Correction_for_Dip ['] = 1.760 ['/sqrt(m)] * sqrt (He [m])

  Distance_to_the_Horizon [Nm] = 2.075 [Nm/sqrt(m)] * sqrt(He [m])

These equations are empirical and include the effect of the curvation of the Earth as well as the effect of the atmospheric refraction on the dip of the horizon. A more detailed discussion on this can be found in the Notes on the Dip of the Horizon


Correction Table for Refraction

Refraction is an optical characteristic of the Earth's atmosphere. Light coming from outer space is deflected to the Earth's surface when it travels through the atmosphere. This is caused by the density gradient of the air in the atmosphere. Air closer to the Earth's surface is denser that in the higher atmospheric layers.

The result is that at sea level a celestial object is seen at a different angle as it would be seen at the same location without the presence of the Earth's atmosphere. Corrections for refraction are always negative and they primary depend on air temperature and on barometric pressure.

Since the atmospheric processes involved are very complex and air masses with different temperatures are continuously stirring the troposphere, it is very difficult to obtain good models describing refraction (especially for low values of altitude). For this reason, a "Line of Position" obtained from an Altitude lower than 15° can be considered to be rather inaccurate.

The largest corrections for refraction occur at high pressure and low temperature. Under these conditions the "air density" and thus refraction will obtain the highest value.

NOTICE: The barometric pressure used in these tables is the "real" atmospheric pressure at the location of observation, not the barometric pressure reduced to the sea level. Therefore, the pressure range of this table will not be sufficient to cover locations significantly above the sea level.

A simple empirical formula for calculating refraction from the apparent Altitude was developed by G.G. Bennett in 1982. The formula is used in the U.S. Naval Observatory's "Vector Astronomy Software" and is reported to be consistent with more complex algorithms within 0.07' over the full range of apparent Altitudes from horizon (0°) to Zenith (90°).

  Refr ['] = 1.0 ['] / tan(Ha [°] + 7.31 / (Ha [°] + 4.4) )

The formula is valid for so called atmospheric standard conditions for atmospheric pressure (1010 hPa) and air temperature (10°C), both measured at sea level. For conditions other than these, the value of 'Refr' should be be multiplied by the following (linear) correction factor:

  f = P [hPa] /(273+T [10°C] ) * 283/1010 

The resulting refraction value for arbitrary temperature and air pressure is: Refr(P,T) = f * Refr.

The predictions for refraction calculation are normally based on a theoretical standard atmosphere with an average atmospheric pressure distribution. The actual refraction may differ significantly from the calculated values if anomalous atmospheric conditions occur. The influence of these atmospheric anomalies may become large for low altitude values. Particularly refraction calculation for an apparent altitude below 5 degrees may be very unreliable. This also affects the sight of the visible horizon, which is affected in the same way by atmospheric refraction. Calculated or tabulated values in the low-altitude range should be used with particular care.


Tables for Parallax Correction

Parallax correction of apparent altitudes (Ha) is needed for celestial bodies which are "close" to the Earth. This applies to the Moon, which may have a parallax angle of more than 1°; but also the altitudes for the planets Venus and Mars may be corrected for parallax effects to obtain the best possible results.

The horizontal parallax only depends on the radius of the Earth (which is constant) and and the distance of the celestial object to the Earth. Since the distance of celestial objects to the Earth changes very gradually, the horizontal parallax does not change significantly during one day. Therefore, the Nautical Almanac records the "horizontal parallax" angle (HP) only for a for a small number of distinct hour values each day. The HP value corresponding to the hour nearest to the moment of observation can be used without interpolation.

The horizontal parallax angle (HP) and the "apparent altitude" corrected for Refraction and Semi-Diameter (SD) are used as entries for the Parallax Correction Tables.

A detailed presentation of the geocentric parallax and how it is taken into account while performing celestial observations, is summarized in the "Notes on the Calculation of Geocentric Parallax".


Tables for combined Parallax - Semi-Diameter Correction

For the Moon, both the corrections for Semi-Diameter and for Parallax are significant and must be considered in the process of reducing the measured topographic altitude at the surface of the Earth to the geographical altitude as would be measured at the center of the Earth.
Due to its' close proximity to the Earth, the Moon has a considerable larger Parallax effect than all other Celestial bodies (including the Sun). For this reason the Parallax Correction Tables have an extra section for the Moon. In order to simplify the correction scheme for Sextant measurements, the Moon section of the Parallax Tables could be compiled such that the Semi-Diameter correction is included. That means that these combined Moon Correction Tables will have to be compiled for upper-limb observations and lower-limb observations separately. This results in more pages in the tables for the convenience of a simpler correction scheme.
A more elaborate description of these combined Correction Tables is given in the "Notes on combined Parallax - Semi-Diameter Correction for the Moon"

Cover  <<  Sail Away  <<  Celestial Navigation  <<  . last updated: 08-Dec-2018