File Formats

There are many different types of file formats for data storage when shooting images. Most digital cameras however, usually only work with three basic types - JPEG, TIFF and Raw.

Most astronomical image processing programs also work with another scientific image file format - FITS.

• JPEG (Joint Photographic Experts Group) - JPEG is an industry standard format that yields smaller file sizes by compression. It uses the .jpg file extension. Most digital cameras shoot JPEG by default. Unfortunately, to make smaller file sizes, the compression algorithm must throw away data. This is called "lossy" compression. They do a pretty good job of it, but data is lost nonetheless, and the file is only stored in 8 bits. JPEGs are ok for snapshots, but are not the best choice for astrophotography.

• TIFF (Tagged Image File Format) - TIFF is a standard file format for uncompressed images that is one of the most widely used and supported formats. It uses the .tif file extension. Because of the large size on an uncompressed TIFF file, only some DSLR cameras offer it as an option that must be specially selected in the camera set-up.

• Raw - The raw file format is almost always proprietary to the specific camera manufacturer. This means that these image files must be opened either in the manufacturer's dedicated software, or in an image processing program such as Images Plus or Photoshop that supports that particular flavor of raw file format. Most of these image processing programs do support the raw file formats by the major manufacturers such as Canon and Nikon, but it is a good idea to check to make sure before purchasing hardware or software. Raw files use a proprietary file extension such as .nef (Nikon Electronic image Format) or .cr2 (Canon Raw version 2).

• FITS (Flexible Image Transport System) - FITS files were developed for scientific images. It uses the .fit file extension. Many DSLR astronomical image processing programs will open up a raw file and translate it to FITS file format for processing, but DSLR cameras do not shoot or record in FITS file format in the camera.

Raw images can be opened in linear mode so the data can be better manipulated. Raw files also work in a higher bit-depth of 12 bits or 14 bits for more levels of tone. JPEG images are only 8 bits, which give only 256 steps of tone. This is important because of the large amount of manipulation that must be performed on the data to make an astronomical image look good.

For astronomical images, where data from recorded photons are precious and sometimes difficult to extract from the inherent noise present in all images, it is a really good idea to shoot raw file format. Raw is a lossless file format that preserves all of the data that comes out of the sensor as black and white information in its original high-bit depth and before it is de-Bayerized, turned into a color image and has tonal curves applied to it. This gives you much more flexibility and control later, and allows better calibration and data reduction because the operations can be done on the linear, high-bit-depth raw data before it is interpolated for constructing the color information and before tonal curves are applied. Some raw file formats may use lossless compression, but it may be a proprietary type.

Raw files can be opened to be worked on, but if the image is changed, it cannot be saved back to the original raw file format. The unchanged raw file can be closed however, and none of the data will be changed. A manipulated file must be saved as a TIFF or Photoshop file format to preserve all of the information in the file at a high-bit depth. If an astronomical image processing program is used, the image can be stored as a FITS file.

JPEG images have non-linear tonal curves applied to the original data that can't be removed later. JPEG files also will exhibit compression artifacts in varying amounts depending on the amount of compression used.

Some cameras offer the option to shoot and save raw and JPEG files concurrently. In this case, the normal tonal curves, white balance, sharpening and other in-camera selectable adjustments are applied to the JPEG file, but not to the raw data. This information is saved in the raw file's EXIF data though and is usually applied to the raw file as the default when the raw file is opened in the manufacturer's image processing program, or in Adobe Camera Raw converter in Photoshop. These settings in the raw file are not permanently applied, they only come up as the defaults, so they can be changed. For example, the white balance may be shot as daylight in the camera, and come up as the default in Canon's Digital Photo Professional, but it can be changed to another white balance setting, or a custom white balance can be set by clicking on a neutral area in the image.

Camera settings such as contrast, sharpening, white balance, etc. are applied to the JPEG file when the image is processed in the camera and before the file is written, so they are permanent.

Sometimes it can be convenient to shoot both JPEGs and Raws concurrently. This allows the JPEG to be opened quickly on a computer so that focus and tracking can be checked. Raw files are usually linear 16-bit files. They can also be opened, but some processing must be done before they resemble a "normal" image. This processing can be relatively simple, but it does take a little bit longer and is a little bit less convenient than just taking a quick peak at a JPEG.

Data Storage in the Field

Shooting raw files in megapixel cameras can eat up large amounts of storage space very quickly, especially when the support frames are added to the total.

Make sure you have a large enough memory card in the camera, or enough hard drive space if you are downloading images to a computer, to hold all of the images in an automated session. Make sure you have enough memory cards for an entire night of astrophotography if you plan to shoot without a computer. If you only have one card with limited capacity, you may need to periodically download the files from the card to the computer if the card is not large enough.

If downloading images to a computer, and if you have enough memory cards for the camera, also try to save the images to the card in the camera simultaneously (if it's big enough). That way you have the luxury of having an immediate backup in case something awful happens.

Downloading

Images shot in the camera are usually saved as files to a removable memory device, such as a compact flash card or secure digital card. These cards come in various different sizes and hold anywhere from 16 Megabytes to 64 Gigabytes of data. Memory cards will undoubtedly continue to get larger in the future, but while larger cards can be convenient, they cost more, and put more data at risk if something goes wrong with them.

Most DSLR cameras also allow the camera to be hooked up to a computer and then controlled through the manufacturer's, or third-party, software. When they are connected to a computer, sometimes the image files can be written directly to the computer's hard drive, bypassing the camera's memory card, such as when the camera is controlled by Images Plus, BackYard EOS, AstroPhotography Tool or Nebulosity.

With either storage method, care should be taken to have enough memory space available for the total number of images that will be taken, including support frames. This is especially important if the image acquisition is automated.

In the latest Canon DSLR cameras, the JPEG Files can be anywhere from 4-8 megabytes and the raw files can be 25 megabytes, depending on the content of the image. This means that you might need as much as 34mb total, per image, if you shoot raw and JPEG concurrently. These size files can eat up a memory card quickly if you are shooting light frames as well as a complete set of support frames including bias, flats and darks.

When the camera is connected to the computer, with most operating systems, you should be able to download the raw files directly from the camera through the operating system's file manager. The camera's memory card should mount just like an external removable storage disk.

Older cameras, such as the Canon 10D, connect to the computer with a USB1 cable. This connection can be quite slow, taking a long time to download a large amount of data. If image files are downloaded to a computer from a camera, it is important to make sure that the camera's battery has enough power to last this entire operation, or data may be lost or corrupted if the battery dies in the middle of a file-transfer operation.

A better method of data transfer is to use a separate memory card reader. If you have an older generation camera that uses only USB1, you can still use a USB2 compact flash card reader if your computer supports it, and file transfers will be about 10 times faster.

When using a compact flash or secure digital card reader, the memory card will be visible as a removable disk directly through the latest generation of operating systems.

EXIF Header Info

Technical data and information such as the date and time (if correctly set in the camera), exposure, ISO, white balance, and other custom function settings is usually recorded in the EXIF header info that is included with the file. This information can be accessed in Photoshop via File Info or with very nice little freeware programs such as EXIF Reader by Ryuuji Yoshimoto, or EXIFLOG by Michael Covington.

If the camera is used with a system lens, data for the focal length and f/number is also recorded. If the camera is used on a telescope, there is no computer chip in the scope to relay this info to the camera, so this information can not be written into the IPTC header. Other standard camera information, such as exposure time and ISO, are still written to the file however.

The latest generations of Canon cameras also records the camera's internal temperature in the EXIF header information. This can help match the temperature of dark frames to light frames in the calibration process.

Archiving

Original data files should be archived as soon as possible after they are shot. I usually copy the contents of the memory card to my computer's hard drive. I copy them by right clicking and dragging them to a new folder on the hard rive, rather than selecting "move" as the operation. That way, the files stay on the original memory card, and they are copied to the hard drive. This gives me an immediate, short-term back up.

Then, before I erase and re-format the memory card, I burn a DVD with the original Raw and JPEG files on it from that night's imaging session. I then burn a second DVD, on a different brand of DVD blank. For both different brands of blank DVD media, I try to buy the best that I can find, such as Mitsui, Verbatim, or TDK. There is a significant difference in longevity and dependability between the cheapest media and more expensive media.

Only after I have verified that the DVDs are good by opening up several different files at random, do I re-format the memory card. It is a good idea to always re-format the card in the camera. Do not just move the files or erase them. Re-format the card in the camera.

Despite claims by manufacturers of hundred-year archival stability of CD-R and DVD media, I am skeptical and cautious. I use two different brands of DVD media in case one might be a bad batch that might not show up immediately. It is generally accepted that CD-R is a much more dependable media than DVD+ or -R. Because so much more data can be written to a DVD disk, it is even more important to use a good brand of blank media, such as Tayio-Uden, TDK, Sony or Verbatim.

I also store a copy of all of my original image files on the hard drive for processing, and I make a backup of this entire archive to a removable hard drive.

This hard drive backup is archived at a completely different physical location than either of the DVD backups. One DVD copy is stored in my office with the computer, and the other is stored in a safety deposit box. Call me paranoid, but I had a friend who's house almost burned down and he had only one copy of all of his original digital files on his computer in his house. I had another friend whose house was hit by lightning and his computer's hard drive fried.

If original files are stored on a hard drive for convenience when working on them, it is a good idea to change the file permissions and make them write-only so the original file cannot be accidentally over-written and lost. In the file manager, right click on a file, or shift click a number of multiple files to select them, and then right click and select "Properties". Then, under the General tab, check on the box that says Attributes: Read Only, and click Apply.

When processing an image, always work on a copy so the original remains virgin and untouched.

Different programs, like Images Plus, may by default create different file formats for images in the data reduction and calibration stages. Some of these formats, like FITS, are not natively supported in Photoshop. This means to open a FITS image file you will need to get some type of plug-in for Photoshop that will read a FITS file, like FITS Liberator. These programs can usually also save files in the TIFF file format, which is almost universally supported by all imaging programs.

If your original file is in the JPEG format, it should be archived separately just like a raw original, as well as write-protected. Again, a copy should be worked on when processing or enhancing the image. It is important that once the original JPEG image is opened, it should be saved as a non-lossy, non-compressed image file format such as a TIFF for any subsequent image processing. This is because you will lose data by opening up a JPEG file and re-saving it as a JPEG. Any time a compressed file format is re-compressed, additional data is lost.