6.2. Creating an image mask¶
Calibration cannot compensate for every defect in a CCD. Some examples (a non-exhaustive list):
Some hot pixels are not actually linear with exposure time.
Some pixels in the CCD may respond less to light than others in a way that flat frames cannot compensate for.
There may be defects in all or part of a row or column of the chip.
Cosmic rays strike the CCD during every exposure. While those are eliminated in the combined calibrated frames with the proper choice of combination parameters, they are not removed from science images.
The first three points are discussed in this notebook. Removal of cosmic rays from science images is discussed in the cosmic ray notebook.
from pathlib import Path
import numpy as np
import matplotlib.pyplot as plt
from astropy import units as u
from astropy.nddata import CCDData
import ccdproc as ccdp
from photutils import detect_sources
from convenience_functions import show_image, image_snippet
WARNING: AstropyDeprecationWarning: block_reduce was moved to the astropy.nddata.blocks module. Please update your import statement. [astropy.nddata.utils]
# Use custom style for larger fonts and figures
plt.style.use('guide.mplstyle')
6.2.1. Detecting bad pixels with ccdmask¶
The ccdproc function ccdmask uses a method that is
based on the IRAF task ccdmask. The method works best when the
input image used to detect flaws in the CCD is the ratio of two flat frames with
different counts. That may or may not be available depending on what images are
collected.
In the example below, which uses images from Example 2 in the reduction notebooks, the two extreme exposure times are 1 sec and 1.2 sec, but the average counts in the images differ by 10,000. These were twilight flats taken just after sunset.
Even with dome flats where the illumination is supposed to be constant, the counts may actually vary. If they do not, use a single flat for identifying bad pixels instead of a ratio.
We begin by creating an image collection and then the information for all of the calibrated, uncombined, flat images.
ex2_path = Path('example2-reduced')
ifc = ccdp.ImageFileCollection(ex2_path)
for long_values in ['history', 'comment']:
try:
ifc.summary.remove_column(long_values)
except KeyError:
# These two columns were not present, so removing them failed.
# Just keep going.
pass
flats = (ifc.summary['imagetyp'] == 'FLAT') & (ifc.summary['combined'] != True)
ifc.summary[flats]
Table masked=True length=10
| file | simple | bitpix | naxis | naxis1 | naxis2 | extend | date-obs | exptime | exposure | set-temp | ccd-temp | xpixsz | ypixsz | xbinning | ybinning | xorgsubf | yorgsubf | readoutm | filter | imagetyp | focallen | aptdia | aptarea | swcreate | swserial | sitelat | sitelong | telescop | instrume | notes | flipstat | swowner | date | time-obs | ut | timesys | radecsys | purged | latitude | longitud | altitude | lst | jd-obs | mjd-obs | biassec | trimsec | bunit | trim_image | trimim | subtract_dark | subdark | cstretch | cblack | cwhite | pedestal | bscale | bzero | subtract_bias | subbias | combined | hjd-obs | bjd-obs | azimuth | ha | readmode | st | lat-obs | long-obs | alt-obs | observat | ra | objctra | dec | objctdec | fwhm | zmag | _quinox | epoch | pa | _type1 | _rval1 | _rpix1 | _delt1 | _rota1 | _type2 | _rval2 | _rpix2 | _delt2 | _rota2 | _d1_1 | _d1_2 | _d2_1 | _d2_2 | tr1_0 | tr1_1 | tr1_2 | tr1_3 | tr1_4 | tr1_5 | tr1_6 | tr1_7 | tr1_8 | tr1_9 | tr1_10 | tr1_11 | tr1_12 | tr1_13 | tr1_14 | tr2_0 | tr2_1 | tr2_2 | tr2_3 | tr2_4 | tr2_5 | tr2_6 | tr2_7 | tr2_8 | tr2_9 | tr2_10 | tr2_11 | tr2_12 | tr2_13 | tr2_14 | pltsolvd | airmass | secz | alt-obj | az-obj | object | _ate | flat_correct | flatcor | wcsaxes | crpix1 | crpix2 | pc1_1 | pc1_2 | pc2_1 | pc2_2 | cdelt1 | cdelt2 | cunit1 | cunit2 | ctype1 | ctype2 | crval1 | crval2 | lonpole | latpole | mjdref | radesys | equinox | a_order | a_0_2 | a_1_1 | a_2_0 | b_order | b_0_2 | b_1_1 | b_2_0 | ap_order | ap_0_0 | ap_0_1 | ap_0_2 | ap_1_0 | ap_1_1 | ap_2_0 | bp_order | bp_0_0 | bp_0_1 | bp_0_2 | bp_1_0 | bp_1_1 | bp_2_0 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str31 | bool | int64 | int64 | int64 | int64 | object | str19 | float64 | float64 | float64 | float64 | float64 | float64 | int64 | int64 | int64 | int64 | str10 | object | str5 | float64 | float64 | float64 | str34 | str32 | str13 | str14 | str39 | str9 | str1 | str1 | str10 | object | object | object | object | object | bool | str13 | str14 | float64 | str13 | float64 | float64 | str11 | str11 | str3 | str6 | str13 | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object | object |
| AutoFlat-PANoRot-r-Bin1-001.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:25:17 | 1.0 | 1.0 | -20.0 | -20.0225025 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:25:17 | 01:25:17 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:04:39.2507 | 2458353.559224537 | 58353.05922453704 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-002.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:25:43 | 1.0 | 1.0 | -20.0 | -20.041362000000003 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:25:43 | 01:25:43 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:05:05.3219 | 2458353.559525463 | 58353.05952546297 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-003.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:26:08 | 1.0 | 1.0 | -20.0 | -20.041362000000003 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:26:08 | 01:26:08 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:05:30.3903 | 2458353.559814815 | 58353.05981481481 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-004.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:26:34 | 1.0 | 1.0 | -20.0 | -20.06965125 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:26:34 | 01:26:34 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:05:56.4615 | 2458353.560115741 | 58353.06011574074 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-005.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:27:00 | 1.0 | 1.0 | -20.0 | -20.0602215 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:27:00 | 01:27:00 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:06:22.5327 | 2458353.560416667 | 58353.06041666667 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-006.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:27:25 | 1.02 | 1.02 | -20.0 | -20.066508000000002 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:27:25 | 01:27:25 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:06:47.6012 | 2458353.560706018 | 58353.06070601852 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-007.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:27:51 | 1.06 | 1.06 | -20.0 | -20.053935000000003 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:27:51 | 01:27:51 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:07:13.6723 | 2458353.561006945 | 58353.06100694444 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-008.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:28:17 | 1.11 | 1.11 | -20.0 | -20.050791750000002 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:28:17 | 01:28:17 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:07:39.7435 | 2458353.56130787 | 58353.06130787037 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-009.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:28:42 | 1.16 | 1.16 | -20.0 | -20.0225025 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:28:42 | 01:28:42 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:08:04.8120 | 2458353.561597222 | 58353.06159722222 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | |||
| AutoFlat-PANoRot-r-Bin1-010.fit | True | -64 | 2 | 4096 | 4096 | True | 2018-08-23T01:29:08 | 1.21 | 1.21 | -20.0 | -20.013072750000003 | 9.0 | 9.0 | 1 | 1 | 0 | 0 | Monochrome | r | FLAT | 3200.0 | 400.0 | 125663.70964050293 | MaxIm DL Version 6.18 190601 00KPP | 00KPP-F3HN8-9JMTJ-1CE5W-A4AX5-F3 | +46:52:00.408 | -96:27:11.8008 | AspenCG16 | Matt Craig | 23/08/18 | 01:29:08 | 01:29:08 | UTC | FK5 | True | +46:52:00.408 | -96:27:11.8008 | 311.7999999995668 | 17:08:30.8832 | 2458353.561898148 | 58353.06189814815 | [4096:4109] | [1:4096, :] | adu | trimim | ccd=<CCDData> | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s, scale=True | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
The best we can do here is the ratio of the first and last of the flat images listed above.
first = ifc.summary['file'][flats][0]
last = ifc.summary['file'][flats][-1]
ccd1 = CCDData.read(ex2_path / first)
ccd2 = CCDData.read(ex2_path / last)
WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system
the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]
WARNING:astropy:FITSFixedWarning: RADECSYS= 'FK5 ' / Equatorial coordinate system
the RADECSYS keyword is deprecated, use RADESYSa.
ratio = ccd2.divide(ccd1)
The ratio is roughly 0.85:
ratio.data.mean()
0.8424684771223742
Running ccdmask takes a little time but only needs to be done once, not once
for each image.
%%time
maskr = ccdp.ccdmask(ratio)
CPU times: user 1min 51s, sys: 321 ms, total: 1min 51s
Wall time: 1min 51s
The result of ccdmask is one where there is a defect and zero where the chip
is good, which matches the format of the mask NumPy expects.
The input image and derived mask are shown below:
fig, axes = plt.subplots(1, 2, figsize=(20, 10))
show_image(ratio, cmap='gray', fig=fig, ax=axes[0], show_colorbar=False)
axes[0].set_title('Ratio of two flats')
show_image(maskr, cmap='gray', fig=fig, ax=axes[1], show_colorbar=False, percl=99.95)
axes[1].set_title('Derived mask')
WARNING: The following attributes were set on the data object, but will be ignored by the function: uncertainty, mask, unit [astropy.nddata.decorators]
WARNING:astropy:The following attributes were set on the data object, but will be ignored by the function: uncertainty, mask, unit
Text(0.5, 1.0, 'Derived mask')
Two comments are in order:
The “starfish” pattern in the first image is an artifact of the camera shutter. Ideally, a longer exposure time would be used for the flats to avoid this.
It appear at first glance that there were no pixels masked. The problem is that the masked regions are very small and, at the scale shown, happen to not be visible.
Two defects in this CCD are shown below. The first is a small patch of pixels
that are vastly less sensitive than the rest. The second is a column on the left
edge of the CCD. It turns out this column is not actually exposed to light.
ccdmask correctly identifies both patches as bad.
fig, axes = plt.subplots(2, 2, figsize=(10, 10))
width = 100
center = (3823, 2446)
plot_row = 0
image_snippet(ccd1, center, width=width, fig=fig, axis=axes[plot_row, 0])
axes[plot_row, 0].set_title('Flat, camera defect')
image_snippet(maskr, center, width=width, fig=fig, axis=axes[plot_row, 1], is_mask=True)
axes[plot_row, 1].set_title('Mask, same center')
center = (0, 2048)
plot_row = 1
image_snippet(ccd1, center, width=width, fig=fig, axis=axes[plot_row, 0], percu=99.9, percl=70)
axes[plot_row, 0].set_title('Flat, bad column')
image_snippet(maskr, center, width=width, fig=fig, axis=axes[plot_row, 1], is_mask=True)
axes[plot_row, 1].set_title('Mask, same center')
Text(0.5, 1.0, 'Mask, same center')
6.2.1.1. Saving the mask¶
The mask can be saved in a FITS file as an image. We will see in the summary notebook on masking how to combine the mask generated here with a mask generated from the dark current and with a cosmic ray mask for each science image.
mask_as_ccd = CCDData(data=maskr.astype('uint8'), unit=u.dimensionless_unscaled)
mask_as_ccd.header['imagetyp'] = 'flat mask'
mask_as_ccd.write(ex2_path / 'mask_from_ccdmask.fits')
6.2.2. Making the mask with a single flat¶
The flats we used in Example 1, taken with the Large Format Camera at Palomar,
are dome flats taken with nearly constant illumination. In that case the best we
can do is run ccdmask on a single flat image. As we will see, this still
allows the identification of several clearly bad areas of the chip.
First, a look at the calibratted, but not combined, flat images.
ex1_path = Path('example1-reduced')
ifc1 = ccdp.ImageFileCollection(ex1_path)
flats = (ifc1.summary['imagetyp'] == 'FLATFIELD') & (ifc1.summary['combined'] != True)
ifc1.summary[flats]
Table masked=True length=6
| file | simple | bitpix | naxis | naxis1 | naxis2 | date | origin | latitude | longitud | telescop | fratio | instrume | detector | frame | ccdpicno | object | imagetyp | exptime | darktime | date-obs | ut | jd | ra | dec | equinox | epoch | ha | st | airmass | filter | gain | secpix1 | secpix2 | ccdbin1 | ccdbin2 | rotangle | datasec | ccdsec | biassec | loginfo | chipid | subtract_overscan | suboscan | trim_image | trimim | bunit | extend | subtract_dark | subdark | flat_correct | flatcor | combined |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str27 | bool | int64 | int64 | int64 | int64 | str10 | str34 | float64 | float64 | str17 | float64 | str3 | str14 | int64 | int64 | str7 | str9 | float64 | float64 | str10 | str12 | float64 | str12 | str12 | float64 | float64 | str12 | str12 | float64 | str2 | float64 | float64 | float64 | int64 | int64 | float64 | str15 | str15 | str18 | str18 | int64 | str8 | str46 | str6 | str13 | str3 | object | object | object | object | object | object |
| flat-ccd.014.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 14 | 14 | flat_g | FLATFIELD | 70.001 | 70.673 | 2016-01-16 | 04:17:28.00 | 2457403.678796 | 04:08:58.50 | 33:25:12.69 | 2000.0 | 2000.0 | 00:00:04.00 | 04:10:04.46 | 1.0 | g' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
| flat-ccd.015.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 15 | 15 | flat_g | FLATFIELD | 70.011 | 70.683 | 2016-01-16 | 04:20:35.00 | 2457403.680961 | 04:12:05.19 | 33:25:16.79 | 2000.0 | 2000.0 | 00:00:04.00 | 04:13:11.98 | 1.0 | g' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
| flat-ccd.016.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 16 | 16 | flat_g | FLATFIELD | 70.001 | 70.684 | 2016-01-16 | 04:23:41.00 | 2457403.683113 | 04:15:12.00 | 33:25:20.79 | 2000.0 | 2000.0 | 00:00:04.00 | 04:16:18.49 | 1.0 | g' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
| flat-ccd.017.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 17 | 17 | flat_i | FLATFIELD | 7.0 | 7.683 | 2016-01-16 | 04:27:23.00 | 2457403.685683 | 04:18:54.58 | 33:25:25.70 | 2000.0 | 2000.0 | 00:00:04.00 | 04:20:01.09 | 1.0 | i' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
| flat-ccd.018.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 18 | 18 | flat_i | FLATFIELD | 7.0 | 7.672 | 2016-01-16 | 04:29:26.00 | 2457403.687106 | 04:20:58.15 | 33:25:28.39 | 2000.0 | 2000.0 | 00:00:04.00 | 04:22:04.43 | 1.0 | i' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
| flat-ccd.019.0.fits | True | -64 | 2 | 2048 | 4128 | 2016-01-16 | California Institute of Technology | 33.4 | -116.9 | Hale 5m Telescope | 7.5 | LFC | SITe SI002 x 6 | 19 | 19 | flat_i | FLATFIELD | 7.0 | 7.673 | 2016-01-16 | 04:31:30.00 | 2457403.688542 | 04:23:01.65 | 33:25:31.20 | 2000.0 | 2000.0 | 00:00:04.00 | 04:24:08.77 | 1.0 | i' | 1.1 | 0.182 | 0.182 | 1 | 1 | 0.0 | [1:2048,1:4128] | [1:2048,1:4128] | [2049:2080,1:4127] | Status=0x00000000 | 0 | suboscan | ccd=<CCDData>, overscan=<CCDData>, median=True | trimim | ccd=<CCDData> | adu | True | subdark | ccd=<CCDData>, master=<CCDData>, exposure_time=exptime, exposure_unit=s | -- | -- | -- |
We can double check that a ratio of flats will not be useful by calculating the mean counts in each flat image:
ccs = []
for c in ifc1.ccds(imagetyp='flatfield', filter="g'"):
if 'combined' in c.header:
continue
print(c.data.mean())
ccs.append(c)
WARNING: FITSFixedWarning: 'datfix' made the change 'Set MJD-OBS to 57403.000000 from DATE-OBS'. [astropy.wcs.wcs]
WARNING:astropy:FITSFixedWarning: 'datfix' made the change 'Set MJD-OBS to 57403.000000 from DATE-OBS'.
19906.38951623411
19902.514437929316
19890.652615963663
The variation in counts is so small that the ratio of two flats will not be useful.
Instead, we run ccdmask on the first flat. There is nothing special about that
one. The kind of defects that ccdmask tries to identify are in the CCD sensor
itself and should be the same for all filters.
%%time
ccs1_mask = ccdp.ccdmask(ccs[0])
CPU times: user 55.9 s, sys: 128 ms, total: 56.1 s
Wall time: 56.1 s
Displaying the flat we used and the mask side by side demonstrates that the defects which are clear in the flat are picked up in the mask.
fig, axes = plt.subplots(1, 2, figsize=(15, 10))
show_image(ccs[0], cmap='gray', fig=fig, ax=axes[0])
axes[0].set_title('Single calibrated flat')
show_image(ccs1_mask, cmap='gray', fig=fig, ax=axes[1], is_mask=False)
axes[1].set_title('Derived mask');
WARNING: The following attributes were set on the data object, but will be ignored by the function: uncertainty, mask, meta, unit [astropy.nddata.decorators]
WARNING:astropy:The following attributes were set on the data object, but will be ignored by the function: uncertainty, mask, meta, unit
A couple of cutouts are shown below illustrating some of the individual defects identified.
fig, axes = plt.subplots(2, 2, figsize=(10, 10))
width = 300
center = (512, 3976)
plot_row = 0
image_snippet(ccs[0], center, width=width, fig=fig, axis=axes[plot_row, 0])
axes[plot_row, 0].set_title('Flat, partial bad column')
image_snippet(ccs1_mask, center, width=width, fig=fig, axis=axes[plot_row, 1], is_mask=True)
axes[plot_row, 1].set_title('Mask, same center')
center = (420, 3250)
width = 100
plot_row = 1
image_snippet(ccs[0], center, width=width, fig=fig, axis=axes[plot_row, 0])
axes[plot_row, 0].set_title('Flat, bad patch')
image_snippet(ccs1_mask, center, width=width, fig=fig, axis=axes[plot_row, 1], is_mask=True)
axes[plot_row, 1].set_title('Mask, same center')
Text(0.5, 1.0, 'Mask, same center')
