FITS file doubles in size after preprocessing

Image processing, astrometry, photometry, etc.
Post Reply
User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

FITS file doubles in size after preprocessing

Post by Rudi » 13 May 2019, 07:55

I just got a ZWO ASI 1600 MM Cool this weekend, and I tested it last night.
As expected, the 4656x3520 pixel images takes up approximately 32 MB disk space, the camera is 12 bpp => 2 bytes.
However, after stacking, the resulting fits file is 64 MB !

The fits header for a single light, says "BITPIX = 16", where as the stacked fits file says: "BITPIX = 32".

I have upgraded from a Canon DSLR, so I have made sure, that in [Preferences]->[RAW images], "DCRAW demosaic on load" is disabled and also that Custom RAW is Disabled.
Could I be missing something else out?

A single luminance:
https://www.dropbox.com/s/scdclciu0uncb ... 1.fit?dl=0

Stacked luminance:
https://www.dropbox.com/s/f7ocjyp1yz4rd ... L.fit?dl=0
/Rudi

Dunmunro
Posts: 225
Joined: 10 Jan 2019, 01:11

Re: FITS file doubles in size after preprocessing

Post by Dunmunro » 14 May 2019, 06:49

Rudi wrote:
13 May 2019, 07:55
I just got a ZWO ASI 1600 MM Cool this weekend, and I tested it last night.
As expected, the 4656x3520 pixel images takes up approximately 32 MB disk space, the camera is 12 bpp => 2 bytes.
However, after stacking, the resulting fits file is 64 MB !

The fits header for a single light, says "BITPIX = 16", where as the stacked fits file says: "BITPIX = 32".

I have upgraded from a Canon DSLR, so I have made sure, that in [Preferences]->[RAW images], "DCRAW demosaic on load" is disabled and also that Custom RAW is Disabled.
Could I be missing something else out?

A single luminance:
https://www.dropbox.com/s/scdclciu0uncb ... 1.fit?dl=0

Stacked luminance:
https://www.dropbox.com/s/f7ocjyp1yz4rd ... L.fit?dl=0
Which stacking method were you using?

User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

Re: FITS file doubles in size after preprocessing

Post by Rudi » 14 May 2019, 10:43

Hi Dunmunro,

I am using "Sigma+Sum".
/Rudi

Dunmunro
Posts: 225
Joined: 10 Jan 2019, 01:11

Re: FITS file doubles in size after preprocessing

Post by Dunmunro » 15 May 2019, 00:58

Rudi wrote:
14 May 2019, 10:43
Hi Dunmunro,

I am using "Sigma+Sum".
It seems that AA7 converts 16 bit fits files to 32 bit whenever it makes changes to the image, such as demosaicing and/or stacking. I haven't really considered this but it is probably necessary to preserve the data in the resulting files. It happens with my 16bit images as well.

Hopefully Fabio will comment further.

fabdev
Posts: 461
Joined: 03 Dec 2018, 21:43

Re: FITS file doubles in size after preprocessing

Post by fabdev » 15 May 2019, 10:17

Hello, the stacked image has a range of one million ADU (because it's a Sum) so it's correctly saved as 32 bit.
16 bit FITS images are limited to approx. 0..65000 ADU.
The ADU range of an image is always shown on the status bar of the image (R = ... )

You may keep the range to 16 bit using "Average" or "Sigma with average" (normal Sigma).
"Sum" is useful for 8-12 bit images where quantization is an issue.

User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

Re: FITS file doubles in size after preprocessing

Post by Rudi » 15 May 2019, 15:37

Thanks for the answer Fabio.
What stacking method would You recommend for a ZWO ASI 1600MM Pro at unity gain or below?
It’s a 12 bit camera.
/Rudi

fabdev
Posts: 461
Joined: 03 Dec 2018, 21:43

Re: FITS file doubles in size after preprocessing

Post by fabdev » 16 May 2019, 14:35

I think it depends on the number of images:
up to 20-30 images there will not be any quantization and I'd use "Average" to obtain a 16 bit result.
For 50+ images I would use "Sum".

User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

Re: FITS file doubles in size after preprocessing

Post by Rudi » 17 May 2019, 06:59

Hi Fabio,

I tried using stacking method "Average", but still, the result file is 64 MB and it has BITPIX = 32 !!!

If I follow the steps below, the change seems to happen when the picture is saved.
  1. Preprocessing 16 bit fit files using "Average" method
    Image
  2. AstroArt 7 grinding the data
    Image
  3. Data has been processed, file still not saved, BITPIX is 16
    Image
  4. After the file has been saved, and without re-opening it, BITPIX is now 32!!!
    And the saved file is twice the size as each of the input fits files.
    Image
If I am doing it right, it seems that something is going wrong in AA7 when saving the file?
/Rudi

fabdev
Posts: 461
Joined: 03 Dec 2018, 21:43

Re: FITS file doubles in size after preprocessing

Post by fabdev » 17 May 2019, 09:25

Hello, what is the Range of the image?

Only if the Range is less than 0..65535 an image can be saved as 16 bit.
https://en.wikipedia.org/wiki/16-bit

( BITPIX is updated only when the image is saved. )

User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

Re: FITS file doubles in size after preprocessing

Post by Rudi » 17 May 2019, 09:52

Statistics for one of the lights:

Code: Select all

 Name  ...........  #2 m100_L_60s_001.fit
 Path  ...........  C:\Users\Bruger\Rudi Dropbox\Main\astro\2019\0512\
 Size  ...........  4656 x 3520
 Telescope  ......  12 22.9  +15 48
 Pixels  .........  16389120
 Min  ............  3952 @ 80,3357
 Max  ............  65504 @ 4210,2681
 Background  .....  4976
 Average  ........  5071.38
 Sum  ............  83115456176
 Std.deviation  ..  432.87219
Statistics after stacking:

Code: Select all

 Name  ...........  #1 Preprocessing00.fit
 Path  ...........  C:\Users\Bruger\Rudi Dropbox\Main\astro\2019\0512\
 Size  ...........  4656 x 3520
 Telescope  ......  12 22.9  +15 48
 Pixels  .........  16389120
 Min  ............  437 @ 0,0
 Max  ............  68691 @ 36,488
 Background  .....  2988
 Average  ........  2991.8873
 Sum  ............  49034400157
 Std.deviation  ..  359.3689
So, somehow it exceeded 2^16 ?

Max from all lights:

Code: Select all

 #1:	Max  ............  65504 @ 4210,2681
 #2:	Max  ............  65504 @ 4210,2679
 #3:	Max  ............  65504 @ 4210,2678
 #4:	Max  ............  65504 @ 4209,2679
 #5:	Max  ............  65504 @ 4211,2679
 #6:	Max  ............  65504 @ 4210,2679
 #7:	Max  ............  65504 @ 4208,2679
 #8:	Max  ............  65504 @ 4208,2679
 #9:	Max  ............  65504 @ 4208,2677
#10:	Max  ............  65504 @ 4208,2678
#2-1:	Max  ............  65504 @ 1461,2481
#2-2:	Max  ............  65504 @ 2205,164
#2-3:	Max  ............  65504 @ 2202,164
#2-4:	Max  ............  65504 @ 2204,162
#2-5:	Max  ............  65504 @ 2204,162
#2-6:	Max  ............  65504 @ 2203,164

/Rudi

fabdev
Posts: 461
Joined: 03 Dec 2018, 21:43

Re: FITS file doubles in size after preprocessing

Post by fabdev » 17 May 2019, 10:57

OK, the range is 437..68691 (it can be read on the status bar of the image too), so it exceeds the 16 bit range.

The cause is the flat field correction. This can happen if the bias of the camera is low and the vignetting is strong.

To force a 16 bit range before saving there are several ways, for example the command: Arithmetic / Clamp / Maximum.

For a scientifically correct result the flat field must also be corrected from its dark frame.

User avatar
Rudi
Posts: 152
Joined: 08 Jan 2019, 04:47

Re: FITS file doubles in size after preprocessing

Post by Rudi » 17 May 2019, 13:44

Ok Fabio. I will use dark flats next time.
I was also recommended doing that for my new camera on a danish Astro site.
/Rudi

Post Reply