How many hot pixels is excessive?

Last night I was taking some fireworks shots in town and noticed quite a few hot pixels in the jpeg. I put the raw into Darktable, and enable hot pixels. With default settings it fixes 70 pixels. With detect by 3 neighbors that number jumps up to 337 pixels. The reason I enabled detect by 3 neighbors is I noticed one hot pixel wasn't fixed, no matter how much I mess with threshold and strength.

• Camera- Canon refurb'ed M200, less than 1 year old (~1900 pictures taken by me, don't know actual shutter count).
• Shot settings- 3.6s shutter speed, f8, 100 iso.
• Ambient temp- mid 80F with high humidity.
• Demosaicing- LMMSE in Darktable.
• I'm not comfortable putting up the shot since it's where I live.

That is a lot of hot pixels isn't it?

Take a "dark frame", that is, a capture with the lens cap on, maybe also in a dark room if there are light leaks, then use that capture to test for hot pixels. If you did that, well and good, your post didn't specify... I don't think I'd trust an image of fireworks to isolate hot pixels.

With adjacent pixel correction, you can probably tolerate a large number of isolated hot pixels in a given image. The impact is certainly image-specific, as different patterns in the scene will tolerate fixed pixels differently. That tolerance will also be affected by sensor size; more pixels comprising the sensor will materially affect the failure rate computed from the pixel total / number of hot pixels.

Here's two straight out of camera jpg with lens cap on and no noise reduction settings in camera on:

f16, ISO 800, ss 1/1000, 15mm. In raw file, Darktable says 5 hot pixels fixed. Jumps to 7 with 3 neighbors enabled.

f16, ISO 800, ss 4 seconds, 15mm. In raw file, Darktable is saying 51 hot pixels fixed. Jumps to 55 with detect by 3 neighbors enabled.

It's a refurb direct from Canon. The camera doesn't have the option. I added two more stats above with sample images with a lens cap on. Still excessive?

I've heard rumors that some later Canons don't map out pixels on demand, and I don't know if that is true, but I don't think that Canon ever put the function in the menus explicitly; what you had to do is initiate a manual sensor cleaning, and just let the camera time out the cleaning itself, and it would map hot pixels at the end of the cleaning period.

The M200 doesn't have manual sensor cleaning, nor automatic cleaning on shut down. It's one feature they stripped away from this camera compared to it's predecessor (M100) and peers (M50/II, M6/II, etc).

I don't think so, but that's only a personal opinion with no statistical characterization or knowledge of your shooting (well 'cept fireworks!). Especially if they're each isolated, which means they can be corrected for each image.

Sigma digital cameras transfer a bad pixels list (actually layer sub-pixels) in meta-data for use by raw converters. The lists are pretty big - I had one around 4k if I recall correctly. Later models (Merrill) had sensors made by Dongbu Hitek which were quite bad in that respect and opening files with an app that did not average out the bad was quite revealing. The top layer which is quite thin had a good few pixels not stuck dark or hot but were brighter that their neighbors.

Starry, starry night ...

www.dpreview.com/forums/post/62274022

There was an artifact in the Canon 10D at ISO 3200 that showed that pixels were mapped out. Apparently, Canon tried to reduce the horizontal banding noise at ISO 3200 by adding or subtracting 1DN or 0.5DN (with a periodic dither) after pushing to 3200 from the same analog gain as ISO 1600, which would otherwise give only even numbers, as everything was doubled from the original digitization. I don't remember what led me to do so, but I found this by looking at only the least significant raw bit as a 1-bit B&W image, and it had 3 types of horizontal lines, white, black, and alternating white and black, except that there were a number of pixels which were the exact complement of the pattern that they were on, like so:

00100000000001000000000000000000
01010111010101010001010101010101
11111111111011111111111111111011
etc

Those minority anomalies are apparently artifacts of the replacement of mapped-out pixels, filtered after the digitized values were multiplied by 2, using the extra bit of precision. So, mapped-out pixels left their fingerprint in the raw, even though they were supposed to be hidden. The number of mapped out pixels is likely to be about double what you see in the 1-bit images, because you would expect about half of them to wind up even after filtering.

Are you using Long Exposure NR? You might have to turn off the Electronic Shutter if you are using it. Given the test shots and the conditions of the original problem (it was pretty warm), it seems to me that your problem is predictable and manageable.

Long Exposure NR wasn't enabled. From what I understamd it doubles your exposure time as it takes another shot to map out the hot pixels? It didn't seem feasible as I was trying to catch the fireworks as fast as possible.

I've never done it because it isn't something I do, but there is a dark-frame reduction technique where you take one shot using the same exposure time and ISO setting and use it as a reference to remove the noise from a batch of shots.

As previously mentioned that is impossible on the Canon M200. Canon completely stripped out the sensor cleaning function. I'd have to send it to Canon to get a hot pixel mapping.

Out of interest, I looked into the meta-data "BadPixelsList" on a few of my Sigma Foveon-sensored cameras ... the "winner", a DP2s, scored over 30k out of 14M sub-pixels! ... about 0.2%

This is a bit speculative, but putting the camera somewhere warm, and recording 4k video for as long as possible might be worth a try.

The idea is that heating the sensor increases the rate at which the crystal defects that cause hot pixels fix themselves (anneal).

Hubble does roughly this with some of its sensors, which gather hot pixels from radiation damage.

OTOH, semiconductor exponential failure rates increase with temperature, doubling roughly every 10 degrees C.

😀 Yes. Following the same Arrhenius rate law as annealing.

But we're not pushing MIL specs here - the M200 is supposed to be able to record 4k video, and presumably not only in Svalbard.

I'll give it a shot later this week.

[/quote]

As previously mentioned that is impossible on the Canon M200. Canon completely stripped out the sensor cleaning function. I'd have to send it to Canon to get a hot pixel mapping.
[/quote]

The sensor cleaning function is about removing dust, not remapping dead pixels.

Okay, nothing better to do this afternoon, so I wrote a hotpixels program. It works on the raw data, and offers an adjustable threshold:

github.com/butcherg/hotpixels

There's a Windows executable in the release; unix users can compile the code. It uses LibRaw, so if you're compiling you'll have to get/compile that.

Also, it's a command line program, so if you're GUI-centric sorry 'bout that.