Astro camera, weird artifacts when looking more closely

[Zarhan]

Hi,

  I'm pondering the Astro's camera. I'm using opencamera (set to use the camera 2 API).

  I'm attaching a picture. It's nothing special, it's just a pic of the rental car we drove around in New Zealand, taken at the rental firms parking lot. Hence I used Astro's camera.

  Here's first the full pic, in all it's glory.

 

  Take a look at any part of the picture, zoomed in:

 

  What on earth is causing these artifacts? Wiggly bits going all over?

  You'll notice that this is taken on bright summer day, ISO 115. It's *not* noise due to heavy amplification.

  This seems to be present in lots of other images as well, especially when there's lots of white or black surfaces. For skintones and the like it's not that bad (but e.g. take a look at the hills in the background). Any ideas on what's going on and if there's some opencamera setting that would fix it?

  Also, there seems to be a distortions (rectangular fisheye effect, so to speak - see how distorted the buildings on the side are), but that's more of an optical issue.

[vader]

I'm out and about at the moment, but will try too post again with examples later. The Astro uses a quad bayer sensor. This means instead of the normal bayer pattern, you have 4 pixels in a square for each colour. You normally have one line with green, red, green red etc, and another with blue, green, blue, green etc. The cmera device interpolates (guesses) what the colour shouuld be where it is missing. Green is most sensitive to the human eye, hence it has twice the pixels. In a quad bayer, it has to do more guessing on the colour. Secndly, quad bayer tends to be used with high pixel sensors, so each well (pixel) is quite small.

Appoologies in advance fr the physics stuff......

A pixel is like a bucket, and photons (light) are ike rain dops. The drops fall into the buckets, and if you time the exposure to the rain, the bucket will fill up dependiing on how much rain goes in. A small bucket still fills up to the same height as a large bucket in heavy rain, however in light rain, if the bucket is small, it starts missing rain drops. Sometimes it might get a few eetra, some times a few less. This is noise. The larger the bucket, the lower the percentage of missed or extra raindrops and the more accurate the reading of how much rain was captured.

So going back to cameras, the bucket is a pixel on a sensor, and the image is made up of the levels of all the bckets. The larger the bucket, the lower the noise. When more photons bombard the well, the lower the noise. Combinnig pixels in a quad bayer sensor results in effectively a bigger bucket. Interpolating (guessing) also adds noise, so using the full sensor wll result in more noise than pixel binning (bigger bucket). Add in to this HDR, which amplifies llow light levels and you add even more noise. I did a few very quick tests, and using the default camera (not open camera), when you select 12MP and turn off HDR, you actually get quite an impreessively low noise, high quality image.

What I am trying to say in this overly long, but hopefully helpful post, is that the noise is a result of the processing options chosen for the photo. High res, hdr images will produce large gobs of noise when you zoom in, but lok quite good zoomeed out. For the best result, give the camera a chance and shoot 12MP, low ISO (if available) and turn off HDR.

Phones like the pixels, iphones, galaxies have even mre tricks and take multiple photos, averaging them to remove noise. That is called computational photography. You can do this yourself after the fact on a PC with excellent results if you really want.

I'll post examples later today.

[Zarhan]

What I am trying to say in this overly long, but hopefully helpful post, is that the noise is a result of the processing options chosen for the photo. High res, hdr images will produce large gobs of noise when you zoom in, but lok quite good zoomeed out. For the best result, give the camera a chance and shoot 12MP, low ISO (if available) and turn off HDR.

As I said, the example pictures are taken during a bright summer day, with ISO 115 autodecided by the camera.

It's *not* thermal noise. I've been doing digital photography for 20+ years, right now my primary camera is Canon R6, and I recognize what thermal noise is.

Also, another related issue is that I forgot - if I try to tell opencamera to save image in raw format (DNG), the camera just locks up. Trying to restart OpenCamera says that "Camera may be in use by another application".

My guess is that Mediatek's binary blob is again messing up the image by destroying the raw data from the sensor.

But ok, if the issue is the quad Bayer sensor, then that means that the advertised 48 Mpix is not the true resolution, that's just the number of raw pixels behind the Bayer filter. I'll make a few comparisons and take 48 MP and 12MP pics of same subject, and then see what happens if I scale 48MP image down to 12 MP image with Gimp.

[sobukus]

Well, the MP rating of digital cameras usually is the total number of pixels, not the number of pixels per colour. The Bayer interpolation magic is used to construct three times the pixels. That's the deal with every colour sensor. It works good enough, so we do it. Though, people are removing colour filters from sensors to get true greyscale megapixels for astro photography.

The lockup with RAW sounds unpleasant. Ich checked with Open Camera on my device … it produce a huge DNG file just fine. I didn't check it on my PC (the gallery cannot view it), though. I remember Open Camera sometimes having trouble with the camera on a Galaxy S5 with LineageOS. When that happened, I had to reboot to get camera functionality back.

With the huge size of image files (at least raw) and the limited quality of the optics, I intended to opt for 12 MP photos, anyway. That is what my Canon 5D full frame DSLR uses to produce beautiful and detailed pictures, with proper glass. 48 MP out of a phone camera is just a waste of bits, IMHO. The quad bayer also clearly lends itself towards binning to the quarter resolution, anyway.

I don't think you'd be happy with 48 MP vs 12 MP on this sensor, and perhaps part of that is what you describe with these patterns on close inspection. We may differ in the opinion that I don't think it's a great loss. 48 MP is too much data to my taste, anyway (well, there goes the quip about 8K video …).

[Zarhan]

I did a couple of tests, by compositing same picture. Bright winter day. Comparing 12 megapixels (4000x3000) vs 48 megapixels (8000x6000) shows a few things.

First, the shot settings, as per OpenCamera:

Camera make  : Planet_Computers
Camera model : Astro_Slide_5G_Transformer
Resolution   : 4000 x 3000
Flash used   : No
Focal length :  4.7mm
Exposure time: 0.0005 s  (1/1838)
Aperture     : f/1.8
ISO equiv.   : 115
JPEG Quality : 90

Note that JPEG quality is 90. I also tried with 98, but didn't see any discernible difference to 90 except much larger files.

Anyway, here's first the full composition, scaled to 640x480, just as reference:

Now, let's look at couple of details.  The rocks near the middle of picture:

48MP:
 

12MP:
 

No difference really, the squiggles are still there.

The sign near left side of image:

48MP has *slightly* more detail. The sign says "Kalevan kirkko". It's *barely* more legible in this version, so the extra pixels are not completely an illusion.
 

12MP:
 

Finally, the lamppost and windows of the building in the background. No difference whatsoever, and really simple details:
48MP:
 

12MP:
 

Taking a DNG still crashes my camera, need to see if I can somehow fiddle that to start working.

[Zarhan]

Addendum. I can take DNG if I tell Opencamera to take *only* dng. No RAW + JPEG.

However, the resulting DNG is...weird, at least when looking at it through RawTherapee. Sky and snow is overexposed (saturated, really), and the parts that are not sky are really...fuzzy. There are no clear boundaries anywhere. Everything is fuzzy. But at least there are no squiggles directly in this raw image. Also, the DNG is almost 100 MB each.

With a liberal use of unsharp mask, edge sharpening and similar settings I can get something resembling a proper image - however, the result is also somewhat similar to the JPGs captured directly by the camera. This is just based on 10 minutes of fiddling around, but the weird squiggles are back, perhaps less pronounced.

So my guess is that the sensor actually, well...is not too good. There's some Mediatek binary blob-algorithm that tries to make something out of it but the results are not all that good. However, if you look at the raw DNG, it's also not all that good starting point.

In theory, it might be possible to come up with some sort of raw processing profile that would convert the DNG to something satisfactory - perhaps with 12Mpix resolution - and maybe even install rawtherapee-cli on the Astro to do it automatically. But that's going to take a bit of doing first.

[shuntcap]

48MP has *slightly* more detail. The sign says "Kalevan kirkko". It's *barely* more legible in this version, so the extra pixels are not completely an illusion.
 

12MP:
 

The apparent difference in detail is more likely because the second (12MP) photo simply shot it slightly sharper: either less shake, or a slightly (very slightly) different focus.  I've seen that happen between shots when testing my mirrorless cameras and lenses.

Thanks for the tests.  I've been wondering for over two years how the Astro's camera would turn out.  Unfortunately it sounds like my predictions weren't too far off:
https://www.oesf.org/forum/index.php?topic=36385.msg297221#msg297221

I don't have an Astro, and I'm done porting GCam (for the Cosmo or otherwise), period.  But I heard from some Pro1 X users that an existing GCam port works well on their devices, which also have 48MP sensors:

Google Camera Port Updates
BSG: MGC_8.4.600_A10_V13:
https://www.celsoazevedo.com/files/android/google-camera/dev-bsg/f/dl92/

Maybe it'll work on the Astro.

[Zarhan]

I don't have an Astro, and I'm done porting GCam (for the Cosmo or otherwise), period.  But I heard from some Pro1 X users that an existing GCam port works well on their devices, which also have 48MP sensors:

Google Camera Port Updates
BSG: MGC_8.4.600_A10_V13:
https://www.celsoazevedo.com/files/android/google-camera/dev-bsg/f/dl92/

Maybe it'll work on the Astro.

Unfortunately, doesn't look like it. I tried https://1-dontsharethislink.celsoazevedo.com/file/filesc/MGC_8.4.600_A10_V13_ENG.apk and basically got a black screen out of it.

[Zarhan]

Someone mentioned on IGG that "camera is much better" after firmware update.

With just a quick look, those weird squiggles I've pointed out in earlier messages appear to be gone.

With a slightly longer look, I'm noticing the following:

- It *looks* like they have made some changes in the lowpass filter for JPGs. It appears to be more aggressive than before. To test this, I took a picture of my bookshelf from a distance. The book titles with larger fonts are legible, but smaller fonts are just blended in with the background.
=> Artifacts do get removed, but so do very small details
- RAW output seems to work much better. In previous version I got the "whiteout" effect. This time, the DNG opens up in RawTherapee without anything weird.
- Taking both RAW (DNG) and jpeg at the same time works now!

Attaching two screenshots to illustrate really small details. These are just from my bookshelf (Alastair Reynolds: Poseidon's Children Trilogy, Finnish translation).

This is with a 400% zoom so you can clearly see individual pixels.

Shot settings were

Exposure time: 0.020 s  (1/50)
Aperture     : f/1.8
ISO equiv.   : 2361  <= Note the ISO level, there's lots of thermal noise going on here. This is on purpose, just to see how low-light conditions are handled.

Photo was taken maybe 4-5 meters away with 48 MP. JPEG quality was set at 98%.

Remember to click on the images so that forum shows them in original size!

First, the JPG:
 

Then the DNG:
 

Especially of note:
 - The JPG appears to be much less noisy (because of aggressive filtering). Look at the sleek blackness of each book cover.
 - The lowpass filter also eats through some details. Compare the smaller "O" in "Poseidonin lapset" of the middle book between RAW and JPG. The O in author's name ("Reynolds") is fine in JPG as well. However, in JPG, the D in Reynolds looks more like an O, while in the RAW it's clearer that it's a D.
 - The publisher's logo is reduced to a blob in both versions, although in the raw version you can (maybe) make out the word "Scifi" (publisher is 'Like Scifi').
 - Several horizontal lines (e.g. the letter I in center book's Alastair) show "checkered" pattern. In RAW, this is visible much more clearly and in more letters. In JPG the lowpass removes most of them. I would imagine this is due to the sensor's physical (Quad Bayer?) filter blocking individual pixels. The weird part is that the text is white, so it should come through regardless of individual pixel's filter.

Anyway, suddenly the camera is *much* more usable - and they are not even mentioning this in release notes!

[Robsno5]

I would suggest using "Camera for Android" as when zooming in the detail seems less simplified while still removing grain. I went into settings and set to 48mp and the zoom in looks much better than Open Camera. yes I know it has adverts but the pictures are worth it.

[Robsno5]

Actually changing the processing settings part of the settings in open camera to fast works well too.

[Zarhan]

Actually changing the processing settings part of the settings in open camera to fast works well too.

Are you referring to Edge mode algorithm or Noise reduction algorithm (or both)?

[Robsno5]

Noise processing as this reduces the weird effects, still seems to look a bit better on the camera for android app for me

[Dom (shymega)]

I haven't really noticed much of a difference, tbh. Maybe my sensor is dirty?

[Robsno5]

It is not always a huge difference but is either slightly better or a lot better depending on the light so sometimes the artifacts are visible but less prominent and sometimes they are significantly improved on my Astro. Certainly I am leaving open camera on fast noise processing going forward as the high quality is over processed.