Camera

[shuntcap]

Took a couple of pics to compare the built-in Camera app to OpenCamera:

According to the EXIF data embedded in the photos, they both came from "Camera Software: MediaTek Camera Application"

This is probably because it's MediaTek's JPEG codec that ultimately creates the image file regardless of what capture app is used.  MediaTek's camera kernel drivers are fairly simple and limited in flexibility: set up some parameters, tell the sensor to capture, and the data is shot over to the proprietary JPEG engine.  Out pops a file and that's all we get.

[NormMonkey]

This is probably because it's MediaTek's JPEG codec that ultimately creates the image file regardless of what capture app is used.  MediaTek's camera kernel drivers are fairly simple and limited in flexibility: set up some parameters, tell the sensor to capture, and the data is shot over to the proprietary JPEG engine.  Out pops a file and that's all we get.

That may well explain why these two photos both seem to have similar compression artifacts even though I told OpenCamera to do 100% JPEG quality.  Then the best OpenCamera can do is to not make worse what MediaTek's JPEG engine spits out.  I would expect a true 100% quality JPEG to be larger than 6-7MB for a 24MP photo.

That would also explain why RAW doesn't work.

I really hope this is something that PC can work out in software.  e.g. maybe MediaTek's driver has a RAW option that is currently disabled.  If so, we might yet get good performance from this thing.  If not, we may have here a good camera with a spudmaker bolted to its tail so that it only ever produces potato quality pics.

[shuntcap]

That may well explain why these two photos both seem to have similar compression artifacts even though I told OpenCamera to do 100% JPEG quality.  Then the best OpenCamera can do is to not make worse what MediaTek's JPEG engine spits out.  I would expect a true 100% quality JPEG to be larger than 6-7MB for a 24MP photo.

Possibly, but I would expect the MediaTek JPEG codec to respect the Camera2 API option for JPEG compression level.  It's more likely that the artifacts are due to software interpolation of a (much!) lower resolution image.

That would also explain why RAW doesn't work.

I really hope this is something that PC can work out in software.  e.g. maybe MediaTek's driver has a RAW option that is currently disabled.  If so, we might yet get good performance from this thing.  If not, we may have here a good camera with a spudmaker bolted to its tail so that it only ever produces potato quality pics.

The CMOS sensor does output RAW data over the MIPI bus, but how that gets handled is up to the OEM (MediaTek).  MediaTek (and other hardware vendors) supplies a board support package and board reference design to its customers.  It's very unlikely that Planet has the resources to write additional kernel drivers, so they're more likely to use whatever code MediaTek has already written.  That limits their selection of an image sensor.  I had originally assumed that Planet would choose the OV23850 sensor for their board design because that sensor is already in the MediaTek kernel (3.18 for the Gemini).  That is a 23.8 MP 1/2.3" image sensor.  Interestingly, it does have a 6MP resolution setting, the apparent resolution Daniel W calculated.  In a very far-fetched scenario, the proprietary libraries could be incorrectly coded to work at only 6MP, but I highly doubt it.  I've emailed Planet support asking what sensor is used.  If they don't answer or don't know (they had to ask their ODM with the Gemini), I'll have to wait until they release the Cosmo kernel source code to know for certain what sensor is employed.  I did not see any RAW option in the OV23850 kernel driver.  It wouldn't be in there anyway, not in MediaTek's camera software model at least.

When I offered to assist Planet in tweaking their SP5509 image sensor code for the Gemini's rear camera, they essentially and politely said, "Thanks, but no thanks."  Worse, they really didn't know anything about the sensor except that it was performing to QA specifications.  They did very kindly try to get documentation for me but couldn't obtain any.  I even tried to contact the vendor, SuperPix, but got no response.  So Planet probably doesn't have much control over the behavior of the camera.

[NormMonkey]

FWIW, termux-camera-info suggests a max size of 4208x3120 for the rear camera.  It does suggest it has "raw" capabilities.
I'll have to play with what termux-camera can do with it.

Has anyone had any interesting experience with camera apps other than the built-in Camera app and OpenCamera?

[drpeter]

FWIW, termux-camera-info suggests a max size of 4208x3120 for the rear camera.  It does suggest it has "raw" capabilities.
I'll have to play with what termux-camera can do with it.

Has anyone had any interesting experience with camera apps other than the built-in Camera app and OpenCamera?

Is there any way simply to get a report of the sensor used?  Other than asking Planet, which might prove unhelpful?

Quoting a 24MP camera which is in fact simply an upscaled 6MP or 12MP sensor seems a bit like smoke & mirrors. Although, I sincerely doubt there is true 24MP resolution to be found in any sensor of this size. In which case, capping the resolution of output images to the true effective resolution of the camera system would be the sensible approach, unless they can be output in RAW for bespoke post-processing.

In the end however, if you're looking for that kind of quality you'd only be using a 'phone camera in extremis, when no more capable camera was available...

[Daniel W]

Do I spot a pattern here? We know the Gemini SP5509 is a 5MP camera module, as advertised. Yet, NO photo I have taken with it, has seemed to contain anywhere near 5MP worth of actual detail. In some thread, I think we even half joked that perhaps they were, by mistake, capturing, and interpolating, a frame from the viewfinder video stream, which would likely be 1440x1080, as the display is 1080 pixels tall and the sensor has a 4:3 aspect ratio.

What if... it is, actually, worse? What if the default behavior of the "MediaTek Camera Application", which appears in the EXIF for Gemini rear camera photos too, for whatever silly reason, is to demosaic by clumping photosites together 2x2 and then interpolate back up to the advertised resolution? It would be CPU-friendly for sure, so it would reasonably be decently fast even on really low end hardware. It could perhaps be a factory fallback default, that the device maker is supposed to change.

That would give 24MP Cosmo photos 6MP worth of details, stretched back to 24MP and Gemini photos 1.25MP (1280x960) worth of details, stretched back to 5MP. It would be really bonkers, but it does kind of match the amount of detail that seems to come out of the Gemini rear camera.

In [a href=\'index.php?showtopic=35308\']this post[/a], I posted a photo of a page of text, from Gemini rear camera. I repeated my experiment with that picture today, reducing it to 50%, both ways, which, thanks to the crude code in Windows MS-paint, throws away half the pixels, both ways. Then I zoomed it to 200%, and - again - apparently NO detail was lost... The reduced and zoomed in picture was, to me, exactly as readable as the 5MP image. If anything, the processed image felt less objectionable, as it was fairly sharp and detailed, for a 1.25MP image, that is, while the 5MP image feels a tad out of focus.

Here's a composite example: (please click to see all the pixels)
 

Never mind the language (it's a fragment of an article about a local toy store). The left half is a completely unprocessed crop from the Gemini rear camera, warts and all. The right half, has been shrunk down, by throwing away every other pixel, both ways, and then scaled back by doubling the remaining pixels both ways. I saved it as a PNG, so I've added no other artifacts. No, they're not identical. The technically inclined can subtract one half of the image from the other and the result won't be completely black, but with the naked eye, can you spot ANY difference in the amount of detail?

I'm getting that sinking feeling in my stomach, that I might, regretfully, be on to something here... They shouldn't be THAT similar, lest the left half was created by stretching something that was half the resolution to begin with... On the other hand, maybe this suggests the SP5509 isn't QUITE as bad as we all think. Maybe it's just really craptastic image processing, that turns it into a glorified 1.25MP camera. Maybe, in theory, it would somehow be possible to get somewhat better pictures out of the SP5509 and, by extension, out of whatever module the Cosmo is using?

[Daniel W]

I sincerely doubt there is true 24MP resolution to be found in any sensor of this size.

As far as the sensors are concerned, they do exist. The OmniVision OV23850, mentioned above by shuntcap is "smartphone-sized" ~1/2.3" (the way sensors are measured is a weird story in itself, but that's for another day) and has 23.8MP for real. [Source: https://www.ovt.com/sensors/OV23850]. The Sony IMX576 is of similar size and has 24.8MP. The Samsung S5KGM2 even squeezes 48MP onto a 1/2.25" sensor, though, as it uses a so called Quad Bayer filter array, it's really meant to produce (really nice) 12MP final images, but it still has 48000000 photosites on a sensor that comfortably fits inside a phone. Whether the optics in a smartphone could resolve that much detail, is another question. My grasp on the diffraction limit for such systems is a bit fuzzy (pun intended), and is perhaps a bit off topic anyway, but 24MP should be possible to resolve, I think.

[shuntcap]

24MP should be possible to resolve, I think.

Yes, but it would be extremely noisy leading to a very high degree of noise reduction.  A lot of texture detail (grass, clothing, hair) would be smeared by the noise reduction resulting in sharp contrast edges but oil painting-like textures elsewhere.  It's usually better to go with a smaller sensor as the larger pixels take in more light and produce less noise.

On that note, here is the reply I received from Planet:
"The chipset on the Cosmo (Mediatek P70) is compatible with a single Samsung-made 20MP sensor. I couldn't, however, find any details about the part itself. It might be more relevant if you have a look at https://www.mediatek.com/products/smartphon...iatek-helio-p70, as our support team does not directly get involved with the supply chain."

At least it's a Samsung, not a sensor from SuperPix like the Gemini's.  Here is Samsung's selection of mobile image sensors (under Product Selector):
https://www.samsung.com/semiconductor/image...e-image-sensor/
There are three sensors in 20MP resolution.  There are also two 24MP sensors.  Until we get the Cosmo's kernel source, we won't know which one is used.  A quick look at the Gemini's kernel source indicates that the MT6797 (the Gemini's processor) supports a handful of Samsung sensors.

Assuming that "20MP" is not a typo, our hunch that the images are being interpolated could be correct.  But interpolated from what resolution?  Certainly not 20MP!  It's more like 6MP as we've discovered.  Anyone care to try capturing at 20MP instead of 24MP to see if the detail level increases in the hope that the sensor is misconfigured to create Planet's advertised 24MP from a mere 6MP?

[shuntcap]

The Samsung S5KGM2 even squeezes 48MP onto a 1/2.25" sensor, though, as it uses a so called Quad Bayer filter array, it's really meant to produce (really nice) 12MP final images, but it still has 48000000 photosites on a sensor that comfortably fits inside a phone.

Daniel W may be onto something here.  All of the Samsung 20MP and 24MP mobile sensors are Tetracell, which is Samsung's answer to Sony's Quad Bayer array.  Quoting Samsung (https://www.samsung.com/semiconductor/insig...-day-and-night/):
"When the camera is shooting in low-light conditions, the algorithm combines information and data from each group, essentially transforming four smaller pixels into a larger pixel. In bright lighting conditions, the re-mosaic algorithm remaps the pixels into a conventional RGB pattern through an algorithm called re-mosaic algorithm to produce high-resolution photos."

In other words, in dim lighting, 24MP becomes 6MP (24 divided by 4); in bright lighting, 24MP supposedly remains 24MP.

This invariably raises two questions:
1) Was fluffypony's iPhone/Cosmo comparison really done in dim lighting? Doesn't look it to me, and the low ISO ratings in the EXIF tag data confirm good lighting.  To be sure, vader's night scenes report an ISO rating of 6399 (thanks, vader!)
2) How well does this "re-mosaic algorithm" really work with respect to reproducing an image of full resolution, not interpolated quarter resolution?  Maybe it just doesn't work and that's the apparent software interpolation we're seeing, in which case we're stuck with a 6MP camera.

Everyone grab your smelling salts and sit down:
https://www.gsmarena.com/quad_bayer_sensors...-news-37459.php
"A Quad Bayer filter is a bit of a misnomer as it’s actually the same as a regular Bayer filter. What really changes is not the filter but the sensor behind it – these new sensors put four pixels behind each color square instead of just one.
"So, really these 48MP Quad Bayer sensors can’t offer much more detail than a 12MP sensor. Sensor and phone makers alike will tell you that smarter demosaicing algorithms can capture more detail, but our experience is that the gain is small – if there’s a gain to be had at all."

That spells a very low effective resolution for our Cosmo.  If I'm right, don't blame Planet Computers, blame Samsung/Sony marketing.

[shuntcap]

For those of you still breathing, the Android Authority article "Don’t fall for the 100MP camera hype" further debunks the Tetracell/Quad Bayer marketing gimmick:
https://www.androidauthority.com/100mp-cameras-1016587/

Highlights:
"Samsung’s 64MP GW-1 sensor calls this Tetracell technology... This is definitely not equivalent to a full resolution Bayer-filter sensor. The GW-1 provides just 16MP worth of color data, with some extra contrast data. The re-mosaic algorithm will pull out slightly more detail than a regular 16MP sensor, but certainly not anywhere near proper 64MP worth of detail."
"Quad-Bayer sensors are not equivalent to full resolution Bayer-filter cameras."
"These sensors help advertise a smartphone with a high pixel count and bigger pixels, but they're really neither."
"Ultimately, these huge megapixel numbers are mostly about marketing."

The facts seem to be leading to this conclusion: while the Cosmo's camera might produce better low-light images, it's really just a 6MP camera.

[shuntcap]

Has anyone seen an option to disable pixel binning in any camera app used so far?  If so, disabling it MIGHT allow the Cosmo's sensor to spit out a full-resolution 24MP image.  Open Camera enables pixel binning (merging four pixels into one) in its Noise Reduction photo mode.  I don't know if this directly affects the sensor's output or if it's just an Open Camera algorithm.  I don't have high hopes for this, though.  Again quoting Android Authority:
"In our experience, the level of detail when switching between pixel binning is not huge. Many phones, such as the Redmi Note 7 Pro, actually perform better with pixel binning left on. This is due to the superior light capture performance and because the re-mosaic algorithm produces so-so results. For this reason, manufacturers often default to pixel binning rather than giving users high-resolution pictures."

My Cosmo is still sealed in a drawer... I won't open it until we get to the bottom of this.

[drpeter]

I sincerely doubt there is true 24MP resolution to be found in any sensor of this size.

As far as the sensors are concerned, they do exist. The OmniVision OV23850, mentioned above by shuntcap is "smartphone-sized" ~1/2.3" (the way sensors are measured is a weird story in itself, but that's for another day) and has 23.8MP for real. [Source: https://www.ovt.com/sensors/OV23850]. The Sony IMX576 is of similar size and has 24.8MP. The Samsung S5KGM2 even squeezes 48MP onto a 1/2.25" sensor, though, as it uses a so called Quad Bayer filter array, it's really meant to produce (really nice) 12MP final images, but it still has 48000000 photosites on a sensor that comfortably fits inside a phone. Whether the optics in a smartphone could resolve that much detail, is another question. My grasp on the diffraction limit for such systems is a bit fuzzy (pun intended), and is perhaps a bit off topic anyway, but 24MP should be possible to resolve, I think.

I don't doubt that phone sensors with such a theoretical pixel count exist, but whether they output images that contain genuinely higher spatial definition than sensors of the same size but with half the pixel count is another matter...

Given that mid-to-high-end dedicated SLR cameras ($1000+) haven't yet really usefully progressed beyond 24MP sensors, it's probably unrealistic to expect better genuine spatial resolution than 6-8MP from any phone camera sensor, still less from a low-volume non-specialist mid-range phone like Cosmo.  Adequate for good snaps taken in favourable lighting conditions for viewing on a 6 inch screen is a realistic expectation.

Bear in mind that a 1/2.3" camera sensor has less than 1/10 the sensor area to work with as APS-C format DSLRs and even the somewhat larger sensors in high-end phone cameras such as the 1/1.7" Huawei P30 Pro have only 1/8 the sensor area of APS-C...

IMHO if you need better than full 8MP spatial resolution (which you don't for viewing snaps on a phone screen), you still need to buy and suffer the inconvenience of carrying a 'proper' camera- you can get a pretty good one second-hand for much less than the difference in price between a Cosmo and a high-end phone.  Not forgetting the myriad other advantages a 'proper' camera will inevitably have over any phone camera for 'serious' as opposed to 'snap' photography.

[vader]

After a bit of digging, I found out some info on what I believe is the module in the cosmo. I believe it is the Samsung tetracell (quad bayer filter) 24MP 0.9um 1/2.3 inch sensor. This is a true 24MP sensor, which in low light, can automatically pixel bin for greater well depth. It isn't an 8mp upscaled to 24, and in my tests, it performs quite well. The actual model appears to be the 24MP ISOCELL Slim 2x7 ( https://www.samsung.com/semiconductor/minis...ocell-slim-2x7/ ). Now there are conversations to be had about sensors with sub micrometer pixel sizes, but the main benefit is that in good light, it is easier to fill the well, and you can get good DR with high resolution. In lower light (eg. night mode), the sensor will bin pixels to gather more light per "pixel", but at lower resolution.

The main problem with small pixels, is a small well size. This means that amplification (ie. ISO) causes noise due to quantization. Keeping the ISO down, and exposure times long decreases the amplification, and hence noise for a better picture. The default camera app tends to go for quick shots with high ISO, resulting in more noise. Apps like OpenCamera can give you manual control so that you can lower the ISO and get a better shot, or alternatively in NR mode, take 4, 8 or even 12 shots and median filter to get rid of noise. So, I believe the camera module is quite capable, and with the correct software can take good shots.

EDIT: should have reread shuntcap's post a few above. Basically says the same thing. Quad bayer does have advantages (and disadvantages), but the take away is that it needs the correct settings to achieve good results, probably more so than standard bayer pattern sensors. With the right settings, however, it can produce excellent results.

[drpeter]

After a bit of digging, I found out some info on what I believe is the module in the cosmo. I believe it is the Samsung tetracell (quad bayer filter) 24MP 0.9um 1/2.3 inch sensor. This is a true 24MP sensor, which in low light, can automatically pixel bin for greater well depth. It isn't an 8mp upscaled to 24, and in my tests, it performs quite well. The actual model appears to be the 24MP ISOCELL Slim 2x7 ( https://www.samsung.com/semiconductor/minis...ocell-slim-2x7/ ). Now there are conversations to be had about sensors with sub micrometer pixel sizes, but the main benefit is that in good light, it is easier to fill the well, and you can get good DR with high resolution. In lower light (eg. night mode), the sensor will bin pixels to gather more light per "pixel", but at lower resolution.

The main problem with small pixels, is a small well size. This means that amplification (ie. ISO) causes noise due to quantization. Keeping the ISO down, and exposure times long decreases the amplification, and hence noise for a better picture. The default camera app tends to go for quick shots with high ISO, resulting in more noise. Apps like OpenCamera can give you manual control so that you can lower the ISO and get a better shot, or alternatively in NR mode, take 4, 8 or even 12 shots and median filter to get rid of noise. So, I believe the camera module is quite capable, and with the correct software can take good shots.

EDIT: should have reread shuntcap's post a few above. Basically says the same thing. Quad bayer does have advantages (and disadvantages), but the take away is that it needs the correct settings to achieve good results, probably more so than standard bayer pattern sensors. With the right settings, however, it can produce excellent results.

Small correction: according to the link it's a 1/2.8" format sensor  

[vader]

Small correction: according to the link it's a 1/2.8" format sensor  

Yep, absolutely correct, I didn't pick that one up