How do you fix a broken download update tool with a broken download update tool?

[MonkeyControl]

I've got a question that I'm probably going to regret asking.  Not sure if this belongs here, on the firmware update thread : https://www.oesf.org/forum/index.php?showtopic=36044, on the thread that opened the scary DigiTime can of worms : https://www.oesf.org/forum/index.php?showtopic=36031, or in the rubbish bin.  
If the OTA upgrade tool is now broken, (I've got a spinning circle that's about as responsive as Windows10), looking at a site that effectively doesn't exist, (for good reason, following The Chronicles of Ninji, (Thanks Ninji for investigation, Daniel W for raising the subject here, all others involved, and Planet for taking some prelim action)), we need a new update site or tool.  But if the update tool to fix the update tool is broken, how do you fix that?  
How can our Cosmos that don't 'know' this, download an OTA update to change the OTA update details, isn't that a chicken and egg thing.  How do you fix the thing, if the thing that needs fixing is the thing fixer?  If you see what I mean.  
Or will we be maybe given a browser link to an alternative service to download a one time fix for the download tool, or to download a new download tool?  Or will we need to do some other type of manual update?  Or am I missing something?  
Apologies if this seems like a dumb question, I'm just an Electrical Engineer with a groovy mobile.  Answers in crayon on A3 paper please.  
Thanks again to Ninji for all your sterling work to keep us sleeping safe and sound with our Planet devices.  I read your articles end to end a couple of times and got about one word in every three, but from what I understand in basic principles, and from the responses of other OESF members, I truly appreciate the work you have done.  The way you have given up your free time and expertise 'for fun' for the good of us all is so very refreshing to a cynical old fart like myself.

[Daniel W]

How do you fix a broken download update tool with a broken download update tool?

We are aware that some of you have issues upgrading the Android firmware of their Cosmo Communicator This is most likely due to a misconfiguration of the software update server, which will be dealt with top priority early next week when people will come back to work from their Chinese New Year holidays.. Unfortunately, until then this issue is completely outside our control.

As mentioned in the previous update, we are also working on a new system to replace the current software update mechanism with an in-house solution, which will give us more flexibility. A first software update prototype has already been built and in the next few weeks we will shape it into a full-feature product. When ready, the new update mechanism will be released as a normal FOTA update, so the update will be transparent to users. In the meantime, please do not attempt to update the firmware using any other method as the update files may not be compatible with your Cosmo OS version and may cause your Cosmo not to boot or function properly. Similarly, please do not attempt to use Gemini Linux or other OS images on your Cosmo.

Planet seems to think it's a "misconfiguration of the update server", and thus believes they will be able to use the Digitime FOTA system to distribute the next update as soon as the servers are fixed. To me, as the Digitime web site is still offline and the first few URL:s the Cosmo is supposed to query, still just gives me "Welcome to nginx!" or nothing, it seems someone just wiped the servers clean. As the Lantern Festival 2020, ending the CNY celebrations, is on Saturday, February 8, I guess people will come back to work on Monday, February 10, though what that might lead to, remains to be seen.

Anyway, the FOTA updater already on our devices, can perform an update from a local file, so should Digitech stay offline, Planet can make such a file available for manual download and e-mail us a link. Regardless of how it's distributed, it will most likely be a so called binary patch, just changing what needs to be different, as those can typically be much smaller than an entire image, but they only work if what's already on the device is as expected, so a binary patch for V15 might not work on a device already updated to V16 or V19. That's why, I think, they're urging us to not get creative.

[Ninji]

I'm also curious myself as to how this will be resolved...

There are four different situations going on here, all at the same time:

1. The main Digitime website was taken down by repointing www.digitimetech.com to a random IP address - this happened ~3 weeks ago, but this didn't affect the updates as they talk to a different subdomain.

2. Following Planet's release of V19, the Digitime servers stopped offering the V16 update to V15 Cosmos, meaning that they were stuck on the initial version until they manually updated. This still seems to be the case today.

3. The server at eu-app-fota.digitimetech.com (IP address 47.254.145.86) has stopped responding to requests - this is different to point 1, as the IP has not changed!

4. The fall-through logic in the updater which contacts a different domain (eu.fota.digitimetech.com) if a request fails doesn't seem to work properly - that server still works, but the app isn't talking to it

My guess is that Digitime will resolve whatever has broken the eu-app-fota server and that will fix point 3 (and make point 4 irrelevant for now, until something breaks again)... These guys don't seem particularly competent.


re: Daniel W bringing up "Welcome to nginx" - that's just a Digitime hallmark it seems, they don't change the default pages. Indeed, app.fota.digitimetech.com currently has one, but my script is able to contact it under /v3/mob/chk for updates and receive the correct reply. You see a very similar page for OpenResty (instead of nginx) on their malware control server if you go to it: http://statistics.flurrydata.com:10000/

[Daniel W]

2. Following Planet's release of V19, the Digitime servers stopped offering the V16 update to V15 Cosmos, meaning that they were stuck on the initial version until they manually updated. This still seems to be the case today.

I slightly doubt this. According to the [a href=\'index.php?showtopic=36044\']Firmware update[/a] thread, the last time anyone got V19 from the Digitech server seems to be on January 18 or 19. My Cosmo arrived on January 23. Based on a link in [a href=\'index.php?act=findpost&pid=294268\']Missing firmware updates[/a] I manually updated to V16 on January 24 (which was potentially stupid, but at least made my Cosmo useable, as opposed to before), but I'm still getting nothing from the update servers, so, to me it, seems Digitime has completely stopped serving us any firmwares.

Daniel W bringing up "Welcome to nginx" - that's just a Digitime hallmark it seems, they don't change the default pages. Indeed, app.fota.digitimetech.com currently has one, but my script is able to contact it under /v3/mob/chk for updates and receive the correct reply.

Yes, I'm aware that just asking for port 80 with a web browser on the main URL is a VERY crude test. Your findings seems to suggest the servers are still working, which may make the Planet "misconfiguration" hypothesis a bit more plausible. Anyway when you do get a reply from the servers, does that reply imply that V19 should still be available under certain circumstances? Just having V16 doesn't seem to be enough.

[Ninji]

Yes, I'm aware that just asking for port 80 with a web browser on the main URL is a VERY crude test. Your findings seems to suggest the servers are still working, which may make the Planet "misconfiguration" hypothesis a bit more plausible. Anyway when you do get a reply from the servers, does that reply imply that V19 should still be available under certain circumstances? Just having V16 doesn't seem to be enough.

Yes. You can test this yourself if you have Python 3 and the 'requests' module by running my script here: https://gist.github.com/Treeki/823ce5e6fafb...dfd61abe2ef557e

Replace the 'softver' field by Cosmo-9.0-Planet-12062019-V16.<20191206_1615> and enter any string into the 'phone_id' field. This is what I get:

Code: [Select]

$ python3 checker_gist.py
{'state': 1, 'patch': {'rstyle': 1, 'dl_type': 0, 'patch_url': 'https://flare02.iofota.com/EASTAEON_FTPRO16945_200118213137.zip', 'patch_version': 'Cosmo-9.0-Planet-01182020-V19', 'patch_name': 'EASTAEON_FTPRO16945_200118213137', 'patch_date': '2020-01-18 21:31:37', 'content_md5': '779a26993a13b8596c7c6368c6f96970', 'patch_intro': '', 'min_mem': 0, 'archive_type': 'zip', 'inst_type': 0, 'inst_time': '', 'mem': 0, 'path2': '', 'patch_id': 19561, 'patch_type': 1, 'app_style': 'S', 'app_id': 20107, 'patch_size': 79845, 'wifi': 1, 'path1': 'data/media/0'}, 'gen': {'project_id': 'FTPRO16945', 'count_succ': 1, 'phone_id': 'dummy string', 'is_gdpr': 1, 'state_device': 0, 'interval_hour': 0, 'path1': 'data/media/0'}}
On the other hand, with the V15 version id (as in the gist) you do not get any updates offered.

[Daniel W]

You can test this yourself if you have Python 3 and the 'requests' module...

Getting comfortable with Python, would require more time than I can spend right now. In Wireless update, my Cosmo shows Cosmo-9.0-Planet-12062019-V16.<20191206_1615> so it is on V16, yet seems unable to ask the right question, as it finds no update. Is it my IP range (45.150.160.0 - 163.255) or is this V19 update for another PCB revision, or what? If I enter https://flare02.iofota.com/EASTAEON_FTPRO16...00118213137.zip into a web browser, it downloads what seems to be to the correct update, since your query presented itself as a V16 Cosmo and the reply explicitly says V19, but I'm not sure installing it is advisable, since Planet urges us to NOT update by other means than FOTA (but didn't say so until after I had manually upgraded to V16).

I'm a bit concerned that Planet says "this will be dealt with top priority" but not until "early next week when people will come back to work from their Chinese New Year holidays." and absolves themselves with "Unfortunately, until then this issue is completely outside our control." instead of owning their mess by immediately putting clearly labelled files and instructions on a server they control. Also, as fas as I understand https://chinesenewyear.net CNY 2020 does NOT end next week, but on Saturday, February 8, and past experience makes me doubt things will just resolve on the Monday after. Their view on the urgency of "top priority" seems very different from mine.

[Edit Wednesday, February 12]:This time, Planet managed to accomplish what they said, so credit where it's due. No later than Tuesday, February 4, according to this post: www.oesf.org/index.php?act=findpost&pid=294567, the update server was responding again (and I successfully updated my firmware the day after).