Outage - Server issue 25th February 2021 (Resolved: New Firmware Release)

Ciaran

UPDATE 25/02/21:

Hi all, 

As some of you have reported, we have experienced a repeat of the server issue impacting a limited amount of Fingbox’s. 

We completely appreciate that if you are a user affected, the fact that it is a minority of Fingboxs impacted is irrelevant. Our engineering team are once again investigating this issue. We completely understand the frustration due to this repeated issue & sincerely apologise for any inconvenience. 

UPDATE: We have released a new Fingbox firmware version 1.15.1 which has fixes from backend and this would probably resolve the issue.

Thank you for your feedback and reporting this issue.

Fing Team

_________________________________________________________________________________________

UPDATE 15/02/21:

We are currently experiencing minor server issues that are impacting a very small number of Fingbox users.

Our engineering team is investigating and trying to understand the root cause of the issue. We are working to fix this asap.

This situation is regrettable and we sincerely apologize for any inconvenience caused.

---------------------------------------------------------------------------------------------------------------------------

UPDATE 11/02/21:   we are doing a maintenance on the AWS DataBase to improve performances.
Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.

____________________________________________________________________________________________________________

UPDATED: Fixed

Hi All,

From approximately 23:30 CET (9th Feb), we experienced an outage on our servers affecting some Fing services (e.g. account log in issue & Fingbox connectivity).  

Root cause: server degradation issue with AWS (Amazon Web Services) that impacted some of our services.

Our engineering team has taken action to resolve and the situation was completely resolved at 09:45 CET, 10th Feb.

This situation is regrettable and we sincerely apologise for any inconvenience caused.

Thank you for your feedback and reporting this issue.

Fing Team

PeterP

My fingbox is unreachable from the app  on my phone and I get a 504 Gateway timeout trying to access app.fing.com.  This is seem just like the outage back in December.

Marc

Hey @PeterP, I know of at least two people on the forums, myself included, who are experiencing this as well.  Desktop, Fingbox and mobile app affected.  Tagging @Robin and @Ciaran for visibility.

rooted

I also have been experiencing the same issue all day.

rooted

Confirmed, the issue is resolved here.

kgrier

Guys. U really need a way to report status outside of the user early warning system and fingboxes warp-core breaching orange alert. Sign up for statuspage.io and link an API call in. Or publish an AWS cloudwatch dashboard. Just not that hard. Really. Heck UptimeRobot is free. I pay for the Pro plan and I'm a retired IT dude helping friends out. Just a poor showing here folks. 

Ciaran

Hi @kgrier, thanks for this feedback. They are really constructive suggestions and we have given these to our Product & Engineering team for consideration. I sincerely apologise for the inconvenience & thanks once again for the feedback.

InventiveSole

Given that it took nearly 9 hours to recognise that most of the FingBox ecosystem had died, what assurances can you offer that you actively monitor your systems for vulnerability exploitation?

Ciaran

Hi @InventiveSole, to confirm, the issue was identified within 1 hour. Our engineers were investigation & consultation to find the root cause and resolve. I do appreciate the frustration & our Engineers are working to prevent this from reoccurring. Thanks again for the feedback.

InventiveSole

Where the issue was identified in an hour, when was it shared with those affected? I only noticed a tweet ~9hrs in, (via Google search) and couldn't find anything on your main website or the community forums... I may well be looking in the wrong place. It's not the end of the world, but I did waste an 1 or so hour trying to troubleshoot (and sure many others did the same). Perhaps worth adding 'back-end services failure' to the list of Amber warning causes in the Troubleshooting guide?
Thanks for restoring service, good to have Fing back👍.

Horseman

OK this may be a tad OTT as I have been sonimpressed by original V1 FB( original £49 offering)but decided to stress test this further between several IP ‘s from my same ISP ( trialling different Hub/gateways) and subsequently bought an additional V2 FB(£99) to test against/via both IP Hub/gateways....
Circa 9Feb21 having swapped between both FB’s several times W/O resetting either ultimately both were red lighted ...... and I lost most of my Lan connectivity!
Attempting to follow Robin’s  “reset” procedure with same/similar variations from his recent posting...   it has also failed and I’ve temporarily had to remove Router<>FB v2 connection to restore Internet connectivity to my Trial Hub/Gateway.

However I suspect this is more “pilot error” on my part but I’m too lazy to preemptively diagnose further...   ..... until publically “shamed” via forum admin to PM/DM the inevitable private access details....

In anticipation of an expected and deserved reprimand/critique ....... but never-the-less appreciating any constructive pointers/hints...
Rgds Tony. 

Ciaran

Hi @InventiveSole, you are correct regarding the communication. This was largely due to the timing when this occurred. As you rightly point out however, this does highlight the need for an auto update (e.g. advising of outage), similar to @kgrier 's suggestion. We have given this feedback/suggestion to our Product & Engineering teams. Thanks again and sorry for the inconvenience.

PeterP

Given that this is the second outage in just over two months, you clearly need to step up your game in regards to notifying users. At a minimum, a notice should have been posted here as this is the first place many of us look when experiencing a problem.  In addition to that, an email blast to all registered fingbox users would be very beneficial.

Shooter

I agree wholeheartedly with the need for implementation of an early warning system in the wake of these system-wide outages. Considering that I've been trying to determine if my Samsung tablet was recently infected with malware, or possibly hacked, not being notified why the FingBox went offline, unfortunately exacerbated my already too high stress level! I'm confident we'll see something soon. 

MDavide

Same problem here but not fixed yet. 10:35 am UTC+1

Ciaran

UPDATE:   we are doing a maintenance on the AWS DataBase to improve performances.
Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.

rooted

https://community.fing.com/discussion/comment/25507#Comment_25507

I don't understand your post?

rooted

It's hard to determine the meaning of what you initially said so yes it clears it up.

What issues are you currently having?

rooted

You should not be receiving ads since you have a Fingbox, I'm guessing when the app can't detect the Fingbox it falls back into non-paid mode.

Is this correct @Ciaran ?

Is it a v1 or v2, just curious since I only have the v2 and it has been fine since yesterday.

OU812

My issue resolved yesterday but failed again over night - Fing box unreachable. Is there another server outage?

Nexx

I am still having same issue . Flashing orange ring. Tried rebooting and following steps from the  Outage chat but My fingbox is already in home location. Only "remove from home" and "deactivate" appears for me.  What should I do at this point?

Shooter

My FingBox was offline again for several more hours from late evening (2/11) to early morning (2/12 - US/DST). Strangely though, when I signed in to the forum to get an outage status update, my FB immediately came back online. Coincidence, or did signing-in somehow bring it back online? 

rooted

@Shooter I'm going out on a limb and saying coincidence, it was up and down.

rourman

Just curious - is anyone trying to capture data on whose FingBox was having issues? Mine was showing the flashing-orange all night and into the next morning (tried rebooting with no success), but not sure if chiming in here now (two days late) is helpful. Is someone collecting statistics?

OU812

I don’t think logging in resolves the issue as it didn’t for me last night. Starting working again this morning which was strange. As usual no communication from Fing so not sure if server issue their side.

Browser Fing site said connected but no home network. Logged into app and home network was set up.

strange

OU812

V1 btw

OU812

Last thing, in app there was an error on the account setting which was resolved by logging in and out.

Robin

Hi All,
To update: Our Engineering team have confirmed that following the server issue on the 9th/10th Feb, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
If you are still facing issue then DM me the MAC address of the Fingbox. Thanks

OU812

Could we not have a status flag on the app or community so users aren’t wasting time trying to fix issues that they think are local network related?

AviBob

Many of us spend time trouble shooting, only to find out way later it 9n your end. Please figure out a way to let us know sooner

Robin

nozero said:

To update: Our Engineering team have confirmed that following the server issue on the 9th\/10th Feb<\/b>, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
If you are still facing issue then DM me the MAC address of the Fingbox. Thanks
","bodyRaw":"Hi All,
To update: Our Engineering team have confirmed that following the server issue on the&nbsp;9th\/10th Feb<\/b>, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
If you are still facing issue then DM me the MAC address of the Fingbox. Thanks
","format":"wysiwyg","dateInserted":"2021-02-12T10:57:11+00:00","insertUser":{"userID":21,"name":"Robin","url":"https:\/\/community.fing.com\/profile\/Robin","photoUrl":"https:\/\/us.v-cdn.net\/6031733\/uploads\/userpics\/022\/nHQS48ATSNJKZ.jpg","dateLastActive":"2021-02-12T17:11:12+00:00","label":"admin"},"displayOptions":{"showUserLabel":false,"showCompactUserInfo":true,"showDiscussionLink":false,"showPostLink":false,"showCategoryLink":false,"renderFullContent":false,"expandByDefault":false},"url":"https:\/\/community.fing.com\/discussion\/comment\/25551#Comment_25551","embedType":"quote"}"> https://community.fing.com/discussion/comment/25551#Comment_25551

I sent my Fingbox mac to you via DM, but received no response. What am I to understand? Did you get it?

Thanks, I have received your MAC address and our usual response time is between 12-24 hours. As I can see your Fingbox has been reset and you need to add your Fingbox to your Fing app again. I have sent you some steps to add Fingbox again to your network if you are unable to do so.