Ask any devices or smart home question in the Devices/Security or Network Troubleshooting Categories.
Happy posting!
Outage - Server issue RESOLVED
Ciaran
Administrator Posts: 1,180
UPDATE 27/03/21:
Description of issue:
In recent weeks we have experienced an issue whereby a limited amount of Fingbox units have briefly, but repeatedly lost temporary connection with our servers. Our Engineering team are investigating this issue to establish root cause and apply a fix. They have provided the following update on the issue & steps taken to date:
Fing architecture has dedicated servers (shards) on AWS for a group of Fingboxes. They get assigned when it's activated and help keep the system scalable. If something goes wrong, the Fingbox detects it and self-protects by disconnecting a retrying to reconnect at random time, to prevent the so called "wave" effect of groups of devices reconnecting at the same time.
Recently, one such server started misbehaving. All the communication is managed by timeouts to sustain the connection for flexibility, but after timeouts expire the connections are dropped and retried. That means also that a blocked server can cause accumulation of requests when you have thousands of Fingbox users trying to use it, and that further fills the queues for all users, which are in the millions. This ripple effect caused also the web app and recognition services to be unable to work, all waiting for a time slot that wouldn't arrive.
Steps we have taken to date:
- The first mitigation in place was to revisit all timeouts in the infrastructure to make sure request wouldn't hold too long in the queues. Although we didn't figure out the root cause yet due to the large amount of noise caused by the ripple effect.
- When the issue happened again, it didn't affect the whole system but only the specific users on that server and investigation would come around the specific server that has caused the issue.
- The re-start of the service doesn't fix but a physical reboot of the Virtual Machine (VM) does, pointing out to something strange in the machine.
- We are now verifying if the root cause lies inside some specific Fingbox or network they monitor, tracing which Fingbox disconnects before the issue is raised.
- The internal queues are being tuned to avoid larger accumulation of tasks which may further bring the system down, to avoid side-effects of too many communications to be sent out (especially push notifications).
We hope, of course, to be able to identify the root cause sooner rather than later. If not, we hope at least that all these mitigations and tunings in place might make the server/service hold the ground so that users won't be affected.
How can I stop the repeated 'online/offline' messages?
Regarding repeated offline notifications, users can adjust notification settings on your Fing Account or at mobile level in the Notification Settings of ioS/Android (it may save you unplugging the unit or being woken up at night by notifications). I just want to re-enforce, I am not proposing this as a solution to the overall problem, but merely to address the repeated notifications until the overall issue is sorted.
Once again, apologies for the inconvenience caused to date and rest assured that we are committed to resolving this issue.
UPDATE 25/02/21:
Hi all,
As some of you have reported, we have experienced a repeat of the server issue impacting a limited amount of Fingbox’s.
We completely appreciate that if you are a user affected, the fact that it is a minority of Fingboxs impacted is irrelevant. Our engineering team are once again investigating this issue. We completely understand the frustration due to this repeated issue & sincerely apologise for any inconvenience.
UPDATE: We have released a new Fingbox firmware version 1.15.1 which has fixes from backend and this would probably resolve the issue.
Thank you for your feedback and reporting this issue.
Fing Team
_________________________________________________________________________________________
UPDATE 15/02/21:
We are currently experiencing minor server issues that are impacting a very small number of Fingbox users.
Our engineering team is investigating and trying to understand the root cause of the issue. We are working to fix this asap.
This situation is regrettable and we sincerely apologize for any inconvenience caused.
---------------------------------------------------------------------------------------------------------------------------
UPDATE 11/02/21: we are doing a maintenance on the AWS DataBase to improve performances.
Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.
____________________________________________________________________________________________________________
UPDATED: Fixed
Hi All,
From approximately 23:30 CET (9th Feb), we experienced an outage on our servers affecting some Fing services (e.g. account log in issue & Fingbox connectivity).
Root cause: server degradation issue with AWS (Amazon Web Services) that impacted some of our services.
Our engineering team has taken action to resolve and the situation was completely resolved at 09:45 CET, 10th Feb.
This situation is regrettable and we sincerely apologise for any inconvenience caused.
Thank you for your feedback and reporting this issue.
Fing Team
Comments
Thank you!
Mike
Could you try replacing the power cable? Sometimes, this can help solve the issue.
What color are the Fingbox LEDs displaying?
Have you tried a hard reset of the device?
Is it only Fingbox going offline, or is the rest of the network going down too?
Community Manager at Fing
Could we try the hard reset?
Here are the full steps to go through! It's important to complete them all.
Reset Fingbox
- Unplug the Fingbox power adaptor from the wall outlet
- Find the RESET hole beside the Fingbox power port
- Gently insert a paperclip in the RESET hole (very gently)
- While keeping the RESET button gently pressed, plug the power adaptor back in
- Keep the RESET button pressed for 30 seconds and release (or until a single white light flashes on the device)
- You should now see a single white light on
- After a few seconds, the top circle will start flashing, while the original factory software is loaded
- This process lasts for a few minutes (~3 min)
- A single white light will mark the end of the process
- Power off the Fingbox
- Wait 5 seconds
- Power it back on
Please let me know if anything changes.Community Manager at Fing
To do a hard reset:
Here are the full steps to go through!
Reset Fingbox
- Unplug the Fingbox power adaptor from the wall outlet
- Find the RESET hole beside the Fingbox power port
- Gently insert a paperclip in the RESET hole (very gently)
- While keeping the RESET button gently pressed, plug the power adaptor back in
- Keep the RESET button pressed for 30 seconds and release (or until a single white light flashes on the device)
- You should now see a single white light on
- After a few seconds, the top circle will start flashing, while the original factory software is loaded
- This process lasts for a few minutes (~3 min)
- A single white light will mark the end of the process
- Power off the Fingbox
- Wait 5 seconds
- Power it back on
Please let me know if anything changes.Also are things working Ok for you?
Let me know.
Community Manager at Fing
My fing box goes offline 3 to 4 times a day and i dont know why . I have it for ups along with my cordless phone and WiFi router. There has been no problem for last 2 years but now it's going offline on a regular basis. I have posted this question earlier but as these fourms are not on a app find it very hard to check on answers. The Facebook fourm allowed me to engage more
If you click on your name scroll down the page you will see a link to your comments and discussions.
You should have a private message, click your icon on the top right of the page then click the icon that looks like an envelope.
Here is your previous post:
Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
Do you have a router that supports logging?
Could you check the logs and see if the Fingbox has been having trouble with obtaining an IP address?
Could you reboot your router? If that doesn't resolve the issue try rebooting the Fingbox.
There have been no firmware updates.
As I stated in my original post, if I reboot the router, the fingbox comes right back online, for a few hrs to a day or two but then goes offline again — no other devices on my network are going off line, just the fingbox.
Sorry I missed that somehow, I'm not sure what it could be if not an IP address conflict.
Have you tried a different power cable and or network cable?
I have not tried those, but can see that the fingbox has power (even when it’s off line) and network connection because I can see the orange network activity light on the fingbox flickering. My guess is that my fingbox is dying if others aren’t seeing this type of issue. I do have another never-used fingbox, I’ll probably install it and see that it does.
My Fingbox is only two months old.
Thanks for your post. If you experience issues with network connectivity for Fingbox, you can either increase the network size on Fing App.
To do this:
Open the Fing app, ensure that you are on the Network associated with your Fingbox
Tap on the Avatar icon in the top-right corner
Choose 'App Settings'
Scroll down and go to the "Maximum Network Size" setting and select the size based on the number of devices connected to network in your house.
Or, you can check if the Ip address assigned to Fingbox is not being used by other Device which disconnects the Fingbox from the connection. I would request you to assign a Static IP address to Fingbox which will help us to isolate the issue. Assigning Static IP address to most devices is easy, just go to its network settings, look for DHCP option and turn it OFF. Once you do that, you'll see a text area to enter Static IP address. Enter the new IP address there, save changes.
Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
Well looks like going to static IP was the ticket, been a few days now and hasn’t gone off line again yet.
Hi,
My fing box is going offline again and again. I had this earlier and posted it here post my post this stopped after you monitored this and fixed something. I had to leave for a few weeks and the net was switched off after coming back this problem has started again. Can any one help me
We experienced an issue whereby our servers intermittently had some outages, affecting Fingbox’s contacting our servers recently. The issue was resolved within approx. 5 hours and is now resolved. I believe this may be related to what you have experienced. If the issue continues, please do not hesitate to contact us again."
Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
Ok it was happened for last few days and about 5 hours ago the fingbox went down. Again I am putting it up it's holiday in india of it goes down in next 24 hours will paste this again on your wall