Outage - Server issue ONGOING - Dev team investigating

CiaranCiaran Administrator Posts: 1,168
1000 Comments 250 Likes 50 Answers 100 Awesomes
admin

UPDATE 27/03/21:

Description of issue:

In recent weeks we have experienced an issue whereby a limited amount of Fingbox units have briefly, but repeatedly lost temporary connection with our servers. Our Engineering team are investigating this issue to establish root cause and apply a fix. They have provided the following update on the issue & steps taken to date:


Fing architecture has dedicated servers (shards) on AWS for a group of Fingboxes. They get assigned when it's activated and help keep the system scalable. If something goes wrong, the Fingbox detects it and self-protects by disconnecting a retrying to reconnect at random time, to prevent the so called "wave" effect of groups of devices reconnecting at the same time.


Recently, one such server started misbehaving. All the communication is managed by timeouts to sustain the connection for flexibility, but after timeouts expire the connections are dropped and retried. That means also that a blocked server can cause accumulation of requests when you have thousands of Fingbox users trying to use it, and that further fills the queues for all users, which are in the millions. This ripple effect caused also the web app and recognition services to be unable to work, all waiting for a time slot that wouldn't arrive.


Steps we have taken to date:

  • The first mitigation in place was to revisit all timeouts in the infrastructure to make sure request wouldn't hold too long in the queues. Although we didn't figure out the root cause yet due to the large amount of noise caused by the ripple effect.
  • When the issue happened again, it didn't affect the whole system but only the specific users on that server and investigation would come around the specific server that has caused the issue.
  • The re-start of the service doesn't fix but a physical reboot of the Virtual Machine (VM) does, pointing out to something strange in the machine.
  • We are now verifying if the root cause lies inside some specific Fingbox or network they monitor, tracing which Fingbox disconnects before the issue is raised.
  • The internal queues are being tuned to avoid larger accumulation of tasks which may further bring the system down, to avoid side-effects of too many communications to be sent out (especially push notifications).


We hope, of course, to be able to identify the root cause sooner rather than later. If not, we hope at least that all these mitigations and tunings in place might make the server/service hold the ground so that users won't be affected. 


How can I stop the repeated 'online/offline' messages?

Regarding repeated offline notifications, users can adjust notification settings on your Fing Account or at mobile level in the Notification Settings of ioS/Android (it may save you unplugging the unit or being woken up at night by notifications). I just want to re-enforce, I am not proposing this as a solution to the overall problem, but merely to address the repeated notifications until the overall issue is sorted.


Once again, apologies for the inconvenience caused to date and rest assured that we are committed to resolving this issue. 


UPDATE 25/02/21:

Hi all, 

As some of you have reported, we have experienced a repeat of the server issue impacting a limited amount of Fingbox’s. 

We completely appreciate that if you are a user affected, the fact that it is a minority of Fingboxs impacted is irrelevant. Our engineering team are once again investigating this issue. We completely understand the frustration due to this repeated issue & sincerely apologise for any inconvenience. 

UPDATE: We have released a new Fingbox firmware version 1.15.1 which has fixes from backend and this would probably resolve the issue.

Thank you for your feedback and reporting this issue.

Fing Team

_________________________________________________________________________________________

UPDATE 15/02/21:

We are currently experiencing minor server issues that are impacting a very small number of Fingbox users.

Our engineering team is investigating and trying to understand the root cause of the issue. We are working to fix this asap.

This situation is regrettable and we sincerely apologize for any inconvenience caused.

---------------------------------------------------------------------------------------------------------------------------

UPDATE 11/02/21:   we are doing a maintenance on the AWS DataBase to improve performances.
Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.

____________________________________________________________________________________________________________

UPDATED: Fixed

Hi All,

From approximately 23:30 CET (9th Feb), we experienced an outage on our servers affecting some Fing services (e.g. account log in issue & Fingbox connectivity).  

Root cause: server degradation issue with AWS (Amazon Web Services) that impacted some of our services.

Our engineering team has taken action to resolve and the situation was completely resolved at 09:45 CET, 10th Feb.

This situation is regrettable and we sincerely apologise for any inconvenience caused.

Thank you for your feedback and reporting this issue.

Fing Team

Ciaran (Admin at Fing)
Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
CiaranRobinrooted
«13456732

Comments

  • mikea9999mikea9999 Member, Beta Tester Posts: 3
    First Comment Photogenic
    After a reapplying power, my Fingbox  will only stay online for ~20 minutes and then go offline.  Any ideas on how to resolve?  
    Thank you!
    Mike
  • VioletChepilVioletChepil London, UKMember Posts: 2,471
    100 Answers 500 Likes 1000 Comments 100 Agrees
    ✭✭✭✭✭✭
    Hello @mikea9999
    Could you try replacing the power cable? Sometimes, this can help solve the issue. 
    What color are the Fingbox LEDs displaying?
    Have you tried a hard reset of the device?
    Is it only Fingbox going offline, or is the rest of the network going down too? 

    Community Manager at Fing

  • mikea9999mikea9999 Member, Beta Tester Posts: 3
    First Comment Photogenic
    Orange lights.  I have not tried a hard reset.  Everything else on the network is up and running.
    VioletChepil
  • mikea9999mikea9999 Member, Beta Tester Posts: 3
    First Comment Photogenic
    What is the process for a hard reset?

    VioletChepil
  • VioletChepilVioletChepil London, UKMember Posts: 2,471
    100 Answers 500 Likes 1000 Comments 100 Agrees
    ✭✭✭✭✭✭
    Accepted Answer
    Ok, maybe its stuck trying to reach Fing servers for an auto update.
    Could we try the hard reset?
    Here are the full steps to go through! It's important to complete them all. 
    Reset Fingbox
    1. Unplug the Fingbox power adaptor from the wall outlet
    2. Find the RESET hole beside the Fingbox power port
    3. Gently insert a paperclip in the RESET hole (very gently)
    4. While keeping the RESET button gently pressed, plug the power adaptor back in
    5. Keep the RESET button pressed for 30 seconds and release (or until a single white light flashes on the device)
    6. You should now see a single white light on
    7. After a few seconds, the top circle will start flashing, while the original factory software is loaded
    8. This process lasts for a few minutes (~3 min)
    9. A single white light will mark the end of the process
    10. Power off the Fingbox
    11. Wait 5 seconds
    12. Power it back on
    Please let me know if anything changes.

    Community Manager at Fing

  • mikea9999mikea9999 Member, Beta Tester Posts: 3
    First Comment Photogenic
    I did the hard reset and have not gone offline.  Thank you,
    VioletChepil
  • VioletChepilVioletChepil London, UKMember Posts: 2,471
    100 Answers 500 Likes 1000 Comments 100 Agrees
    ✭✭✭✭✭✭
    Hi @mikea9999
    To do a hard reset:
    Here are the full steps to go through!
    Reset Fingbox
    1. Unplug the Fingbox power adaptor from the wall outlet
    2. Find the RESET hole beside the Fingbox power port
    3. Gently insert a paperclip in the RESET hole (very gently)
    4. While keeping the RESET button gently pressed, plug the power adaptor back in
    5. Keep the RESET button pressed for 30 seconds and release (or until a single white light flashes on the device)
    6. You should now see a single white light on
    7. After a few seconds, the top circle will start flashing, while the original factory software is loaded
    8. This process lasts for a few minutes (~3 min)
    9. A single white light will mark the end of the process
    10. Power off the Fingbox
    11. Wait 5 seconds
    12. Power it back on
    Please let me know if anything changes.

    Also are things working Ok for you?
    Let me know. 

    Community Manager at Fing

  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic

    My fing box goes offline 3 to 4 times a day and i dont know why . I have it for ups along with my cordless phone and WiFi router. There has been no problem for last 2 years but now it's going offline on a regular basis. I have posted this question earlier but as these fourms are not on a app find it very hard to check on answers. The Facebook fourm allowed me to engage more

  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭

    If you click on your name scroll down the page you will see a link to your comments and discussions.

    You should have a private message, click your icon on the top right of the page then click the icon that looks like an envelope.

    Here is your previous post:


    Amolakh
  • CiaranCiaran Administrator Posts: 1,168
    1000 Comments 250 Likes 50 Answers 100 Awesomes
    admin
    Thanks @Amolakh. I am sending you a private mail to investigate further (we will need some specific information such as MAC address, e-mail etc.). Thanks
    Ciaran (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
    Amolakh
  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic
    I assume you mean you are sending it on this platform or as an email? in my inbox here I see nothing from you only from Voilet
  • RobinRobin Administrator, Fing Team Posts: 4,097
    250 Answers 2500 Comments 500 Likes 100 Awesomes
    admin
    Accepted Answer
    Amolakh said:
    I assume you mean you are sending it on this platform or as an email? in my inbox here I see nothing from you only from Voilet
    We have sent you a personal email on your registered email address. If you are unable to find, can you check under Spam? Thanks 
    Robin (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic
    can some one help please 
  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic
    I have responded to your email. Please let me know next steps
  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment
    Starting a month or two ago, by fingbox is frequently going off line, I've found that if I just reboot my router without physically touching anything it comes back online but then within 1 to a few days it will go offline again.  It has been running flawlessly for a number of years until, as I said, a month or two ago.  Not sure if the fingbox is dying, this was caused by a SW update to either the fingbox or router, or what.  Any ideas?
  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭

    Do you have a router that supports logging?

  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment
    yes
  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭

    Could you check the logs and see if the Fingbox has been having trouble with obtaining an IP address?

  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment
    It does not show any errors in the logs.  The Fingbox was online at address 192.168.1.10.  The only entry in the log current for either this IP address or the NetAddress of the Fingbox is this line ([DHCP IP: 192.168.1.10][Device Name: ] to MAC address f0:23:b9:eb:3a:8c, Sunday, February 23, 2020 05:03:21) and yet ~ 3hrs and 12 mins later (8:15am) I received a mail that my Fingbox was offline -- and trying to reach it via the app also showed it offline.

  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭

    Could you reboot your router? If that doesn't resolve the issue try rebooting the Fingbox.

    There have been no firmware updates.

  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment

    As I stated in my original post, if I reboot the router, the fingbox comes right back online, for a few hrs to a day or two but then goes offline again — no other devices on my network are going off line, just the fingbox.

  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭

    Sorry I missed that somehow, I'm not sure what it could be if not an IP address conflict.

    Have you tried a different power cable and or network cable?

  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment

    I have not tried those, but can see that the fingbox has power (even when it’s off line) and network connection because I can see the orange network activity light on the fingbox flickering. My guess is that my fingbox is dying if others aren’t seeing this type of issue. I do have another never-used fingbox, I’ll probably install it and see that it does.

  • rootedrooted Gulf Coast, USModerator Posts: 863
    250 Likes 500 Comments 50 Answers 25 Agrees
    ✭✭✭✭✭
    edited February 2020
    My Fingbox has done this and it was showing network lights on the Ethernet jack, it would not connect until I rebooted my router. It has done this twice, since I set a static IP address for the Fingbox I have not had this issue again.

    My Fingbox is only two months old.
    OKC
  • RobinRobin Administrator, Fing Team Posts: 4,097
    250 Answers 2500 Comments 500 Likes 100 Awesomes
    admin
    Hi @GeoJohn23
    Thanks for your post. If you experience issues with network connectivity for Fingbox, you can either increase the network size on Fing App. 

     To do this: 

    • Open the Fing app, ensure that you are on the Network associated with your Fingbox 

    • Tap on the Avatar icon in the top-right corner 

    • Choose 'App Settings' 

    • Scroll down and go to the "Maximum Network Size" setting and select the size based on the number of devices connected to network in your house. 


    Or, you can check if the Ip address assigned to Fingbox is not being used by other Device which disconnects the Fingbox from the connection. I would request you to assign a Static IP address to Fingbox which will help us to isolate the issue. Assigning Static IP address to most devices is easy, just go to its network settings, look for DHCP option and turn it OFF. Once you do that, you'll see a text area to enter Static IP address. Enter the new IP address there, save changes. 

    Robin (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment
    I saw an earlier comment about the Max Network Size and adjust that in the app before I submitted my issue -- this didn't have any impact on my fingbox going off line.  I will try the static IP workaround.  Note, however, that I've been using this fingbox since 2017 (I was a founding supporter on Indiegogo in 2016 and so received a pair of them in one of the first unit shipments) and have not had this offline issue until relatively recently (and its always been on DHCP).  So if going static IP does fix this issue, then I'm suspecting something in a recent software update to either my router or the fingbox has gone wonk.  Will post an update after I've had a chance to go static IP and give the box a few days....
  • GeoJohn23GeoJohn23 Member Posts: 6
    Photogenic First Comment

    Well looks like going to static IP was the ticket, been a few days now and hasn’t gone off line again yet.

  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic

    Hi,

    My fing box is going offline again and again. I had this earlier and posted it here post my post this stopped after you monitored this and fixed something. I had to leave for a few weeks and the net was switched off after coming back this problem has started again. Can any one help me

  • RobinRobin Administrator, Fing Team Posts: 4,097
    250 Answers 2500 Comments 500 Likes 100 Awesomes
    admin
    Hi @Amolakh
     We experienced an issue whereby our servers intermittently had some outages, affecting Fingbox’s contacting our servers recently. The issue was resolved within approx. 5 hours and is now resolved. I believe this may be related to what you have experienced. If the issue continues, please do not hesitate to contact us again."
    Robin (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
  • AmolakhAmolakh Member Posts: 22
    10 Comments First Anniversary Name Dropper Photogenic

    Ok it was happened for last few days and about 5 hours ago the fingbox went down. Again I am putting it up it's holiday in india of it goes down in next 24 hours will paste this again on your wall

    OKC
Sign In or Register to comment.