We hope, of course, to be able to identify the root cause. If not, we hope at least that all these mitigations and tunings in place might make the server/service hold the ground so that users won't be affected.
Fingbox disconnects sporadically
Answers
-
Same here over the last few days. Fing v1 has been fine for months, and now suddenly, out of the blue it starts flip/flopping with discovery missing message OFFLINE and ONLINE .... very annoying ! No change in the network (physically or config wise). Is Fing v1 having issues now?
2 -
About 60- 70 notifications of disconnecting last night, shows on my watch so woke me up a lot. I will disable all these notifications.
this is really unacceptable performance. Does it relate to the change of ownership of Fing ? It used to be nice and stable can we have a time to fix estimate please. Really disappointed with current service I’m afraid.
1 -
Even though turning notifications for the Fing Box on the Fing Box do not work, you can turn off the notifications on your phone/watch.Duff556 said:About 60- 70 notifications of disconnecting last night, shows on my watch so woke me up a lot. I will disable all these notifications.
this is really unacceptable performance. Does it relate to the change of ownership of Fing ? It used to be nice and stable can we have a time to fix estimate please. Really disappointed with current service I’m afraid.
Hopefully a fix will be available soon and when they do, they fix the delays in notfications. I used to get notified immediately in the past for an event, now it can be 10, 20 minutes later.
2 -
Seems Fingbox going offline when n nothing is wrong is common problem now. Many similar posts, see also
0 -
This seems like a much wider issue based on all the comments and other posts and it's terrible that Fing isn't addressing or acknowledging this. I'm canceling my subscriptions until this is sorted and if there isn't a statement (at least) by the end of the Month I'm binning the Fingbox.
1 -
This is what I can say after weeks troubling around.... right notifications in time are the most important way to have the feeling of a save or good observed network. It seems that Fing is in a deep trouble and I hope they can solve these problems. It doesn’t make sense to me to proceed with all notifications off. After more than 3 Years working with fing this is a very critical situation.
2 -
It is a coincidence, I have been having the same issue and had a period of several good days as well but also got hammered with notifications last night and I haven't touched my Fing. So no point in doing anything on our end - if the problem is with the Fing servers then resetting our devices isnt going to fix it. A real shame that they haven't solved the problem after all this time.PaVink said:Just adding my frustration to the list. Fingbox essentially unusable the last weeks. Did a factory reset and had few normal days again (could be coincidence), but now back to 20 online/offline messages per day. Not good. Not good at all.0 -
The user and all related content has been deleted.1
-
I too have been plagued by this problem for weeks now and having received 47 offline alerts just over the last 24 hours I have come to the conclusion the Fingbox service is no longer reliable and the end is not in sight. I assume when the discovery is down, my Network is NOT secure and that is not acceptable. I am not in agreement the statement "there is nothing you can do" and I am actively looking for an alternative to Fingbox with a firewall device that is more dependable. I have had my Fingbox since the early release but with no official response I am left to believe Fingbox has abandoned their customer base.
1 -
I have two Fingboxes v1 (one in the US and one in Europe). They have worked flawlessly since installed but recently the one located in the US is experiencing the same offline/online issues as stated above. It happens up to 40 times per 24 hrs. This is VERY ANNOYING and prevents me from relying on Fing. None of my other connected devices experience similar connection issues. And the Fingbox in Europe hasn't experienced a single outage, while the one in the US has seen hundreds. I have also seen the performance of the app severely impacted at the same time. If this continues I will sadly have to seek alternative ways to manage my network.
IF there are issues with the Fing service itself, I would REALLY like to know this. I spent hours trying to troubleshoot this locally already.2 -
Thanks for the reply - we're all waiting here for a solution! :-)Ciaran said:Hi All,
Once again, we sincerely apologise for the inconvenience caused to you. To confirm, we are experiencing an issue whereby a number of Fingbox's are experiencing disconnections from one of our servers. This is not affecting all users, however we are committed to finding the root cause and resolving the issue. To date, this has been a challenge. Our engineering team continue to investigate the issue.
After reading through a number of posts, I can see that there is concern about the future of Fing, and concerns that this may be related to the recent acquisition with Lansweeper. I assure you that the future of Fing is secure and there has been no changes to our services or commitment to our users relating to the acquisition by Lansweeper. This re-occurring issue is unfortunate, and once again, we do apologise for the inconvenience caused.
As soon as we have an update, we will keep you all informed here.
Thank you,
The Fing Team
(I do find it odd that a group of networking specialists has (had) no knowledge of this networking problem, cannot reproduce it (?), and cannot understand what could possibly be the cause for this (networking) problem ... But I am sure you'll crack it.)0 -
I am sure we wish you all "Good Luck" for solving this situation which is really concerning to a lot of us....
0 -
If that's the only support the community can provide ... that's a stunning answer which does not answer my initial question.Shooter said:
You will not get an honest, fully transparent answer from anyone representing Fing's interests.discovery said:Has this been reported to Fing? Is the Fing support engaged on this issue? Tried to reach out but does not seem easy, I'm always redirected to the community forum ... huh? Did someone succeed? Can one confirm? What's the outcome? Thanks,
I would suggest that you disconnect your FingBox for now, then come back here whenever you can. If you keep it connected, you should be sure that all of your most sensitive data is protected by assuming that Fing, Fingboxes, and Fing's servers may have been hacked...just to be safe! Better safe than sorry!
Is Fing aware? Did someone got in touch with Fing support?0 -
This is my main issue with the whole situation - the lack of transparency. Prior to me creating a post yesterday and firing off a number of comments on existing threads, which seems to be the catalyst that rallied everyone yesterday, and actually got you guys to start looking into this as an issue, I saw a good number of open issues that when investigated were all the same. Fing were not admitting fault and just fobbing everyone off with the standard change the cables and turn it off and on (the irony).
This lack of transparency even went to the point that I had previously raised this issue twice before and unknown to me, this was silently merged into other tickets. Whilst I don't mind the merging, there were no comms about this, and to an end-user (me), it looked like things had just deleted as there was no longer a record of these open discussions under my account.
As a Senior IT professional (almost 10 years) and Certified Cloud Architect (AWS), I can tell you that this lack of transparency wouldn't float anywhere else.0 -
Allow me to provide some update from the Team and some context, along with the apologies for all the alerts and inconveniences caused by these outages.SOME CONTEXTFing architecture has dedicated servers (shards) on AWS for a group of Fingboxes. They get assigned when it's activated and help keep the system scalable. Fingboxes connected to these, and expect to maintain a connection. If something goes wrong, the Fingbox detects it and self-protects by disconnecting a retrying to reconnect at random time, to prevent the so called "wave" effect of groups of devices reconnecting at the same time.WHAT IS HAPPENINGRecently, one such server started misbehaving. All the communication is managed by timeouts to sustain the connection for flexibility, but after timeouts expire the connections are dropped and retried. That means also that a blocked server can cause accumulation of requests when you have thousands of Fingbox users trying to use it, and that further fills the queues for all users, which are in the millions. This ripple effect caused also the web app and recognition services to be unable to work, all waiting for a time slot that wouldn't arrive.The first mitigation in place was to revisit ALL timeouts in the infrastructure to make sure request wouldn't hold too long in the queues and cause a massive effect, although we didn't figure out the root cause yet due to the large amount of noise caused by the ripple effect.That helped when the issue represented, as this time it didn't affect the whole system but only the specific users on that server. Now that the noise has been removed the investigation would come around the specific server that has caused the issue.WHAT HAVE FING TRIED TO FIXSo far, we tried quite a lot of alternatives. Sometimes AWS hardware also degrades without warning. Usually AWS replaces it and moves transparently, but it wouldn't be the first time that we had to it ourselves. The re-start of the service doesn't fix but a physical reboot of the VM does, pointing out to something strange in the machine.These are just a few attempts we took
- Monitoring all connection, sockets, actions being executed
- More counters have been added to the dashboard, longer log periods
- We migrated the SSD storage to a new copy. No change.
- We checked the full setup of the server vs other servers. They are identical
- We recreated a new machine from scratch and transferred everything. No change
- We checked in the external systems (Google Assistant, IFTTT) using our APIs have changed their procedure or are abnormally present on just one server. They are not.
A few example KPIs we're monitoring
WHY IS THIS HAPPENING NOW- We are now verifying if the root cause lies inside some specific Fingbox or network they monitor, tracing which Fingbox disconnects before the issue is raised.
- The internal queues are being tuned to avoid larger accumulation of tasks which may further bring the system down, to avoid side-effects of too many communications to be sent out (especially push notifications).
We hope, of course, to be able to identify the root cause. If not, we hope at least that all these mitigations and tunings in place might make the server/service hold the ground so that users won't be affected.IS THIS INTENTIONAL?We noticed how a few Fingbox users are questioning whether these outages are caused by carelessness or downright fabricated. I can assure that the system alerts that the outages generate are very real, and they take people away from their planned work time, sleep and family time, as they tend to happen at the most uncommon hour. Being a team, a few people usually get added to the loop and involved to inspect a problem that gets harder to investigate when it gets cold. Quickly, a large part of the team is absorbed into troubleshooting, and every time this happens we spend hours combing through server logs, queue size charts, timeouts to figure out what may be the root cause.The simple reaction of users putting our alert emails into spam caused our email sending rights to be revoked, as AWS believed we turned into a spamming system overnight due to the spike of alert emails. We had to file evidence of the incident to their support board just to gain back the ability to send emails to our users. It's hard to believe we would self-inflict such a procedure.
Again, we are sorry that this has disrupted your year-long experience with Fingbox, and we work to find the solution soon and get back to creating new and improved features which is what we love to do most.
Marco
50 -
Marco, really appreciate such a detailed response with all the technical information included. I know that post took a long time to write - thank you!
5 -
I really appreciate your explanation I can understandt it is really hard to find a solution in time good luck and thanks for the information
1 -
The user and all related content has been deleted.2
-
I can offer my observations as a gen 1 owner. My unit began this offline online pattern a few weeks ago and just continued to worsen. I also noticed I could not update things using the app, like edit preferences and such. I deleted and reinstalled the app: no help. Then rebooted everything including routers: no help. It began to look like file corruption or permission issues somewhere, so I did a hard reset and let it reload the software. This resolution worked. It may be that corrupted file permissions on multiple aging fingboxes are balling up the servers.
3 -
I went through the whole process with my gen 1 Fing Box which included a hardware reset but none of that worked for me.MrClaude said:I can offer my observations as a gen 1 owner. My unit began this offline online pattern a few weeks ago and just continued to worsen. I also noticed I could not update things using the app, like edit preferences and such. I deleted and reinstalled the app: no help. Then rebooted everything including routers: no help. It began to look like file corruption or permission issues somewhere, so I did a hard reset and let it reload the software. This resolution worked. It may be that corrupted file permissions on multiple aging fingboxes are balling up the servers.
I will wait to see if Fing comes up with a solution because I have already wasted a huge amount of my time attempting to problem solve this.1
Categories
- 5.8K All Categories
- 2.8K Ask about Connected Technology
- How To...
- 1.1K Devices & Security
- 1.6K Network Troubleshooting & Connectivity
- 114 General Discussion, Weird & Wonderful
- 45 Network Infrastructure
- 5.4K Ask about Fing
- 548 Fing Account Change Request
- 1.1K Fing Desktop
- 1.4K Fing App
- 1.7K Fingbox
- 534 Announcements, Beta Testing & Release Notes
- 160 Community Updates
- 29 Getting started
- 13 Community User Guides