Outage - Server issue 25th February 2021 (Resolved: New Firmware Release)

CiaranCiaran Administrator Posts: 1,092
1000 Comments 250 Likes 50 Answers 100 Awesomes
admin

UPDATE 25/02/21:

Hi all, 

As some of you have reported, we have experienced a repeat of the server issue impacting a limited amount of Fingbox’s. 

We completely appreciate that if you are a user affected, the fact that it is a minority of Fingboxs impacted is irrelevant. Our engineering team are once again investigating this issue. We completely understand the frustration due to this repeated issue & sincerely apologise for any inconvenience. 

UPDATE: We have released a new Fingbox firmware version 1.15.1 which has fixes from backend and this would probably resolve the issue.

Thank you for your feedback and reporting this issue.

Fing Team

_________________________________________________________________________________________

UPDATE 15/02/21:

We are currently experiencing minor server issues that are impacting a very small number of Fingbox users.

Our engineering team is investigating and trying to understand the root cause of the issue. We are working to fix this asap.

This situation is regrettable and we sincerely apologize for any inconvenience caused.

---------------------------------------------------------------------------------------------------------------------------

UPDATE 11/02/21:   we are doing a maintenance on the AWS DataBase to improve performances.
Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.

____________________________________________________________________________________________________________

UPDATED: Fixed

Hi All,

From approximately 23:30 CET (9th Feb), we experienced an outage on our servers affecting some Fing services (e.g. account log in issue & Fingbox connectivity).  

Root cause: server degradation issue with AWS (Amazon Web Services) that impacted some of our services.

Our engineering team has taken action to resolve and the situation was completely resolved at 09:45 CET, 10th Feb.

This situation is regrettable and we sincerely apologise for any inconvenience caused.

Thank you for your feedback and reporting this issue.

Fing Team

Ciaran (Admin at Fing)
Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
CiaranRobinrooted
«1345

Comments

  • PeterPPeterP Member, Beta Tester Posts: 56
    25 Agrees 25 Likes 10 Comments First Anniversary
    ✭✭✭
    My fingbox is unreachable from the app  on my phone and I get a 504 Gateway timeout trying to access app.fing.com.  This is seem just like the outage back in December.
    Chuckles_GrUnknown
  • MarcMarc Moderator, Beta Tester Posts: 2,171
    100 Answers 1000 Comments 500 Likes 250 Agrees
    ✭✭✭✭✭✭
    Hey @PeterP, I know of at least two people on the forums, myself included, who are experiencing this as well.  Desktop, Fingbox and mobile app affected.  Tagging @Robin and @Ciaran for visibility.
    Thats Daphnee, she's a good dog...
  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭

    I also have been experiencing the same issue all day.

  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭

    Confirmed, the issue is resolved here.

  • CiaranCiaran Administrator Posts: 1,092
    1000 Comments 250 Likes 50 Answers 100 Awesomes
    admin
    Hi @kgrier, thanks for this feedback. They are really constructive suggestions and we have given these to our Product & Engineering team for consideration. I sincerely apologise for the inconvenience & thanks once again for the feedback.
    Ciaran (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
    AviBob
  • InventiveSoleInventiveSole Member Posts: 3
    First Comment
    Given that it took nearly 9 hours to recognise that most of the FingBox ecosystem had died, what assurances can you offer that you actively monitor your systems for vulnerability exploitation?
    Chuckles_GrnozeroPrac2cul1
  • CiaranCiaran Administrator Posts: 1,092
    1000 Comments 250 Likes 50 Answers 100 Awesomes
    admin
    Hi @InventiveSole, to confirm, the issue was identified within 1 hour. Our engineers were investigation & consultation to find the root cause and resolve. I do appreciate the frustration & our Engineers are working to prevent this from reoccurring. Thanks again for the feedback.
    Ciaran (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
    InventiveSole
  • InventiveSoleInventiveSole Member Posts: 3
    First Comment
    Where the issue was identified in an hour, when was it shared with those affected? I only noticed a tweet ~9hrs in, (via Google search) and couldn't find anything on your main website or the community forums... I may well be looking in the wrong place. It's not the end of the world, but I did waste an 1 or so hour trying to troubleshoot (and sure many others did the same). Perhaps worth adding 'back-end services failure' to the list of Amber warning causes in the Troubleshooting guide?
    Thanks for restoring service, good to have Fing back👍.
    Chuckles_GrFromanPrac2cul1
  • HorsemanHorseman Member, Beta Tester Posts: 0
    First Anniversary Photogenic
    OK this may be a tad OTT as I have been sonimpressed by original V1 FB( original £49 offering)but decided to stress test this further between several IP ‘s from my same ISP ( trialling different Hub/gateways) and subsequently bought an additional V2 FB(£99) to test against/via both IP Hub/gateways....
    Circa 9Feb21 having swapped between both FB’s several times W/O resetting either ultimately both were red lighted ...... and I lost most of my Lan connectivity!
    Attempting to follow Robin’s  “reset” procedure with same/similar variations from his recent posting...   it has also failed and I’ve temporarily had to remove Router<>FB v2 connection to restore Internet connectivity to my Trial Hub/Gateway.

    However I suspect this is more “pilot error” on my part but I’m too lazy to preemptively diagnose further...  :p ..... until publically “shamed” via forum admin to PM/DM the inevitable private access details.... ;)

    In anticipation of an expected and deserved reprimand/critique ....... but never-the-less appreciating any constructive pointers/hints...
    Rgds Tony. 


  • CiaranCiaran Administrator Posts: 1,092
    1000 Comments 250 Likes 50 Answers 100 Awesomes
    admin
    Hi @InventiveSole, you are correct regarding the communication. This was largely due to the timing when this occurred. As you rightly point out however, this does highlight the need for an auto update (e.g. advising of outage), similar to @kgrier 's suggestion. We have given this feedback/suggestion to our Product & Engineering teams. Thanks again and sorry for the inconvenience.
    Ciaran (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
  • PeterPPeterP Member, Beta Tester Posts: 56
    25 Agrees 25 Likes 10 Comments First Anniversary
    ✭✭✭
    edited February 10
    Given that this is the second outage in just over two months, you clearly need to step up your game in regards to notifying users. At a minimum, a notice should have been posted here as this is the first place many of us look when experiencing a problem.  In addition to that, an email blast to all registered fingbox users would be very beneficial.
    HorsemanDiane_MnozeroPrac2cul1
  • ShooterShooter Member Posts: 27
    10 Comments 5 Likes 5 Awesomes First Anniversary
    ✭✭
    I agree wholeheartedly with the need for implementation of an early warning system in the wake of these system-wide outages. Considering that I've been trying to determine if my Samsung tablet was recently infected with malware, or possibly hacked, not being notified why the FingBox went offline, unfortunately exacerbated my already too high stress level! I'm confident we'll see something soon. 
    HorsemanDiane_MnozeroPrac2cul1
  • MDavideMDavide Member Posts: 51
    25 Likes 10 Comments First Anniversary First Answer
    ✭✭✭
    Same problem here but not fixed yet. 10:35 am UTC+1
    Davide
    nozero
  • CiaranCiaran Administrator Posts: 1,092
    1000 Comments 250 Likes 50 Answers 100 Awesomes
    admin
    UPDATE:   we are doing a maintenance on the AWS DataBase to improve performances.
    Between 10:30-11:15 (CET, i.e. now) there could be some little slow down, but all is under control & will return back to normal.
    Ciaran (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
    rooted
  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭
  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭

    It's hard to determine the meaning of what you initially said so yes it clears it up.

    What issues are you currently having?

  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭

    You should not be receiving ads since you have a Fingbox, I'm guessing when the app can't detect the Fingbox it falls back into non-paid mode.

    Is this correct @Ciaran ?

    Is it a v1 or v2, just curious since I only have the v2 and it has been fine since yesterday.

    Robin
  • OU812OU812 Member Posts: 30
    10 Comments 5 Likes 5 Agrees Name Dropper
    ✭✭
    My issue resolved yesterday but failed again over night - Fing box unreachable. Is there another server outage?
  • NexxNexx Member, Beta Tester Posts: 13
    5 Agrees First Anniversary 5 Likes Photogenic
    ✭✭✭
    I am still having same issue . Flashing orange ring. Tried rebooting and following steps from the  Outage chat but My fingbox is already in home location. Only "remove from home" and "deactivate" appears for me.  What should I do at this point?
    OU812
  • ShooterShooter Member Posts: 27
    10 Comments 5 Likes 5 Awesomes First Anniversary
    ✭✭
    My FingBox was offline again for several more hours from late evening (2/11) to early morning (2/12 - US/DST). Strangely though, when I signed in to the forum to get an outage status update, my FB immediately came back online. Coincidence, or did signing-in somehow bring it back online? 
  • rootedrooted Gulf Coast, USModerator Posts: 812
    500 Comments 50 Answers 100 Likes 25 Agrees
    ✭✭✭✭✭

    @Shooter I'm going out on a limb and saying coincidence, it was up and down.

  • rourmanrourman Member, Beta Tester Posts: 6
    First Anniversary Founder First Comment Photogenic
    ✭✭
    Just curious - is anyone trying to capture data on whose FingBox was having issues? Mine was showing the flashing-orange all night and into the next morning (tried rebooting with no success), but not sure if chiming in here now (two days late) is helpful. Is someone collecting statistics?
  • OU812OU812 Member Posts: 30
    10 Comments 5 Likes 5 Agrees Name Dropper
    ✭✭

    I don’t think logging in resolves the issue as it didn’t for me last night. Starting working again this morning which was strange. As usual no communication from Fing so not sure if server issue their side.

    Browser Fing site said connected but no home network. Logged into app and home network was set up.

    strange

  • OU812OU812 Member Posts: 30
    10 Comments 5 Likes 5 Agrees Name Dropper
    ✭✭

    V1 btw

  • OU812OU812 Member Posts: 30
    10 Comments 5 Likes 5 Agrees Name Dropper
    ✭✭

    Last thing, in app there was an error on the account setting which was resolved by logging in and out.

  • RobinRobin Administrator, Fing Team Posts: 3,785
    250 Answers 2500 Comments 500 Likes 100 Awesomes
    admin
    Hi All,
    To update: Our Engineering team have confirmed that following the server issue on the 9th/10th Feb, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
    If you are still facing issue then DM me the MAC address of the Fingbox. Thanks
    Robin (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
    rooted
  • OU812OU812 Member Posts: 30
    10 Comments 5 Likes 5 Agrees Name Dropper
    ✭✭

    Could we not have a status flag on the app or community so users aren’t wasting time trying to fix issues that they think are local network related?

    AviBobnozeroPrac2cul1
  • AviBobAviBob Member Posts: 9
    First Anniversary First Comment Photogenic

    Many of us spend time trouble shooting, only to find out way later it 9n your end. Please figure out a way to let us know sooner

    nozeroPrac2cul1
  • RobinRobin Administrator, Fing Team Posts: 3,785
    250 Answers 2500 Comments 500 Likes 100 Awesomes
    admin
    nozero said:
    To update: Our Engineering team have confirmed that following the server issue on the 9th\/10th Feb<\/b>, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
    If you are still facing issue then DM me the MAC address of the Fingbox. Thanks
    ","bodyRaw":"Hi All,
    To update: Our Engineering team have confirmed that following the server issue on the&nbsp;9th\/10th Feb<\/b>, and the subsequent work completed to resolve, we did experience an issue whereby 10-15% of Fingbox users experienced connected issues this morning. Our engineers have been working on this and it should now be resolved now. Apologies once again for the inconvenience.
    If you are still facing issue then DM me the MAC address of the Fingbox. Thanks
    ","format":"wysiwyg","dateInserted":"2021-02-12T10:57:11+00:00","insertUser":{"userID":21,"name":"Robin","url":"https:\/\/community.fing.com\/profile\/Robin","photoUrl":"https:\/\/us.v-cdn.net\/6031733\/uploads\/userpics\/022\/nHQS48ATSNJKZ.jpg","dateLastActive":"2021-02-12T17:11:12+00:00","label":"admin"},"displayOptions":{"showUserLabel":false,"showCompactUserInfo":true,"showDiscussionLink":false,"showPostLink":false,"showCategoryLink":false,"renderFullContent":false,"expandByDefault":false},"url":"https:\/\/community.fing.com\/discussion\/comment\/25551#Comment_25551","embedType":"quote"}"> https://community.fing.com/discussion/comment/25551#Comment_25551

    I sent my Fingbox mac to you via DM, but received no response. What am I to understand? Did you get it?

    Thanks, I have received your MAC address and our usual response time is between 12-24 hours. As I can see your Fingbox has been reset and you need to add your Fingbox to your Fing app again. I have sent you some steps to add Fingbox again to your network if you are unable to do so. 
    Robin (Admin at Fing)
    Getting Started? Please refer to Community guidelines & Community User Guides("Helping Hand"). HAPPY POSTING!!!
Sign In or Register to comment.