Summary

On October 8, 2026, bfhosting-05.com in New York experienced a storage-access failure that disrupted hosted services.

Hyperping recorded the incident from 00:52 to 03:44 UTC−3, lasting approximately 2 hours and 52 minutes. Additional recovery work continued afterward to restore SSH access and full storage redundancy.

We know our customers rely on these services for their businesses. We’re sorry for the disruption and the uncertainty this caused.

Impact

Websites and account-management services on the affected server experienced errors or became unavailable. SSH access was also affected and required a separate recovery step.

The monitoring recovery time reflects the service checked by Hyperping; it does not establish that every customer-facing function recovered at that exact moment.

Root Cause

The immediate cause was loss of access to both NVMe storage devices. Linux reported both controllers as unavailable, resulting in disk I/O errors and interruption of filesystem writes.

Earlier logs show repeated access failures involving one drive before the full outage. This indicates that the storage problem began before services became unavailable.

The underlying cause remains under investigation. Both drives became accessible again after reboot and currently report no media-integrity errors. Their installed firmware predates manufacturer updates addressing intermittent drive detection and read stability, making firmware a possible contributing factor. This has not been confirmed as the cause.

Resolution

We used the server’s remote console to investigate and recover access. Once Linux could access the drives again, we restored affected services and rebuilt the missing RAID members.

All three RAID arrays, system data, boot, and swap—now have both members active. SSH access was restored by restarting the login-session service after identifying session-creation timeouts.

During the outage, we also began restoring affected accounts onto other servers from available backups to reduce customer downtime.

Lessons Learned

What went well

Remote console access allowed recovery work to continue when SSH was unavailable. Available account backups provided an alternative restoration path. Both storage devices became accessible again, allowing us to restore the server’s mirrored storage.

What could be improved

Our monitoring did not provide an actionable alert about the earlier drive-access failures. Service-availability checks alone were insufficient to warn us that storage redundancy was at risk.

The incident also exposed gaps in our overnight hardware-escalation process and in recovery checks for services that can appear active while failing to handle requests.

Action Items

1. Configure and test Zabbix alerts for RAID degradation, missing arrays, unavailable NVMe controllers, and failed storage-health checks.
2. Verify notification delivery and escalation so storage warnings receive attention before customer services fail.
3. Coordinate a firmware review and update plan with the datacenter after verifying fresh backups.
4. Continue investigating the original controller failures and assess replacement hardware.
5. Enable persistent system journals and evaluate off-server logging to preserve evidence during storage failures.
6. Expand recovery checks to cover actual SSH login and application functionality, alongside service and RAID status.
7. Verify restored accounts and reconcile any newer data with migrated copies before retiring the original server.

Service access and RAID redundancy have been restored. The investigation and preventive work remain open, and we will update this report if the underlying cause is confirmed.