Summarize this blog article with AI:
Ask ChatGPT
Ask Grok
Ask Perplexity
Ask Claude
Ask Google AIKey Takeaways
- Degraded ≠ failed: A drive with I/O latency rising from ~10 ms to 100+ ms can disrupt business continuity even though it hasn't fully failed — a condition that is harder to detect than outright hardware failure.
- Speed matters: Sangfor aSAN (Sangfor's software-defined distributed storage solution) identifies a stalled HDD drive within 13 seconds, compared to vSAN's 120-second default timeout for marking a disk offline.
- Granularity reduces impact: Sangfor aSAN's shard-level isolation avoids taking an entire drive offline over a few inconsistent shards, unlike traditional approaches that isolate at the whole-drive level.
- Software-first design: Sangfor aSAN performs isolation entirely at the application/software layer, without physical drive removal, reducing hardware-compatibility dependencies across vendors.
- Adaptive fault handling: Sangfor aSAN's Distributed Fault Center re-enables replicas that recover naturally and only triggers full isolation and rebuild for persistent or recurring faults, minimizing unnecessary rebuild cycles.
- Proven in production: A manufacturing customer's MES environment saw a stalled-drive incident resolved in 15 seconds after deploying Sangfor aSAN, with multiple degraded-drive events over six months handled while keeping service interruption very short.
- Broader applicability than out-of-band monitoring: Compared to iostat-based competitor solutions, Sangfor aSAN's embedded, per-I/O monitoring provides finer-grained metrics and lower false-positive risk.
As organizations modernize virtualization infrastructure, distributed storage is expected to deliver stable, efficient performance across increasingly complex and diverse hardware environments. Yet a drive does not need to fail completely to disrupt the business. A device that remains online while its I/O response time rises from around 10 ms to more than 100 ms can stall replica writes, reduce cluster throughput, and threaten service continuity.
Sangfor aSAN addresses this challenge with a software-based drive-isolation framework designed to identify degraded components, contain their impact, and protect storage stability without immediately treating every temporary slowdown as a permanent hardware failure.
PART 1. Degraded Drives: The Hidden Threat to Business Stability
Distributed storage systems commonly use strongly consistent, multi-replica write algorithms. A write is acknowledged to the application only after all required replicas have completed the operation. When a drive, host, or another component enters a degraded state, such as when disk I/O response time increases from 10 ms to more than 100 ms, write I/O can stall, significantly affecting application performance and potentially causing large-scale service disruption. The challenge can become more pronounced in heterogeneous hardware environments, where component quality and failure characteristics may vary.
Traditional solutions typically use out-of-band detection to convert a degraded drive into a failed state. Although this can preserve business continuity, it has two major limitations. First, a degraded condition may later return to normal. Second, treating degradation as a permanent failure can accelerate hardware replacement and wear, increasing maintenance costs.
To address these issues, the industry has introduced an improved approach: a application-aware software-isolation framework. The framework periodically monitors drive status indicators such as latency and IOPS. When a drive reaches a specified threshold, it is identified as stalled or slow, and a sequence of remediation actions begins.
Although this approach represents progress, several areas still require improvement:
- Limited timeliness: Replica-consistency checks must list all shards on the drive. From the first sign of a stalled drive to final isolation, the process can take at least several minutes, by which time business continuity may already have been seriously affected.
- Broad impact: Inconsistency in only a few shards on a slow drive can prevent the entire drive from being isolated, allowing the business disruption to continue.
- Imprecise identification: The framework monitors only the drive side, making missed detections and false positives more likely.
PART 2. aSAN Drive Soft Isolation: A New Model for Storage Fault Handling
Sangfor aSAN drive soft isolation combines application-aware fault detection, heuristic fault diagnosis, and precise component quarantine. These capabilities help prevent a single degraded component from affecting business continuity.
1. Soft-Isolation Framework Architecture: Two Layers Protect Storage Stability
The aSAN soft-isolation framework uses the physical virtual storage volume as its management unit and consists of two parts:
- Storage client data plane: Replica-degradation sensing protects replica availability, temporarily isolates a degraded replica within seconds, and rapidly reports fault information so that business operations remain unaffected.
- Soft-isolation control plane: The Distributed Fault Center (DFC) collects fault information reported by Distributed Fault Node (DFN) plug-ins. It correlates and analyzes the reports to diagnose faults accurately and reduce false positives. If a fault can recover within a short period, the degraded replica is re-enabled after recovery, avoiding prolonged operation with fewer replicas. If a fault does not recover, the affected component is fully isolated and the failed replica is rebuilt to protect data reliability.
Figure 1. aSAN drive soft-isolation framework architecture.
2. A Breakthrough in Application-Layer Software Isolation Without Physical Drive Removal
Compared with the industry’s common physical-drive removal method and conventional application-aware soft-isolation approaches, the Sangfor aSAN framework uses a fully software-based isolation mechanism. It eliminates physical drive-removal operations, avoids compatibility issues across different hardware brands, and improves solution versatility and stability.
DFN also provides a fault plug-in interface that integrates the data-plane client plug-in and the stalled-or-slow-drive detection plug-in. This extends fault-handling capabilities and enables more precise remediation.
Overall, the Sangfor aSAN drive soft-isolation solution demonstrates both architectural innovation and practical value at the application layer, offering a new approach to fault handling in distributed storage systems. The following two sets of comparative results show how the solution performs in practice.
PART 3. Validation Results: aSAN Performance Leads Across the Tests
1. Comparison with VMware in Stalled- and Slow-Drive Tests
Sangfor conducted comparative testing and practical validation against VMware across five areas: stalled disk I/O detection, slow disk I/O detection, RAID-controller degradation detection, reconstruction after a stalled-or-slow-drive fault, and the effect of such a fault on business I/O.
| Test Item | aSAN (Sangfor) | vSAN (VMware) | Comparison |
| Disk I/O Stall Detection | For an HDD, if an I/O request remains stalled for 13 seconds or its latency exceeds 500 ms, aSAN identifies the drive as stalled. The drive is not immediately removed from service. | If a device or controller does not respond to an abort command within 120 seconds, the default timeout, vSAN marks the disk or disk group as offline to prevent the issue from affecting the entire vSAN cluster. | aSAN provides faster detection. Combined with replica-rebuild mechanisms, aSAN helps prevent transient faults from being misclassified and enables fast, accurate detection. |
| Slow Disk I/O Detection | For an HDD, if I/O latency exceeds 70 ms for 1 minute within a 5-minute period, aSAN identifies the drive as slow. The drive is not immediately removed from service. | The driver downgrade criterion is triggered when 4 or more delayed writes occur within an approximately 6-hour period and write latency exceeds the predefined driver threshold. The thresholds are 500 ms for HDD writes, 50 ms for SSD reads, and 200 ms for SSD writes. | • aSAN identifies slow drives more quickly, using relatively direct and straightforward criteria. • vSAN uses more complex criteria and a longer evaluation period. |
| RAID Controller Degradation Detection | The storage layer does not separately perform statistical detection of slow I/O at the RAID hardware level. At the system software layer, RAID controller degradation is detected using the same mechanism as slow-drive detection. | If real-time I/O transmission between the RAID controller and host exceeds 30 seconds, vSAN immediately marks the device as Degraded. | aSAN does not depend on hardware-layer monitoring or remediation, providing broader hardware adaptability and reducing reliance on specific hardware vendors. |
| Rebuild After a Stalled or Slow Drive Fault | A rebuild is initiated if the drive fault persists for 5 minutes, or if the fault recurs and automatically recovers 3 times within 1 hour. This helps prevent a rebuild from being triggered by a transient drive fault. | After a disk is marked as degraded, vSAN immediately starts rebuilding if sufficient capacity is available. | aSAN continuously evaluates the fault condition, significantly reducing the possibility of an unnecessary rebuild caused by a transient fault and improving overall system stability. |
| Business I/O During a Stalled or Slow Drive Fault | If I/O times out after 8 seconds or remains slow continuously, the affected drive is downgraded and business I/O is no longer sent to the affected replica. However, if an individual shard has fewer replicas than required, aSAN can continue sending data to the slow replica at the shard level, maximizing business continuity. |
• After a disk is marked as Degraded, all I/O is paused while vSAN recalculates the accessibility of every mirrored object. • If the object remains healthy, meaning more than 50% of votes are available, I/O resumes. The check takes approximately 5-7 seconds. • If a VM has an FTT = 0, the virtual machine can be restored only after the original disk recovers or a replacement disk is inserted. |
• In multi-point fault scenarios, aSAN supports more granular, shard-level degradation and isolation, delivering more precise remediation. • After vSAN marks a disk as degraded, the impact is broader. The 5-7 second check for all mirrored objects can introduce performance variability. If replica requirements cannot be met, I/O at the disk level may become unavailable, affecting business services. |
Figure 2. Comparison with VMware in stalled- and slow-drive tests.
The comparison with VMware vSAN in handling stalled and slow drives showed that the Sangfor aSAN drive soft-isolation solution delivered stronger performance and a more comprehensive monitoring mechanism in the tested environment.
2. Comparison with Another Vendor’s Soft-Isolation Solution
| Solution Detail | Sangfor | Comparison Vendor | Comparison |
| Remediation Approach | Sangfor uses a multi-level isolation strategy based on intelligent I/O analysis, enabling flexible and precise isolation while minimizing the impact on the overall storage system. | • Uses drive-level isolation. • After iostat detects an anomaly, the entire drive is isolated. |
Sangfor provides finer-grained isolation and more precise remediation. The comparison vendor’s out-of-band monitoring supports only drive-level isolation. The approach requires less implementation effort but delivers less effective results and may affect the use of normal data on the drive. |
| Detection Point | Sangfor uses embedded monitoring to observe every storage-client I/O operation in real time and collect more detailed performance metrics. | • Relies on out-of-band monitoring. • Focuses primarily on overall disk-level performance metrics. |
Sangfor’s embedded monitoring provides finer granularity, more comprehensive metrics, and a lower risk of misclassification. The comparison vendor’s out-of-band monitoring is simpler and requires less implementation effort, but its detection granularity is coarser and its metric coverage is limited, making it difficult to identify anomalies accurately. |
Figure 3. Overall comparison with another vendor’s soft-isolation solution.
The comparison produced the following observations:
- Remediation approach: Sangfor aSAN uses a multi-level isolation strategy based on intelligent I/O analysis. The isolation granularity is finer and remediation is more precise. The comparison solution supports only drive-level isolation.
- Detection point: Sangfor aSAN uses embedded monitoring to observe every I/O operation from the storage client. This provides finer granularity, more metric dimensions, and a lower risk of misclassification. The comparison solution uses out-of-band monitoring through iostat. Although this requires less implementation work, it provides less effective detection.
Overall, the Sangfor aSAN drive soft-isolation solution combines accurate and efficient detection, intelligent and flexible policies, and broad scenario coverage. These capabilities provide strong protection for stable business operations and make the solution a reliable option for enterprise storage systems.
PART 4. Production Deployment
aSAN Improves MES Stability and Reduces Operational Effort
In our Manufacturing Execution System (MES), degraded drives used to cause a sharp decline in response time. Production scheduling, material management, and quality monitoring were all affected. The resulting monthly economic losses reached hundreds of thousands of RMB, while troubleshooting consumed significant operational time and effort and seriously disrupted normal production.
Executive Director, Tier-1 Manufacturing Enterprise
After the Sangfor aSAN drive soft-isolation solution was introduced, the MES environment achieved the following improvements:
- Rapid response: A low-frequency HDD stall was handled within 15 seconds, allowing business services to return to normal quickly.
- Sustained stability: Multiple degraded-drive events over six months were addressed while keeping service interruption within a very short period.
- Improved efficiency: Stable and efficient MES operation enabled more timely and accurate production scheduling, significantly improved material-management efficiency, and supported effective real-time quality monitoring, contributing to a lower product-defect rate.
Beyond direct financial savings, the IT operations team was freed from tedious hardware troubleshooting, allowing them to devote key resources to strategic IT initiatives and user experience, driving measurable gains in overall operational productivity.
Sangfor aSAN’s drive soft isolation sets a new benchmark for proactive fault management in distributed storage. Backed by years of in-house storage innovation, Sangfor continues to advance reliable, enterprise-grade solutions that empower organizations to replace legacy infrastructure and navigate complex digital landscapes with total confidence.
FAQs on Sangfor aSAN & Degraded Drive
Sangfor aSAN (also known as Sangfor Distributed Storage Fabric or Sangfor Server SAN) is a software-defined distributed storage solution that aggregates local disks from standard x86 servers into a unified storage pool. It delivers block storage and iSCSI services to virtualized environments, letting VMs dynamically consume storage based on business needs.
A degraded drive is one that remains online but experiences abnormally high I/O response time — for example, latency rising from around 10 ms to over 100 ms. Unlike a failed drive, it doesn't stop working entirely, but it can stall replica writes and slow the whole storage cluster.
Because the drive is still technically online, traditional monitoring can't easily tell whether it's truly failing or just experiencing a temporary slowdown. Treating every slowdown as a permanent failure risks unnecessary hardware replacement, while ignoring it risks prolonged business disruption.
aSAN uses a two-layer soft-isolation framework: a storage client data plane that senses replica degradation and isolates it within seconds, and a control plane (Distributed Fault Center and Distributed Fault Node plug-ins) that correlates fault data, re-enables replicas that recover, and fully isolates and rebuilds replicas that don't.
vSAN marks a disk or disk group offline if it doesn't respond to an abort command within a 120-second default timeout, and pauses I/O for all mirrored objects for 5–7 seconds during recalculation. aSAN detects stalled I/O in as little as 13 seconds, uses shard-level (rather than whole-disk) isolation, and can continue sending I/O to a degraded replica when replica requirements aren't otherwise met.
No. aSAN uses a fully software-based isolation mechanism at the application layer, avoiding physical drive removal and the hardware-compatibility issues that can come with it.
A rebuild starts if the fault persists for 5 minutes, or if it recurs and self-recovers 3 times within 1 hour — a design meant to avoid triggering unnecessary rebuilds from transient issues.
In a Tier-1 manufacturing enterprise's MES environment, aSAN resolved a low-frequency HDD stall within 15 seconds and maintained service interruption at very short durations across multiple degraded-drive events over six months, reducing operational troubleshooting effort.
