Atomic Commands Part II: Vectored Commands

Another factor that’s important to consider in the atomic write story is the advent of PCIe based storage. Multiple vendors now produce PCIe cards that can provide terabytes of data on a single PCIe card. The NVMe interface was developed to take advantage of this class of device, but T10 was not far behind when the SCSI over PCIe (SOP) and the PCIe Queuing Interface (PQI) specifications were developed in response and are now reaching maturity. These developments have allowed database serves to become much more efficient by speeding access to significant amounts of data at much higher speeds – usually by several orders of magnitude.

With this increase in speed, the system overhead required to process each SCSI command CDB becomes a much larger part of the total time required to process a write or read operation.  With rotating media, it may take many milliseconds to process a write command due to the latency inherent in the physical media. In this case, the time required to Process the command CDB is very small compared to the overall operation. For flash devices, the access is measures in micro-seconds and processing of the CDB becomes a significant portion of completing the write operation.   

Vectors Figure 1

A solution to this problem is to define vectored read and write operations. These commands permit read and write operations to multiple data segments which do not have to be contiguous, unlike normal read or write commands. This is analogous to the scatter/gather lists employed in typical HBA interfaces and provided in the PQI and SOP specifications. Historically, there has been strong resistance to vectored commands within the T10 committee due to the complexity of error processing. It also makes little sense in rotating media where the latencies are large and a queue of many individual commands works as well and is more easily implemented and managed. But for flash memory based storage with a PCIe interface, atomics may provide a strong motivator.

…to be continued…