What is benchmark testing, and why does it matter for product certification?
When data storage product vendors want to receive product certification from programs like Microsoft Windows Logo Program or VMware Ready, for example, they typically conduct benchmark testing in order to measure the product’s performance.
With benchmarking, these vendors can acquire extensive data that helps them identify issues, improve functions, validate claims, and ensure they are ready to attain product certification.
While there are a number of benchmark tests available, we want to stick to the most common benchmarks for enterprise-level storage systems, which are SPC and SPECsfs. Let’s take a look at what these tests mean and what vendors should anticipate.
SPC
The Storage Performance Council (SPC) benchmark includes SPC-1 and SPC-2, which are commonly used to test block-based storage subsystems, such as SAN. Some storage vendors use both tests, while others may only use one of them.
The difference between SPC-1 and SPC-2 (aside from their release dates in 2002 and 2006, respectively) is the type of workload that is simulated during the testing. Random block-access workloads (such as in database and OLTP systems, mail server applications) are simulated by SPC-1, while sequential block-access workloads (such as financial processing, scientific computing, video-streaming) are simulated by SPC-2.
When conducting benchmark testing with SPC, storage vendors most commonly receive performance metrics with one or both of these results:
SPECsfs
The Standard Performance Evaluation Corporation (SPEC) benchmark is typically used to test file-based storage subsystems. SPECsfs2008, the latest version of this test, evaluates the speed and request-handling capabilities of file-based storage by simulating typical workloads.
Typical file-based workloads that use the NFS or CIFS protocol – file sharing, web server applications, and NAS database and virtual servers – can all be simulated by the SPECsfs2008 benchmark.
When using SPECsfs, storage vendors can expect performance metrics to be referenced as NFS and CIFS throughput, indicated by ops/sec (ops per second).
Overall, SPC and SPECsfs provide valuable insight for storage vendors who want to measure the performance of their products before submitting to certification. By simulating typical workloads for common file- and block-based storage applications, these benchmarks are considered the industry standard when it comes to enterprise storage systems.
Vedams was able to bring the necessary expertise and resources to meet our business objectives. What I like the most is its flexibility in their business model to meet our needs.
Our partnership with Vedams has been highly successful. The Vedams team is extremely productive and dedicated to ensuring success. Their willingness to go above and beyond has resulted in consistently meeting schedules with high quality releases despite several challenges that needed to be overcome.
They conduct their work and business relationship with the highest degree of honesty and integrity.
Vedams is a highly valuable addition to Xyratex’s own global product test organization. With product availability and reliability requirements being key differentiators for Xyratex in its markets, the need for exhaustive, rigorous and in-depth functional testing is critical.
The knowledge required to perform at this level, being very data storage focused, is not easily obtained but I have been impressed by the way these skills have been cultivated and the low levels of staff turnover.
Vedams was able to bring the necessary expertise and resources to meet our business objectives. What I like the most is its flexibility in their business model to meet our needs.
Vedams’s global delivery model is strongly supported by their onshore project management. Because of the seamless communication maintained by the local project managers, we were able to keep our customer’s aggressive schedules on time and on budget.