As a follow-up to my post about flaws in many SSD benchmark reviews, I just stumbled upon an excellent presentation by Christopher George, founder and CTO of DDRdrive, who presented ZIL Accelerator: DRAM or Flash (pdf) at the OpenStorage Summit 2010 on October 26-27, 2010. His company makes the DDRdrive X1: a DRAM-based storage device in the form of a PCIe card that is optimized for the typical aggressive random write workloads that a "ZIL accelerator" is submitted to, which is a device that the Solaris ZFS file system uses to store the ZFS Intent Log to accelerate synchronous writes.
More than a year ago, in July 2009, I presented a talk entitled MD5 Chosen-Prefix Collisions on GPUs at Black Hat USA. I made the slides and whitepaper available on my homepage shortly after the conference, but the source code was not available at the time. I wanted to fine-tuned 1 or 2 final details and write slightly better documentation before releasing the code. Well, believe me or not but it took me more than a year to do this. I am glad to finally release "bday" (source code): an implementation of the improved birthday search described in On Collisions for MD5 (Marc Stevens' Master's Thesis), using a hand-optimized MD5 compression function written in AMD CAL IL (Compute Abstract Layer Intermediate Language) targeting ATI Radeon HD 4000 series and higher GPUs.
Darren Moffat, one of the lead engineers working on the ZFS filesystem at Oracle, just announced that ZFS encryption has shipped, today, in Solaris 11 Express. This is amazing news. I have been waiting for years for this feature to be available in a production Solaris release. He made 3 posts about it:
- Introducing ZFS Crypto in Oracle Solaris 11 Express
- Assured delete with ZFS dataset encryption
- Having my secured cake and Cloning it too (aka Encryption + Dedup with ZFS)
Probably the most beautiful feature is the last tidbit: ZFS encryption fully leverages ZFS deduplication. A naive design would have prevented deduplication on encrypted datasets from being useful, because in practice a ciphertext block is unique, hence cannot be deduplicated. But Darren contributed a concept called "clone families" that allows the dedup step to happen before encryption. (Now, there are I presume rare situations where one may not want end-users to be able to determine which plaintext blocks exist in the pool by watching for deduplicated blocks, and these situations can be avoided by passing the -K option to zfs key or zfs clone, so the decision of whether to allow or disallow deduplication on encrypted filesystems remains in the hands of the ZFS administrator).
Another great feature is the fact that ZFS encryption automatically uses the hardware-accelerated AES implementation on SPARC T series processors, and the AES-NI instruction set on Intel Nehalem processors. (On a side note, I once met Dan Anderson who was working on adding AES-NI support to Solaris. He is a sharp and passionate guy.)
All in all, thumbs up for ZFS crypto! Use it to provide some amount of data privacy to a server in a remote colo in case the server is shut down and stolen. Or use it in combination with "zfs send | zfs recv" for data replication/backup to off-site encrypted disks. Or even use it on a laptop for data privacy while benefiting from all the other ZFS features: end-to-end checksumming, self-healing, snapshots, clones, compression, deduplication, redundant blocks (zfs set copies=N), etc. People forget that ZFS is useful even on single-drive machines!
NetApp Flash Cache (previously known as the Performance Acceleration Module II, or PAM II) is simply 256 GB or 512 GB of SLC NAND flash on a PCIe x8 card. It plugs into NetApp filers. They use it to cache hot data to accelerate read operations. See picture. Pretty simple, right?
Flash Cache is very effective at improving overall performance, mostly because the amount of memory contributing to caching in a NetApp filer (usually 8-32 GB) tend to be small compared to Flash Cache. For example, NetApp published benchmarks comparing one of their Fiber Channel solution with a SATA + Flash Cache one. The latter matches its performance, with half the drives, one third the power consumption, and at half the price, thanks to its solid-state cache!
In fact, Flash Cache is so effective that NetApp, fearing this would impact their high-end filers sales, decided to sell it at a 40x markup: a 256 GB Flash Cache costs $40k instead of $1k for similar SLC NAND flash-based products as found in markets where vendor competition actually takes place (read: SSDs at $4/GB). The irony is that NetApp says, in the Flash Cache overview linked above: "for most applications it's hard to justify the cost [...] of SSDs", then goes on to introducing Flash Cache as the solution. Hmm. I think you are wrong, NetApp. SSDs are 1/40th the cost
More seriously —who has not seen a marketing department commit a gaffe ?— this is a sad example of what a closed proprietary storage vendor can force upon his customers... Some will say that the high price tag is justified by the incredible performance, but they miss the point. This performance can be had, at $4/GB, from competing products such as Linux storage servers with bcache, or ZFS storage servers which L2ARC cache devices. Both allow end-users to configure any number and any type of solid-state storage devices as cache devices. In the end, this is a usual tradeoff: the prospective customer should ask herself if the NetApp feature set is worth a 40x markup or not.