Uncomplexifing data dedupe - February 6, 2008
Source de-duplication is set to be "more disruptive" than previous technologies
Data de-duplication technology has emerged as a key technology in the effort to reduce the amount of data backed up on a daily basis, which in many enterprises is growing at more than 100% every year.
For example, Jame Thomson, IT manager at Tampa-based medical firm, was able to use data de-duplication technology to reduce the amount of data streamed from more than a dozen remote offices and thereby cut his backup window from 11 hours to 50 minutes. Thomas says his compression ratio for his backups run as high as 55:1, using technologies provided by Sencilo Soutions.
Vendors have taken different approaches to the technology resulting in multiple distinct products that users should become familiar with in order to choose the flavor that best suits their environments.
Data de-duplication uses commonality factoring to reduce the amount of data either at the backup server or at the target storage device. As a result of enormous compression ratios achieved by data de-dup technology, SATA disk are becoming more attractive as a viable, online alternative to traditional tape-based backup. For example, people working at remote and branch offices need instant access to all the data and applications available at their company's headquarters. So IT shops typically set up remote mini data centers, with application servers, block and file data storage, backup tape and report printers, sacrificing administrative control. By utilizing data de-dupe technology, backups can be performed over the WAN using spare nighttime bandwidth, eliminating the need for tape at remote sites.
Brian McCarthy, President and Industry Storage Veteran for Sencilo Solutions says de-duplication technology mainly resides in the backup space, complementing traditional tape libraries with the purpose of lowering costs and reducing data.
The main benefit to de-dup technology is that you're not seeing your virtual tape library fill up, and you're "not seeing your backup targets fill up as fast as it normally would," McCarthy says. Watch for Gartner, Inc.'s Magic Quadrant to show up with several of the leader including Quantum, Overland Storage, EMC, and NetApp.
De-dupe can be done at the target of a backup stream (the storage array or tape drive) or at the source of the data being backed up (the application server). Traditionally, de-dupe products had been used as a target for backup data, but McCarthy says there is "a growing emphasis de-duping back on the server."
Target de-dupe products are generally used as part of a final repository for backup data. Most backup software today supports tape volumes, files or raw disk as targets. Target de-dupe products mimic a tape library and support virtual tape libraries (VTL), or they can act as a network-attached storage file server supporting network file system (NFS) or Common Internet File System files. Target de-dupe technology can also work on raw disk supporting Internet SCSI or Fibre Channel logical unit numbers (LUN). Prominent target based de-dupe products are sold by Data Domain Inc., Diligent Technologies Corp., ExaGrid Systems Inc., Overland Storage., Quantum Corp., and Sepaton Inc.
Today, de-duping data at the target is the leading method, but de-duping data at the source where the data is coming from is even more disruptive, and the benefits are far greater, McCarthy says. But having a integrator who specializes in this type of technology is most important to your success. Sencilo has deployed some 2+ peta-bytes of de-dup based VTLs.
Source de-dupe products replace backup software used in a client/server configuration, where remote clients de-dupe data being backed up and only transmit unique data to the central server. This reduces bandwidth requirements considerably, according to McCarthy. Some prominent source-based de-dupe products include Asigra Inc.'s Televaulting for Enterprises, EMC Corp.'s Avamar, Network Appliance Inc.'s SnapVault, and Symantic Corp.'s NetBackup Pure Disk.
In band or out of band
Another characteristic used to discriminate target de-dupe products is when data de-duplication processing occurs. Data de-duplication takes time to compute and find commonality in the data being backed up. To minimize the effect on backup performance, some vendors de-dupe data in the background. These de-dupe products buffer the backup stream to disk and then after the fact reduce its size via de-duplication. ExaGrid, FalconStor and Quantum provide target de-dupe products that do background data de-duplication. This type of technology will shortly replace first generation products from suppliers like Data Domain, who plays mostly in the small non-critical shops. Earlier products can handle the backup stream and de-dupe in band, in real time. Target vendors that de-dupe in band include Data Domain and Overland Storage. All source vendors de-dupe in band as well. Paradoxically, the in-band vendors are not able to sustain full backup stream performance, and therefore less desirable for anything larger then a few tera-bytes, and found more in the SMB space.
The unit of de-dupe granularity, called chunk size, further differentiates de-dupe products. Only NetApp touts a fixed chunk size equal to data block size, according to McCarthy. Most de-dupe products claim variable chunk size from the file level down to sub-block level. By using variable chunk size for data inserted into a file, you need only show the changed data as being different, and the rest of the file would be the same. Even more impressively, most de-dupe products can not only reduce data from generations of the same file but also eliminate data copies across files.
Yet another de-dupe difference is file-type sensitivity. Some target products open the backup stream to determine data type and invoke file-type-specific policies to provide better de-duplication. Sepaton claims to have the most file-specific policies. Other target vendors claim that their products perform file-specific de-dupe to a lesser extent, including Data Domain, Diligent and Quantum, which all claim they are completely agnostic regarding backup streaming, they may claim it but not all of them are telling the truth. So check with a partner with experience that has installed and supports them.
The numbers for de-dupe compression rates range anywhere from 3:1 to 500:1. De-dupe products can sustain these high compression rates because backups generate duplicate data every time a full backup is run. Moreover, beneath the file level, most data is not unique even though a file is modified. But, some data does not de-dupe well, including audio, photo, movie and other media files that simply don't have excess white space or duplicate data.
Bob Sanders manages 12 De-Dup boxes throughout the U.S that Sencilo sold them last year. They are being used as VTL targets for Symantec NetBackup. The boxes have a capacity of .5TB to 18TB. But, each of Sanders VTL boxes backs up the equivilant of 600TB to 1,500TB of data.
The arrays have been in production for more than nine months. Each office does a local backup to its on-site de-dup boxes. That data is replicated off-site to headquarters in Orlando and then again replicated to the law firm's secondary site. Before the Sencilo solution, Sanders says his company had DLT-3 tape changers and LTO-2 tape. Backups at some offices used to take more than 14 hours, but now with new de-dup VTLs, the backups are under 45 minutes. Sanders backs up one site with 163 TB of data onto 4.6TB of Data Domain for a compression ratio of 34.6:1. Smaller sites have experienced a compression ratio as high as 55:1.
"The hardware was comparable to buying all-new tape hardware, but the speed and the ability to not have to manage the tape is what really got us … the replication is an added bonus," Sanders says.
Grahame McKenzie, manager of IT at Crawford Adjusters Canada Inc. in Hamilton, Ontario, had performed local backups at each of the company's 80 remote sites. He now manages 80 remote and 20 local Asigra clients backing up to a single 2TB Asigra TeleVault. They mirror the central TeleVault to an off-site hardened data center and copy their monthly full backups off onto tape moving these off-site to another location.
McKenzie says that before, "we weren't even doing it [backups] on the smaller XP share points -- there was no backup. It just wasn't economical; if it was an NT server, it would [be] backed up to tape." This process was time-consuming and not subject to easy validation. He says that "with Asigra, backups all take place, and they all come here, … and I get a report on everything the next day." McKenzie says that in some cases, it was difficult to quantify a return on his investment because no backups were being performed, but he did say "we don't invest in tape on our new branch servers, and we have one tape library here."
McCarthy says data de-duplication allows users to retire most of their tape infrastructure at remote and local sites. In some environments, it may not be possible to eliminate all tape processing, but it can be reduced considerably. At essentially the purchase price of replacing tape, you can get all the benefits of tape with the convenience of disk -- mainly quicker, less error-prone and less operator-intensive backups and restores.
Sencilo Solution is a Florida based integrator specializing in storage, networking and security solutions. Sencilo delivers a comprehensive portfolio of products from best-of-breed hardware and software from multiple manufacturers.
Each product has been carefully selected for its unique value to a solution or deployment. Sencilo sales and technical team is thoroughly trained in all of our major products and are able to provide value in up front architecture planning and product selection. Our technical expertise is known throughout the storage, networking and security industry. Call us today at (407) 265-6293 or visit us at www.sencilo.com
About Us
Sencilo Solutions is a Florida-based integrator specializing in storage and security solutions. Sencilo delivers a comprehensive portfolio of products from best-of-breed hardware and software from multiple manufacturers including VMware, EMC, Juniper Networks, Hitachi, Symantec, Barracuda Networks, and HP. Its technical expertise is known throughout the storage and security industry. Clients include leading corporations, major financial institutions, top universities, government facilities, as well as small to medium size businesses. Sencilo's professional services include consulting, integration, project management, installation, maintenance and knowledge transfer.
Sencilo has offices throughout Florida including: Jacksonville, Daytona Beach, Miami, Tampa, St. Petersburg, Orlando, Hialeah, St. Augustine, Gainesville, Ocala, Palm Coast, Clearwater, Kissimmee, Lakeland, Maitland and Cape Canaveral
Offerings Projects: Replication De-Dup De-Dupe iSCSI SAN NAS VMware Security EMC NetApp HP IBM Quantum Compliance VTL Data Domain vs Gartner Magic Quadrant Quadrent LTO Backup Exc Pure Disk NetBackup Networker TSM Commvault BakBone D2D D2D2T compare cloud data deduplication thin provisioning DXi Global Compression DDX virtual tape library Data Reduction SEPATON FALCON compare Celerra CLARiiON Equallogic Dell NS20 NS40 CX4 CX3-20 CX3-40 CX3-80 FAS2050 FAS3050 Xiotech Nexsan Avamar DLD3 1500 D3 Storwiz storage compression data Ocarina Networks A-SIS compare Sepaton infopro BlueArc OnStor Microsoft Unified Storage data protection
