Manage Data Like a Pro

Data management is one of the easier tasks for the home user. Whether they're using Windows or a Mac, they'll find their way around the system pretty quickly; it doesn't take a tutorial or a day of training at either system's developer's headquarters. All the photos and all the documents you may possibly possess can be stored on any hard drive modern laptops come with. It's virtually impossible to take enough photos or videos to reach the limits of your storage capacity, and even if you manage to, an external 2TB hard drive barely costs more than $100. Then there are Blu-ray disks that lose their value faster than a two year old smartphone.

Managing data is not so simple in the business world where there are dozens, if not hundreds, of users hacking away at their keyboards eight hours a day, five days a week. They're not only producing massive amounts of information every day but you're going to have to deal with aggregate meta-data batches, backups, archive data storage, and the like. Without a system to manage all that data, your company is going to run out of the seemingly infinite server disk space, then run out of the additional space those expensive NAS and SAN devices added up to the point where you hit a certain point.

There should be a limit, and there is. There are two ends to this exponential growth of data volume; first is the virtual wallet of the company, there is a point where you can't add more disk space because it would send the costs up on a relatively steep curve. The second naturally occurring barrier is the available floor-space in the server room. Again, after a certain point you can't put in more servers and more data silos because they simply won't fit. It should be immediately apparent that you cannot have virtually infinite disk space in the corporate sector like you could at home.

There needs to be a better way, a way that doesn't soak up hundreds of thousands of dollars just to dampen the most immediate effects of information overload. You may have guessed, it's time to manage some data.

Without setting an arbitrary limit on disk quota, which renders the job of data entry worker and second level users (whose job depends on using information found in the corporate data center) neigh impossible, you'd be looking at an exponential growth of file number and disk load. Setting such a limit pulls the plug on the power users; not setting it pulls the plug on IT. Fortunately there is a golden middle road, instead of putting a cap on how much can be stored, you can put a cap on what kind of data can be stored. This, backed up with additional corporate guidelines will allow you to finally get a handle on the size of the database, and consequently the total cost of ownership (TCO) of IT infrastructure.

Putting data management technology and guidelines to work always starts with the most important step of all; planning. Consult with end users, IT department, and data manager specialists to come up with a data retention policy, which complies with laws but doesn't put unnecessary stress on the hardware. It can mean having an age limit on stored documents anywhere from 1 year to 5 years, or more if you're unlucky enough to be operating in the finance or health sector.

Getting old and obsolete data out of the way is almost always a successful strategy; it's like drilling a hole on the bottom of the bucket when you want to avoid letting it fill up. The shorter you cut the retention period, the larger the hole will be.

Discuss with your team of experts, to see if you really need all the documents you're currently saving. If the members of the staff are filling a database through forms, evaluate the form. Look for redundant or duplicate fields, or look for cells that are no longer needed. Cutting fields as short as they comfortably can be will only win you a few bytes on every entry, but every little bit adds up.

If many images or raw video materials are stored on premises, try using lossy or lossless compression technologies; it's one way to trade disk space for computing capacity. In case the servers are not operating anywhere near full steam pressure, it can effectively halve the load of all partitions.

Data cleansing or scrubbing is another trick in the box; a data scrub purges the database from unnecessary bits and pieces, double entries or incomplete rows.

Some argue, partly in jest, that an outsourced data center is the best way to manage data. Their is some truth to that, but for those who aren't outsourcing and will be managing their own data, here are some tips.

Always use the lowest cost per megabyte storage media you comfortably can. It doesn't mean buying cheap hard disk drives, but going for the optimal solution. For instance, you may have documents that are two years old, and though no user has opened them in five months you're not entirely convinced that they don't hold relevant data. In this case there is no use keeping the file on the live disk in the server. Archive such files on tapes, because tapes tend to be cheaper per megabyte than HDDs.

If you don't need something, throw it out. If you no longer need a backup snapshot, throw it out.

Obviously looking for such cases manually consumes silly amounts of time and effort. That's where your data retention and management policy really kicks in. As soon as you finish the paperwork, and set clearly defined objectives for IT department, they'll become able to configure servers accordingly. Modern server operating systems give myriads of tools for efficient management of intellectual property; let IT use them to save you money.