Data Server: Options

[Scott Brady]

For Expeditions West and Overland Journal, we are currently spec'ing solutions for networked data storage. We are a small business (with a limited hardware budget), but have massive data storage requirements. Just in one instance of Aperture, we have nearly 40,000 images, and have made the transition to RAW only file capture, which has accelerated our data storage requirements.

To further complicate things, we have a distributed workforce, with team members in home offices throughout Arizona and the team often works from the road (trail) requiring remote access to shared data stores.

I figure it is very likely that more than a few ExPo members has experience in this area, and can offer advice.

MAC Solutions:
I would like to exhaust the available Mac solutions first.

The greatest challenge appears to be the remote file sharing, which would be remote access to the data files from the dispersed team over an internet connection (we have 7.2mb DSL to the office). MAC and PC would need to connect to the data server.

Apple's Remote Desktop does not appear to be the correct solution. OS X delivered VNC at first assessment might work.

One interesting option would be to use a Mac Mini and four Iomega Minimax .5TB drives. That would give 2.18TB of networked storage for about $1,800. Again, connection from remote locations will likely still be the kicker.

PC Solutions:

Lots here. Just starting to research options for the MAC users (everyone but me) to connect remotely.

Thanks for any ideas

 

[Wagontrain]

Do you need strictly network storage or the ability to work on files simultaneously. ARD works best on a LAN, used mostly by admins to monitor/update remote comps. Timbuktu (and others like it) give you control of the remote computer Mac or PC. If I was looking for network storage only, I'd look at some type of ftp site. This could be supplied by an ISP or even in your office if you have a static IP address and opened up a port in your firewall to log into. I find this to be much easier with a Mac than PC, but I am heavily biased. I use an ISP called bluehost.com (no affiliation) mostly for their storage space, they offer 300gigs. I'm sure there are others out there. I set up ftp sites for clients to log in or for video storage. I have a link from my web page that opens up a java script, which contains the ftp client. This is used for the clients that have no clue what ftp is. On my Mac I use Transmit, again there are others that work just as well. I realize 300 gigs isn't much, but I'm sure you could buy more. 300g cost me US166.00 for two years. If your team members are logging in from the road, how would they connect to a remote server? I'm sure other will chime in. I'm not a network admin, just a small business (audio/visual) owner who is constantly copying or editing video's, ppt presentations and such.
HTH
Phil

 

[bigreen505]

Scott,

Acquaint yourself with Drobo. PM me with questions.

 

[Desertdude]

http://www.apple.com/xserve/raid/

http://www.apple.com/xserve/

http://www.apple.com/server/

 

[efuentes]

For Storage go for SemiPro NAS solutions like Infrant ReadyNAS NV+, you can easily go multiterabyte for a lot less than Apples Xserver or Dells EMC, just remember to have a backup strategy. For the remote access stuff you have to take a hard look at available network speed and decide over file sharing or some kind of file sync, all of this over a nice and easy VPN like hamachiX.

Feel free to dropme a PM,

Saludos

 

[Scott Brady]

Do you need strictly network storage or the ability to work on files simultaneously. ARD works best on a LAN, used mostly by admins to monitor/update remote comps. Timbuktu (and others like it) give you control of the remote computer Mac or PC.

Mostly for storage and data management, but as my research develops, I am finding more advantages to a computer over a networked data storage devise only.

For example, the unit could also manage a printer or two and also manage music and video centraly and display video and publishing projects on a larger screen, likely located in a larger room used as a conference area. I could easily see a Mac Mini running itunes, and having several .5TB external drives attached.

 

[Scott Brady]

http://www.apple.com/xserve/raid/

http://www.apple.com/xserve/

http://www.apple.com/server/

Ahhhh, I wish

 

[Scott Brady]

For Storage go for SemiPro NAS solutions like Infrant ReadyNAS NV+, you can easily go multiterabyte for a lot less than Apples Xserver or Dells EMC, just remember to have a backup strategy. For the remote access stuff you have to take a hard look at available network speed and decide over file sharing or some kind of file sync, all of this over a nice and easy VPN like hamachiX.

We have a very reliable and 7.2mb DSL connection. I think it has been down one day in 2+ years.

 

[Desertdude]

Ahhhh, I wish

You and me both... hmmm maybe soon

Mac Mini could have a connectiveity issue (on faster through put no card slot?)

A bit on my storage solutions;

I am using with great results, http://www.sonnettech.com/product/fusion400quad.html (removable drives)

and sonnets internal card

for connectivety to the MacBook Pro 34 Esata

I travel with a 500 gig Esata drive enclosure
with eSATA+FW 800/400+ USB 2.0 ports

I can swap and connect to any config. Back ups are easy, swap in a drive clone and store
- Buy more drives



With how much we all now need to store and work with
I look forward to hearing what your final research solution brings

:::

 

[craig]

Amazon S3

Scott,

Consider offloading all of your in-house IT costs and go with something running on top of Amazon's S3 file storage service. I think when you add up the costs of power, IT, and hardware that you'll find that you are saving quite a bit.

http://www.amazon.com/gp/browse.html?node=16427261
http://jeremy.zawodny.com/blog/archives/007624.html

Craig

 

[Wagontrain]

Scott,

Consider offloading all of your in-house IT costs and go with something running on top of Amazon's S3 file storage service. I think when you add up the costs of power, IT, and hardware that you'll find that you are saving quite a bit.

http://www.amazon.com/gp/browse.html?node=16427261
http://jeremy.zawodny.com/blog/archives/007624.html

Craig

Wow! This is an inexpensive option. The ftp client I use just upgraded the software to use this service. I never checked it out until now.
WT

 

[Pskhaat]

Scott,

May I suggest with the quantity of data you must have both in video and other media you consider integration with Content Networks (CDNs)? I am finding less and less need lately for DAS and am personally moving most future projects onto CDN-like storage.

Another option is Amazon S3, and I have thus managed tebibytes of distributed storage for under $200/month; redundancy and network topography built-in.

There are several vendor/value-add companies that expose these types services as networked directories.

 

[gary in ohio]

Have you taken a look at some of the linux based NAS solutions?
http://www.clarkconnect.com/ is one of serveral NAS in a box.
It runs on a x86 box, you hang whatever size disk you want using whatever redundancy you need, raid, mirror, jbod. Allows for NFS, SMB, FTP access to files.

 

[mmtoy]

I work on a NASA mission that's just recently hit 26 Tb of data coming back from Mars (our specific part is "only" 10-15 Tb of raw data, expanded to about 100 Tb right now)... We have a team of scientists accessing these data (images) from all over the world. Most images are in the neighborhood of 1024x80,000 pixels and 16 bits per pixel. The images can get up to about 130,000 pixels long or even longer if they're only 512 pixels across (350,000 pixels).

...So, probably what I say is overkill for you, but...

...I think you will find that a mac mini won't hold up over a couple of years of 10,000+ images per year. Access speed will slow nonlinearly with additional storage and access needs.

It's not clear if you intend for your remote users to upload/download images or just text for articles. I assume they need to upload images as well as text.

I'm a big fan of UNIX systems for servers, so that's what I'll address. BUT, if you're running the system in-house, you'll need to use something that's comfortable for you and your people. Nothing is worse than trying to learn a completely new system while also trying to get your work done. You said you want to exhaust the mac options first, so I'll address those.

If you don't need multiterabytes, I think you could get away with, for now, a powermac/MacPro with an eSATA attached RAID system. You'll want RAID for safety. You'll want eSATA for the speed and the fact that it's external, relatively easily upgradeable storage. You'll also want a tape library for backups. The powermac will respond well for years, giving you time to build up the equity to plan on your next upgrade. I don't think a mac mini's responsiveness will last long enough.

If you are looking at multiterabytes of storage, you should consider a NAS or SAN. These are more expensive, but they're expandable and are generally much better at serving data than just a computer connected to a bunch of disks.

For remote access, I would suggest you use rsync connected via ssh. rsync allows you to sync your client directory/filesystem to the server (or from server <->client). ssh is the secure connection that protects your stuff from crackers. While you're on the road, rsync will allow you to compress (losslessly) your images and other data as you download them to the server, putting the strain on your CPUs rather than your net connection (or at least relieving some of that strain).

For external viewing of images, if you have the money for the disk space, you could consider making copies of your RAW images into JPEG2000 images, which is a lossless compression format that is designed for access across networks--you only download the parts you're interested in (low resolution at first, zoom in and download only that part you want). We use this format for distributing our images to the public and to the science community (but, our images are so long that we really need a way to focus on one part of the image). Because it's lossless, the data stored on the servers is complete.

Another thing you will want to consider is how to FIND a specific image on that giant storage system. If you keep your info data that your cameras write when an image is saved, that will help, but I highly recommend you consider an SQL database like Postgres or mySQL to help you maintain your catalog of images.

Sorry. This is way too long. I hope some of it was a little helpful. I'm happy to answer questions, either on line or via PM.

--mmtoy.

 

[Scott Brady]

Solution in place... Working perfectly. Development of the CRM in FileMaker is next.

Thanks for the help Alvaro!

Full details here

From this:

To this: