I've heard heard them jokingly called "time bombs" by people in the industry. That's why our media server is set up as a RAID 5. When the drives start failing we have enough time to replace and rebuild before any media is lost. Scary stuff!
Storage of video files
- Thread starter iajetpilot
- Start date
I make sure I wear my "No I will not fix your computer" T-shirt when I go visit friends.
Laptops HDD's as I'm sure you know are even worse since they are smaller, generate lots of heat in a small space.
Trying to explain the importance of backup to a non tech person is so frustrating since they don't want to spend a few bucks on a UPS battery backup, external HDD or even schedule it to back up to a flash drive.
Only when their kids/family pics are gone from a HDD crash or a BSOD they take notice....
Laptops HDD's as I'm sure you know are even worse since they are smaller, generate lots of heat in a small space.
Trying to explain the importance of backup to a non tech person is so frustrating since they don't want to spend a few bucks on a UPS battery backup, external HDD or even schedule it to back up to a flash drive.
Only when their kids/family pics are gone from a HDD crash or a BSOD they take notice....
- Joined
- Jun 14, 2016
- Messages
- 1,744
- Reaction score
- 791
- Age
- 66
I burn multiple DVD copies. I've seen way too many external HDs fail. If the external HD fails, it's costly to recover the data. If a DVD fails, I have extra copies so nothing is lost (1 copy onsite, 2 copies offsite). I have 15 yr old CDs of photographs and no failures yet.
Because of the nature of my 'work environment', I'm running a 14TB NAS on 10Gbps Ethernet to my primary workstation and a cloud based backup solution. Even with this solution, I'm frugal about what material I retain and what goes into the bit bucket.
You can get a consumer grade 2-4 bay NAS for a few hundred dollars. Put in 2-4 drives at 3-5TB each you've got plenty of storage for many years to come - and the safety of data redundancy if you setup in RAID 5.
Thanks for all the suggestions! I really appreciate the help. Not sure what I'm going to do yet but perhaps a relatively inexpensive raid array might work, or multiple bluray disks as back up. Time to start doing some research.
You have to be careful about RAID 5 if speed is important. Unless you have true hardware RAID 5, you can cut your array performance by as much as a factor of 10. RAID 5 built into desktop MB and even inexpensive add-on cards handle the RAID in the driver.
The consumer NAS are probably the same way; a general purpose processor is doing all the work. Another problem with RAID 5 is when using desktop HDs. The HDs are designed for maximum data recovery. If a cluster starts reading bad, the firmware is going to spend quite some time making every attempt to read that data, with it assuming that's all you have, rather than giving up quickly and let the RAID handle the recovery. If speed is important, use RAID 1 or RAID 10 (span of mirrors). Costs more in drives since you need 2x the raw storage, but it is faster and you have automatic redundancy.
But if speed isn't a concern, then RAID 5 is a good way to go.
I had RAID 0 for several years as main OS/data storage with great performance but was concerned about the risks so I converted to RAID 5. My performance dropped like a stone.
The consumer NAS are probably the same way; a general purpose processor is doing all the work. Another problem with RAID 5 is when using desktop HDs. The HDs are designed for maximum data recovery. If a cluster starts reading bad, the firmware is going to spend quite some time making every attempt to read that data, with it assuming that's all you have, rather than giving up quickly and let the RAID handle the recovery. If speed is important, use RAID 1 or RAID 10 (span of mirrors). Costs more in drives since you need 2x the raw storage, but it is faster and you have automatic redundancy.
But if speed isn't a concern, then RAID 5 is a good way to go.
I had RAID 0 for several years as main OS/data storage with great performance but was concerned about the risks so I converted to RAID 5. My performance dropped like a stone.
This is true. However, for straight up storage (which is what the OP seems to be asking about), speed isn't that much of an issue. You certainly won't be editing 4K video off a RAID 5 NAS, but you can easily copy the video into a RAID 0 in the desktop to do the edits, then send the final product to the NAS to store. The sequential write speed of my RAID 5 NAS with only 5400rpm drives is perfectly acceptable to max out the 1gbps LAN.
- Joined
- Jun 28, 2017
- Messages
- 38
- Reaction score
- 5
- Age
- 53
You might want to try Dragonfly (Home). Just launched. Special promotions going on right now to build the community including discounted storage pricing, free give away of Spark, Mavic Pro, and DJI Goggles.
Upsides are:
- No hardware required other than your phone. You can transfer video in the field after a flight without removing the SD card
- Web uploader for batch moving collection to the cloud for safe backup/enjoyment
- App to browse your collection, grab files when you need them, and view your flights on the map
- Ability to browse other friends or the community if they give you permission
- Automatic creation of mixes that can be shared to email, text, social media, or saved
I'm honestly pretty skeptical of this "solution". So, I've just finished shooting and I have 64GB worth of 4K footage on the SD card in my P4P. How exactly does this footage get to the cloud (on location) using my phone?
- Joined
- Jun 28, 2017
- Messages
- 38
- Reaction score
- 5
- Age
- 53
=-) That's totally fair. If you used 3-4 batteries, recorded from take-off to landing in 4K, and never transferred after each flight that would be a LOT to transfer to the phone and it wouldn't be a good solution. I would only use the drone to phone approach if I was in a pinch or had video from a single flight.
If this was something I regularly did, I would use the web uploader and drag all of my files from the SD card onto the uploader and let them transfer directly to the cloud. It would be much faster.
Shill a little quieter, m'kay?
~~~
As to actual question from OP... lots of local spinning disks for intermediate storage, ideally in RAID w/ appropriate precautions. SSD for files you're editing/grading or otherwise working with.
And for me, I archive all fully rendered (and actually, most of my raw images and video) to Amazon S3. Technically, I archive to AWS S3 than that ages into AWS Glacier automatically.
Costs are:
- S3: $0.023/GB or $23/TB/month
- Glacier: $0.004/GB or $4/TB/month
So not the cheapest, admittedly... but S3/Glacier are also extremely reliable/durable for long-term storage.
Basically, once it's in there, as long as you pay your bill, it's indestructible.
~~~
If you need a video delivery solution (and bonus of free transcoding/downscaling) I'd also reccomend taking a look at Vimeo Pro.
- Joined
- Jun 28, 2017
- Messages
- 38
- Reaction score
- 5
- Age
- 53
@Shammyh ouch!
Point taken--that was a bit promotionally laden and I apologize. We're super proud of what we've built though, and would love it if some people would give it a try and give us feedback. Cheers.Yeah, cloud storage for lots of 4K video is not practical even if the cost for the storage was free as the cost for data usage would kill almost any mobile plan in a single day for me. Not that there aren't advantages to it but uploading 100GB and possibly more would wipe out my data plan ten times over -- in a single day.
If you're like me and can burn though more than 100GB in a day the only practical option is HD's or SSD's and to protect yourself you really want 2 or 3 for redundancy. My kit when I travel contains, for drone video alone, a 500GB Samsung T1 SSD for fast storage and in addition I have a 4TB Seagate Backup Plus external HD. I also have some older 2TB external HD's that I copy to every other day or so particularly if the 500GB T1 is filled.
When travelling for a week I can burn through 500GB though it's more likely a week will cost me closer to 350GB. Now, I also travel with other video cameras including a Panasonic G85 that also shoots 4K video at 100Mbps so I have anther 4TB external for that as well as older 2TB externals. Then, add to that I also usually bring a couple Nikon D800E's still cameras and can burn though 25GB to as much as 50GB in a day so for them I have still more 2TB HD's -- at least 2 for redundancy. All told that works out to 1 - 500GB T1 SSD, 2 - 4TB HD's, and 5 - 2TB HD's in addition to a couple 16GB to 64GB thumb drives and half a dozen SD and uSD cards ranging from 32GB to 128GB.
Hey, it's only money...
Brian
If you're like me and can burn though more than 100GB in a day the only practical option is HD's or SSD's and to protect yourself you really want 2 or 3 for redundancy. My kit when I travel contains, for drone video alone, a 500GB Samsung T1 SSD for fast storage and in addition I have a 4TB Seagate Backup Plus external HD. I also have some older 2TB external HD's that I copy to every other day or so particularly if the 500GB T1 is filled.
When travelling for a week I can burn through 500GB though it's more likely a week will cost me closer to 350GB. Now, I also travel with other video cameras including a Panasonic G85 that also shoots 4K video at 100Mbps so I have anther 4TB external for that as well as older 2TB externals. Then, add to that I also usually bring a couple Nikon D800E's still cameras and can burn though 25GB to as much as 50GB in a day so for them I have still more 2TB HD's -- at least 2 for redundancy. All told that works out to 1 - 500GB T1 SSD, 2 - 4TB HD's, and 5 - 2TB HD's in addition to a couple 16GB to 64GB thumb drives and half a dozen SD and uSD cards ranging from 32GB to 128GB.
Hey, it's only money...
Brian
- Joined
- Jun 21, 2017
- Messages
- 209
- Reaction score
- 78
- Age
- 55
Yeah - there's a pretty distinct jump in space/usage I think. Stemming in part from if you keep all originals, but that's more of an outcome of what you're intending to do with the video as well as how much you're willing to invest.
After a couple of TB or so it starts to get more expensive - not just bc you're buying more disk but there's a point at which if you are keeping that much data it doesn't make sense to do so unless you're protecting it better and redundancy costs $$.
I'm a photographer, I still have far far more photo data than video. But it's impressive how fast the video is able to close the gap! I'm running a 1TB SSD for live/project data (which is nearly full right now) and 12TB RAId10 (6x 4tb drives) as my large storage (also almost full now!). I've got a 4 bay readynas with 4x 4tb drives for backup - unfortunately that actually now has too small of a capacity to back up everything now, so I'm backup up the ssd only nightly since what's in there isn't backed up elsewhere. Then all of this is disaster recovery backup with Backblaze. They mean it when they say unlimited! (Not a shill, I'm just super happy with the service, been using it for 8 years for myself and implemented at two businesses and never had a single hiccup).
I haven't traveled for >1 day since I got the new drones, so 4x 64gb cards should be enough to cover a full day's shooting if needed. But otherwise it's basically the same setup as Brian above. 512gb ssd for live, a couple of externals as needed for backup. When I travel with my camera gear I copy it off, back it up, but still try to not touch the card until it's home and onto the array. But that's not always feasible unfortunately for a longer trip. Heck, I've been making a bunch of HDR panos on t eh drone and those alone can eat up 9gb for a single pano.
- Joined
- Jun 14, 2016
- Messages
- 1,744
- Reaction score
- 791
- Age
- 66
I had a photographer friend a few years ago that had a great RAID setup (at his house where he did his photo editing). All of his work was backed up numerous times. After shooting a wedding, he put every photo on his computer to edit. Multiple backups via RAID. Worked great until he left to shoot another wedding (eliminating any sign of the previous wedding from his camera cards). After shooting, a neighbor called to inform him that his house was a glowing pile of embers. I had often suggested multiple DVD copies or at least an external drive he could move offsite but he said it was too much work. The RAID setup did him no good at all. He lost everything, including the previous wedding.
He tried an external HDD after that AND made multiple DVD copies, storing several copies offsite at his office. Two weeks (two weeks!) after starting this, his external HDD failed.
I stopped by his office and he informed me of the failure. I jokingly threatened to go to his house and steal his computer. His answer? He held up a DVD and smiled. "Go ahead. I have backups."
He tried an external HDD after that AND made multiple DVD copies, storing several copies offsite at his office. Two weeks (two weeks!) after starting this, his external HDD failed.
I stopped by his office and he informed me of the failure. I jokingly threatened to go to his house and steal his computer. His answer? He held up a DVD and smiled. "Go ahead. I have backups."
I used to use CD then DVD as additional backup but the time required was too much and CD's and DVD's life expectancy isn't great. HD's can fail and so can SSD's and that's why you need more than one for redundancy. I wish I had a secondary location to store the backups but I don't (fingers crossed).
This is another area where cloud storage might be problematic -- do they have redundant storage in case a HD/SSD fails? I don't seem to recall any of them saying they do, but I could be wrong. OTH, if cloud storage was priced reasonably enough and again I'd be looking for at least 4TB at this point but within a few years that could easily be 10TB or more, but if cloud storage was reasonably priced it could serve as both redundant storage AND offline storage so if your house burns down the cloud will still be there.
I think a good marketing strategy for cloud storage companies would be as redundant offline storage -- so long as the price is reasonable. But, the other thing I dislike about cloud storage is the data mining that's done and the fact that there's a crap ton of hackers out there that spend all day everyday looking to break into and destroy peoples data. I'd guess the cloud companies have better security than many others, but you the user has little control over that. Way too many companies store highly valuable personal information and don't think they need to spend the time and money to protect that. Hell, even our government agencies have dropped there pants on security.
Brian
This is another area where cloud storage might be problematic -- do they have redundant storage in case a HD/SSD fails? I don't seem to recall any of them saying they do, but I could be wrong. OTH, if cloud storage was priced reasonably enough and again I'd be looking for at least 4TB at this point but within a few years that could easily be 10TB or more, but if cloud storage was reasonably priced it could serve as both redundant storage AND offline storage so if your house burns down the cloud will still be there.
I think a good marketing strategy for cloud storage companies would be as redundant offline storage -- so long as the price is reasonable. But, the other thing I dislike about cloud storage is the data mining that's done and the fact that there's a crap ton of hackers out there that spend all day everyday looking to break into and destroy peoples data. I'd guess the cloud companies have better security than many others, but you the user has little control over that. Way too many companies store highly valuable personal information and don't think they need to spend the time and money to protect that. Hell, even our government agencies have dropped there pants on security.
Brian
Typically, yes.
That is a common use.
That's a fact of life, unrelated to cloud storage specifically. Generally a good practice to store multiple copies of everything. Regardless of whether your storage is local or remote, it's vulnerable.
While it's hard to find an exact number of copies that Amazon makes for your data, they do store it in multiple facilities. If one center goes down, you still have access to your data. If I was doing photography for business, I would include what ShammyH is doing: Uploading new files to AWS S3 and having it automatically age into Glacier storage.
Amazon is not doing data mining against your data stored in AWS. As for hacking, your data is a secure as your credentials. You are the weak link for access to Azure and AWS, they are both very secure. Never store your keys where they can accessed from a public web site. I've read horror stories about developers who included their access keys in source code that was stored in Github. People regularly scan Github looking for credentials.
- Joined
- Jun 21, 2017
- Messages
- 209
- Reaction score
- 78
- Age
- 55
This is what Backblaze is for me. I don't ever want to have to get my data off Backblaze, but if I needed to, it is all there. If my house burnt down, it might take me a couple weeks to get my data back but I'd get it back. And the price for the amount of data I have up there is pretty crazy.
I have had more than enough CDs and DVDs and HDDs for that matter go bad over years. I don't trust consumer grade CD/DVD/BRay burners for archival, period.
I have had more than enough CDs and DVDs and HDDs for that matter go bad over years. I don't trust consumer grade CD/DVD/BRay burners for archival, period.
Similar threads
