Help figuring out how to manage mass video/data storage.
Discussion
Hi All,
Looking for some advice for a work project.
I have to be vague for obvious reasons as I don't want to give away company/client information.
We have 3 locations throughout the UK where we store a certain product on behalf of our client.
This product needs to be 'on' at all times. If the product is ever off, the cost is significant - sometimes more than six figures.
As part of a new process, we are required to film the product every day, so we have a record of the product being on.
The problem is the sheer volume of the product.
At depot 1, we only have 4-5 of the product on site at any one time.
At depot 2, we have up to 20 of the product on site at any one time.
At depot 3, we have up to 50 of the product on site at any one time.
For each product, realistically you are looking at 3-4 minutes to open it up, go inside, video the part that needs to be recorded, and then exit the product.
We are currently already doing this for one of our customers, but they only have a few products in depot 3. Each product is recorded each morning using a phone camera. The videos are uploaded to our shared drive in the cloud, each video is renamed to the product ID code and the date of the recording.
For a small volume of videos each day, this has been ok but now the requirement to do every product, every day, without fail, this is going to be a massive task.
What do you guys think is the best way to do this?
One thought we had was to use a dedicated video camera, buy hundreds of SD cards, and use 1x SD card per day - this would cut out the lengthy process of transferring the footage, adding it to the shared drive, and renaming the files. This comes at a cost and also a bit of a waste. We likely only need the records for 3-6 months, so would be able to keep working through the SD cards.
I feel that recording on a phone, uploading to our drive, and renaming like we currently do will just take way too long. It would pretty much be a full-time job for someone.
Are there any other options that we are missing?
Looking for some advice for a work project.
I have to be vague for obvious reasons as I don't want to give away company/client information.
We have 3 locations throughout the UK where we store a certain product on behalf of our client.
This product needs to be 'on' at all times. If the product is ever off, the cost is significant - sometimes more than six figures.
As part of a new process, we are required to film the product every day, so we have a record of the product being on.
The problem is the sheer volume of the product.
At depot 1, we only have 4-5 of the product on site at any one time.
At depot 2, we have up to 20 of the product on site at any one time.
At depot 3, we have up to 50 of the product on site at any one time.
For each product, realistically you are looking at 3-4 minutes to open it up, go inside, video the part that needs to be recorded, and then exit the product.
We are currently already doing this for one of our customers, but they only have a few products in depot 3. Each product is recorded each morning using a phone camera. The videos are uploaded to our shared drive in the cloud, each video is renamed to the product ID code and the date of the recording.
For a small volume of videos each day, this has been ok but now the requirement to do every product, every day, without fail, this is going to be a massive task.
What do you guys think is the best way to do this?
One thought we had was to use a dedicated video camera, buy hundreds of SD cards, and use 1x SD card per day - this would cut out the lengthy process of transferring the footage, adding it to the shared drive, and renaming the files. This comes at a cost and also a bit of a waste. We likely only need the records for 3-6 months, so would be able to keep working through the SD cards.
I feel that recording on a phone, uploading to our drive, and renaming like we currently do will just take way too long. It would pretty much be a full-time job for someone.
Are there any other options that we are missing?
Having a recording of a device requiring a human to observe it sounds like a flawed plan.
Are there any other gatherable metrics which you can get from the device ? Power being consumed ? Is it/anything it does network connected ?
If you really can't get around wanting video evidence, a stack of wifi webcams and https://www.ispyconnect.com/ ?
Are there any other gatherable metrics which you can get from the device ? Power being consumed ? Is it/anything it does network connected ?
If you really can't get around wanting video evidence, a stack of wifi webcams and https://www.ispyconnect.com/ ?
A permanent camera is not an option, unfortunately.
We move the products around the UK daily. So each day it is a different group of products that are in our depot.
There are about 400 of them in total.
We also must have a video, not a screenshot.
Whilst the product may be on, a certain part of the product needs to be recorded to show it working (as the entire thing could be on, but that part be broken).
We move the products around the UK daily. So each day it is a different group of products that are in our depot.
There are about 400 of them in total.
We also must have a video, not a screenshot.
Whilst the product may be on, a certain part of the product needs to be recorded to show it working (as the entire thing could be on, but that part be broken).
Yes, the first question is can you locate a camera inside the housing & get cabling to it. Six months of storage for 75 video feds could be quite considerable though.
Most places would do this sort of thing with a log file recording some parameter or other - is there anything you can detect that is sufficient to prove it's working?
Most places would do this sort of thing with a log file recording some parameter or other - is there anything you can detect that is sufficient to prove it's working?
TREMAiNE said:
A permanent camera is not an option, unfortunately.
We move the products around the UK daily. So each day it is a different group of products that are in our depot.
There are about 400 of them in total.
We also must have a video, not a screenshot.
Whilst the product may be on, a certain part of the product needs to be recorded to show it working (as the entire thing could be on, but that part be broken).
So you're responsible for housing a product which can fail even thought it's powered on ?We move the products around the UK daily. So each day it is a different group of products that are in our depot.
There are about 400 of them in total.
We also must have a video, not a screenshot.
Whilst the product may be on, a certain part of the product needs to be recorded to show it working (as the entire thing could be on, but that part be broken).
Can the function that breaks be measured/instrumented in any way - current draw, fluid flow, widgets processed, labels printed, etc ?
Are there any ways you can demonstrate/observe evidence of health/activity of the device other than by looking at it ?
Edited by gavsdavs on Thursday 14th December 12:11
It sounds ludicrous, I know.
They have different manufacturers, and some of the manufacturers can remotely monitor, however, they are not prepared to do this, and as part of our contract, we are required to ensure they're running properly.
We do not have access to remote monitoring and the manufacturers will not give us this ability - it's not negotiable.
The product has a compressor that runs constantly and makes a very unique sound, this is how you know if it is working. While some of the compressors have dials that indicate its running, the needle on the dial will fluctuate a lot, which you cannot see in a picture which is why a video needs to be made. Only 5-10 seconds long each.
It is a ridiculous task, and we've not done it before, but as we've expanded and the number of products we now manage has increased so significantly, we cannot risk the damage that can be caused by the compressor not running.
Also, we could not audio record each product because we need to prove that each recording is of the specific product in the recording (they'll all make an identical noise).
They have different manufacturers, and some of the manufacturers can remotely monitor, however, they are not prepared to do this, and as part of our contract, we are required to ensure they're running properly.
We do not have access to remote monitoring and the manufacturers will not give us this ability - it's not negotiable.
The product has a compressor that runs constantly and makes a very unique sound, this is how you know if it is working. While some of the compressors have dials that indicate its running, the needle on the dial will fluctuate a lot, which you cannot see in a picture which is why a video needs to be made. Only 5-10 seconds long each.
It is a ridiculous task, and we've not done it before, but as we've expanded and the number of products we now manage has increased so significantly, we cannot risk the damage that can be caused by the compressor not running.
Also, we could not audio record each product because we need to prove that each recording is of the specific product in the recording (they'll all make an identical noise).
TREMAiNE said:
It sounds ludicrous, I know.
They have different manufacturers, and some of the manufacturers can remotely monitor, however, they are not prepared to do this, and as part of our contract, we are required to ensure they're running properly.
We do not have access to remote monitoring and the manufacturers will not give us this ability - it's not negotiable.
The product has a compressor that runs constantly and makes a very unique sound, this is how you know if it is working. While some of the compressors have dials that indicate its running, the needle on the dial will fluctuate a lot, which you cannot see in a picture which is why a video needs to be made. Only 5-10 seconds long each.
It is a ridiculous task, and we've not done it before, but as we've expanded and the number of products we now manage has increased so significantly, we cannot risk the damage that can be caused by the compressor not running.
Also, we could not audio record each product because we need to prove that each recording is of the specific product in the recording (they'll all make an identical noise).
For a product which could represent a 6 figure loss if the compressor fails, you'd think they'd provide some way of monitoring/observing health. Not a quick fix, a compressor is driving a pressurised system. Can you get a pressure sensor into the system ?They have different manufacturers, and some of the manufacturers can remotely monitor, however, they are not prepared to do this, and as part of our contract, we are required to ensure they're running properly.
We do not have access to remote monitoring and the manufacturers will not give us this ability - it's not negotiable.
The product has a compressor that runs constantly and makes a very unique sound, this is how you know if it is working. While some of the compressors have dials that indicate its running, the needle on the dial will fluctuate a lot, which you cannot see in a picture which is why a video needs to be made. Only 5-10 seconds long each.
It is a ridiculous task, and we've not done it before, but as we've expanded and the number of products we now manage has increased so significantly, we cannot risk the damage that can be caused by the compressor not running.
Also, we could not audio record each product because we need to prove that each recording is of the specific product in the recording (they'll all make an identical noise).
TREMAiNE said:
...I feel that recording on a phone, uploading to our drive, and renaming like we currently do will just take way too long. It would pretty much be a full-time job for someone...
Get a phone app built that will take the recording, and automatically name it correctly and upload it.However given the manual labour involved in the process what's about just adding the camera to all 400 units and just actively monitoring the ones you need to.
It sounds to me like any system that involves phones, people, and renaming files on SD cards is a recipe for complete disaster (imho). Especially when there are 6 figure losses if it don't work.
I'd be looking to have an automated-as-possible process that can be monitored remotely.
i.e something like some dedicated hardware that records and uploads a clip to some cloud storage automatically.
Ideally left setup and fully automated, but worse case triggered by a button pushed by an operator when pointed at the right thing.
The problem might be that to do this, it will require some up front investment (hardware, software, and development), as well as ongoing running costs - Although these might be offset against the saving of not having a human doing it all day every day instead.
What sort of budget might exist for automating the solution?
I'd be looking to have an automated-as-possible process that can be monitored remotely.
i.e something like some dedicated hardware that records and uploads a clip to some cloud storage automatically.
Ideally left setup and fully automated, but worse case triggered by a button pushed by an operator when pointed at the right thing.
The problem might be that to do this, it will require some up front investment (hardware, software, and development), as well as ongoing running costs - Although these might be offset against the saving of not having a human doing it all day every day instead.
What sort of budget might exist for automating the solution?
I've presented the idea of a pressure sensor system to see if that could be explored.
I've made a case for the fact that this just isn't a viable option long term, especially over the Christmas period where 2-3 times the amount of products are being stored.
It appears that there is no obvious, easy answer that I'd not thought of.
Some good ideas from you guys, really appreciate the input with it being such a vague description from me.
I've made a case for the fact that this just isn't a viable option long term, especially over the Christmas period where 2-3 times the amount of products are being stored.
It appears that there is no obvious, easy answer that I'd not thought of.
Some good ideas from you guys, really appreciate the input with it being such a vague description from me.
Given that it involves a compressor, is mobile, changes & Christmas has a significant impact on the units to be monitored you could guess it's something like refridgerated trailers of various brands. That's not an easy job, especially if the suppliers won't modify them so they can be monitored properly.
I'd suggest whoever signed that contract gets the job of doing the recording.
I'd suggest whoever signed that contract gets the job of doing the recording.
If you need any thoughts on how/where to store large amounts of data I'd be happy to have a chat. I work for an IT reseller and whilst I don't know if I can help with the data capture part, we have done some interesting projects for customers who had requirements for high capacity, high-performance storage for multimedia scaling into petabytes. Cloud providers will get expensive very quickly and with video you won't be able to compress much or dedupe so you get no benefit from data reduction services unfortunately. If you have to store and maintain the data on behalf of your client(s) for any length of time it would probably be worth investigating private cloud or even on-prem hardware solutions if you have suitable connectivity/bandwidth.
I don't know if PH has fixed the issue with my profile not sending me messages via the 'Email me' link - if you do send me a message, pop a note here and I'll check.
I don't know if PH has fixed the issue with my profile not sending me messages via the 'Email me' link - if you do send me a message, pop a note here and I'll check.
Edited by Funk on Thursday 14th December 13:44
Is there scope to have some kind of bar code or QR code on the product? If so it wouldn't be that difficult to have someone write an app that lets you scan the barcode, record the video, and then save/upload the file with a name that relates to the info on the barcode. If the product has some other obvious serial number you may be able to do similar with text recognition but bar/QR codes would be a lot simpler. This would let you automate a lot of the process rather than rely on someone naming the files correctly.
What is the running compressor doing - keeping a container cold or similar?
If so, why not monitor the desired end result (the temperature of the container, for example) and send regular updates to confirm the temperature with alarms when it is out of limits?
It may be easier to fit remote temperature monitors to all 400 units and have them send messages to a central service (could be via internet - mobile network or WiFi - or even SMS).
If so, why not monitor the desired end result (the temperature of the container, for example) and send regular updates to confirm the temperature with alarms when it is out of limits?
It may be easier to fit remote temperature monitors to all 400 units and have them send messages to a central service (could be via internet - mobile network or WiFi - or even SMS).
Please feel free to pull an initial idea apart...
I was wondering if you could do something with one more or mini CCTV cameras.
Blink Mini Wireless cameras are battery powered and pretty cheap (currently £30).
https://www.amazon.co.uk/new-blink-indoor-wireless...
A Sync 2 module (£30) supports a 256GB USB flash drive for local storage
https://www.amazon.co.uk/dp/B08BJ9ZCZC
You can pay £80/year for cloud storage but that only has 30 day retention. Videos recorded on the Sync module are MP4, so can be reviewed away from the Blink apps. Not sure how big the clips are but you could either rotate a few USB drivers and / or copy the files onto other storage for longer term retention (eg a NAS).
One issue is knowing which clip you've recorded relates to which customer device. Do you have a log which lists the date / time each item was checked? That could be cross referenced to a video clip.
Things to check - how quick / easy it to turn on (or pick-up) a camera and then making the recording.
I was wondering if you could do something with one more or mini CCTV cameras.
Blink Mini Wireless cameras are battery powered and pretty cheap (currently £30).
https://www.amazon.co.uk/new-blink-indoor-wireless...
A Sync 2 module (£30) supports a 256GB USB flash drive for local storage
https://www.amazon.co.uk/dp/B08BJ9ZCZC
You can pay £80/year for cloud storage but that only has 30 day retention. Videos recorded on the Sync module are MP4, so can be reviewed away from the Blink apps. Not sure how big the clips are but you could either rotate a few USB drivers and / or copy the files onto other storage for longer term retention (eg a NAS).
One issue is knowing which clip you've recorded relates to which customer device. Do you have a log which lists the date / time each item was checked? That could be cross referenced to a video clip.
Things to check - how quick / easy it to turn on (or pick-up) a camera and then making the recording.
jeremyc said:
What is the running compressor doing - keeping a container cold or similar?
If so, why not monitor the desired end result (the temperature of the container, for example) and send regular updates to confirm the temperature with alarms when it is out of limits?
It may be easier to fit remote temperature monitors to all 400 units and have them send messages to a central service (could be via internet - mobile network or WiFi - or even SMS).
This has legs. The compressor is trying to achieve something, for example if it's to refrigerate, monitor temperature. We are guessing the requirement a bit but that's how I was thinkingIf so, why not monitor the desired end result (the temperature of the container, for example) and send regular updates to confirm the temperature with alarms when it is out of limits?
It may be easier to fit remote temperature monitors to all 400 units and have them send messages to a central service (could be via internet - mobile network or WiFi - or even SMS).
Gassing Station | Computers, Gadgets & Stuff | Top of Page | What's New | My Stuff