DCE 7.3.1 Backup options

[DCIM_Support]

Our backups have become rather large as we are holding several years of data.  Our backup time is usually in excess of 20 hours as it has to pass across several Networks & Corporate firewalls etc.  As we would like to be able to access historical data if required, the current purge options do not offer any retention options.  It would be great to be able to extract old data into a database & maybe just keep a current years data in the backup.  

Does anybody have any similar experiences/solutions that they have encountered?     

(CID:106205461)

[DCIM_Support]

Hi Jeff,

The backup is designed solely to be available should the server suffer a critical issue where the server needs to be restored or if a new server needs to be spun up with the data from the old server. There are no utilities to access the data outside of the DCE server.

If you need to save data for longer periods of time, I would suggest using the option for a remote repository. 

As for the backup time being 20 hours, that sounds awfully long. Have you tried a system on the private LAN of the DCE server (assuming hardware or VM with 2 NICs)? Is the DCE server so active, usually with incoming surveillance images, that the simple act of storing this data is causing these long times?

Steve

(CID:106205518)

[DCIM_Support]

Hi Steve. The backup is in excess of 30 gigs with all the historical data & it has always taken many hours to complete. So if we went to the Remote Repository , obviously all the data is stored there & we just then do a config backup? In case of a critical issue & we had to restore the server, are all the monitored devices present in a config backup & is it just a matter of reconnecting that remote repository to get the historical data back? Thanks. Jeff.

(CID:106205544)

[DCIM_Support]

Hi Jeff, If you use a remote repository and tell DCE to just use remote, then you would indeed have all the data stored there. Yes, a config backup includes all devices as well as server configurations. Everything except the data. Steve

(CID:106205588)

[DCIM_Support]

Hi Jeff,

William in support (I guess your region) e-mailed a comment about this thread and I was about to respond but thought I'd put it all here as well. You wrote:

***************************************************************************

I got the below answer off Steven Marchetti on the forum with regards to our back-ups. I’ve got our people looking into seeing if we can spin up another Windows or Linux VM next to our existing VM to use as a remote repository for our data. Least we wouldn’t have endless hours of back up time happening that way.

Only trouble would be if we made it Windows based, we would be back to square one with not being able to mount the share & NFS on Linux is also giving us a few reliability headaches.

If we went that way, does the existing data automatically move to the repository when we configure it which would obviously take some time or do we back it up, load a basic version of DCE 7.3.1, add the repository & then restore from the back-up & the data moves to the remote repository as part of the restore process?

***************************************************************************

I then responded to him but thought maybe I should also respond her3e so this is what I wrote to him:

If they're having issues connecting to network shares, that is something they have to fix. The delays in backups or remote repositories may indeed be effected similarly but there is little we can do about that on the DCE side. Perhaps a remote repository but on the private LAN? I was unaware they had any mounting specific issues.

Existing data does not automatically move to the remote repository but there is an option to cause that to happen. They can choose to use remote storage only as well as an option to migrate to remote:

 

Again, if they have connectivity issues, this may not solve their issues either as DCE may have issues still. If it:
"has to pass across several Networks & Corporate firewalls etc "
That sounds like they already know the issue and what they have to do to correct it.

Steve

(CID:106205877)

[DCIM_Support]

Hi Steve. Here is the error message I get when I try to create a remote repository via a Window Share. I have confirmed that I am successfully logging into the domain. Can I get a definite answer what this error message means?

Windows Mount Error message.JPG

(CID:106206122)

[DCIM_Support]

Hi Jeff, It just says invalid argument. There are no specifics so I can't really say what is incorrect but it sounds like settings. The best option is to share a folder on your laptop and give yourself (specifically) full rights to that share. If you can add that as a repository then we know it's something specific with the share. If you can, you may want to contact support while you're doing this and if they are able to remote in using webex, they may be able to better see what you're doing and understand what is happening. Steve

(CID:106206210)

[DCIM_Support]

Hi Steve. All good in theory but I cant just add a laptop to the Network, that would be great & easy. The Share settings need Domain names, an access account, be able to get through corporate firewalls etc etc. Also our DCE is a VM sitting in a secure Corparate LAN unaccessable to the outside world so I can't give anyone access to it remotely except on site via APC client. What further info do you need to investigate?

(CID:106206233)

[DCIM_Support]

Hi Jeff, If you have known issues accessing remote shares and the only option is through these known firewalls and across other networks, I'm not sure what else to try. A local system not using multiple networks and firewalls is the only test I can think of to make sure the issue has to do with the environment. To ensure the DCE is not part of the issue, you can try increasing the resources (don't increase the default drive,, you can only add drives) such as local drive space, RAM, and processors. If you take a look at the details on the storage settings page. How is it distributed? If you have most of your percentage going to "other" (wait a minute on this page to determine), the system's IO is likely so high the resources it has are not allowing it to sort the data. That means maybe too much NetBotz camera image or polling devices too often. Steve

(CID:106206381)

[DCIM_Support]

Hi Steve. We successfully back up currently across Networks via NFS to a Linux server. Given that we can successfully do that would suggest that we don't have a data sorting issue. I am really not that keen of touching any settings on the VM as it took us a couple of tries to the get the VM going in the first place & we had a back up server running then, I don't have that now. I am more inclined to go back to my original idea & spin up a second test Windows server VM in the same Network as our VM & see if we can access it. That's about as close to a local system as I can get. We don't any Netbotz cameras attached to our server & device polling is set at default. I have enclosed a shot of our storage figures.

DCE Storage settings.JPG

(CID:106206427)

[DCIM_Support]

Hi Jeff, With all the data showing in the "other" category, assuming you gave it a moment to order things properly that usually means the system is too busy to be able to figure out where things are going. As I mentioned previously, this usually indicates a high I/O or low resources. If that is the case,, you have 2 choices, increase resources or lower the I/O. If you will not add memory or CPU, then lowering how fast the data is coming in is the only option. If data related, it is usually NetBotz camera surveillance that causes it. If that is the case, lowering things like framerates, resolution, or even times that the cameras are recording will help. Please note that the system is designed around surveillance of areas where motion is the exception and not the rule. If you are monitoring an entryway, it's designed to watch when nobody is supposed to be coming in and not when the doors are opening to a line of people around the block. If you are monitoring SNMP devices only, perhaps slowing the poll rate would help. If you're polling 1000 devices each with 1000 sensors every minute, that can be a lot of data, if there are any issues with communicating over a large network, timeouts and retries can increase the I/O for this. Lowering the scan times in this type of case can lower I/O. Steve

(CID:106206505)

[DCIM_Support]

Just to add to the above, this does not mean the data is not ending up in the right place, it means the system is simply too busy to show you. It is simply an indication of a server that is too busy. Taking 20 hours to back the system up may be a similar indication.

(CID:106206507)

[DCIM_Support]

Steve, as I previously mentioned, we don't any Netbotz cameras attached to our server & device polling is set at default. I am happy to try increasing the CPU & Ram as you suggest. If as you say, the server is too busy, why does it mount via NFS & not via a Windows Share,

(CID:106206508)

[DCIM_Support]

Hi Jeff, I must have missed the lack of cameras, with the fact that posts are not all linear in this system, I sometimes have a hard time seeing everything. Resources may have little to do with one mount VS the other however we know the system is too slow. That can cause all kinds of things.I'm just trying to suggest resolving the parts we can figure out.

(CID:106206509)

[DCIM_Support · Accepted solution]

It seems there is a system generated $ at the end of the Share name & good old Windows needs that to be included in the share name for it to find its share.

Most other systems it doesn’t apparently.  Problem is now solved.

Thanks.

(CID:106206946)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.