DCE Backup Fails - Windows Share on NetApp NAS

[DCIM_Support]

When DCE backup is configured to backup to a windows share on our NetApp NAS it will always fail. I believe it is  caused by rsync trying to change the file times, see the final part of the backup logs below.

Any ideas as to a workaround for this issue would be greatly appreciated.

Richard

5/5/15 19:08:38.552 ERRR - Backup Log: rsync: failed to set times on "/data/mount/backup/backup_eccvas1148_20150505190830_7.2.6/opt/netbotz/data/ddfs/.": Permission denied (13)

(com.netbotz.server.services.backup.impl.BackupJob)

5/5/15 19:08:38.552 ERRR - Backup Log: rsync: failed to set times on "/data/mount/backup/backup_eccvas1148_20150505190830_7.2.6/opt/netbotz/data/ddfs/modbus": Permission denied (13)

(com.netbotz.server.services.backup.impl.BackupJob)

5/5/15 19:08:38.552 ERRR - Backup Log: rsync: failed to set times on "/data/mount/backup/backup_eccvas1148_20150505190830_7.2.6/opt/netbotz/data/ddfs/snmp": Permission denied (13)

(com.netbotz.server.services.backup.impl.BackupJob)

5/5/15 19:08:38.552 ERRR - Backup Log: rsync: failed to set times on "/data/mount/backup/backup_eccvas1148_20150505190830_7.2.6/opt/netbotz/data/ddfs/snmp/pending": Permission denied (13)

(com.netbotz.server.services.backup.impl.BackupJob)

5/5/15 19:08:38.552 ERRR - Backup Log: rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]

(com.netbotz.server.services.backup.impl.BackupJob)

(CID:96044994)

[DCIM_Support]

Hi Richard,

We've seen this a few times specifically with Netapp. There is no way to alter the settings in DCE on how we write. The only thing you can change would be to reconfigure the system as a Windows share (CIFS) or use another system. I'm assuming if you test with a Windows server or laptop it works, correct? I do not believe newer versions of DCE will rectify this issue but out of curiosity, what version of DCE are you using? What version of rsync is the netapp server running?

Steve

(CID:96763938)

[DCIM_Support]

Hi Steve, yes if we backup to a normal windows server we see no errors. As far as I'm aware the NetApp is configured as a CIFS share and as an Enterprise we would not be able to alter the way the shares are configured. To get around it we have to configure a NFS share on the NetApp that is unrestricted and then just restict access to the DCE appliances. This then becomes difficult to manage with regards to removing old backups etc given that it cannot be done from within DCE itself. If this scenari has been encountered a number of times already then i would think it is time Schneider looks at a fix or an alternate way to write the backups. Richard

(CID:96763966)

[DCIM_Support]

After some more researching, I came across this forum comment that seems to indicate that the issue was caused by the rsync upgrade in the DCE upgrade from version 5.0 to 5.1 http://forums.apc.com/spaces/9/struxureware-for-data-centers/forums/general/7941/backup-issue-to-a-n... Seems like it has been an issue for so long now and everyone is just finding a work around rather than Schneider resolving the issue.

(CID:96763969)

[DCIM_Support]

Richard, I've seen similar issues and although I can not state unequivocally what version of DCE or versions of rsync are used in each version and potentially causing issues, I can say that this has indeed been seen and has been raised above my level in tech support. It could be that once we get to this point some might just opt for the workaround but I will make note of this thread and ask that they look at it. Steve

(CID:96764049)

[DCIM_Support]

Hi guys

I'm also having trouble with the backup of DCE to a Netapp NFS or CIFS share.  I can backup successfully to any Windows share I like, but have not been able to complete any successful backups on a Netapp array.  I've tried every configuration of setting on the array's NFS Exports or CIFS shares, and can't get any combination to work.

The backup actually commences, creates a new backups folder and writes a whole bunch of files to the folder.  this indicates to me that permissions, and the basic mount of the share are correct.

It seems to fail when it's writing the large TAR file in the root of the backup folder.

Also I get the same behaviour when attempting to create a Remote Storage location as well.  I can create it successfully on any windows share, but not a NETAPP NFS or CIFS share.  Yet every other system we have doesn't have a problem with these types of shares.  DCE is the only one that doesn't want to correctly access the NFS/CIFS on the Netapp.

It'd be great if this was resolved in future releases.  I'm currently on 7.2.1 but upgrading the the 7.2.6 VM shortly.

Cheers

Matt

(CID:96764054)

[DCIM_Support]

Hi Matt,

As with Richard, if it works on a Windows system, we can rule out data corruption being the cause and it is the communications between DCE and the Netapp server. Although it's a good idea to keep up to date, from what I've seen it likely won't make a difference with the saving to Netapp. 

I have forwarded the post on to the next level and have yet to receive a reply but should they like to troubleshoot with you or with you Richard, can I assume you'd be willing to spend the time to work through it knowing that whatever we find (if it is indeed an issue with DCE) will likely not be implemented until a future release?

Steve

(CID:96764119)

[DCIM_Support]

Sure. I have responsibility for both the DCE instance and the Netapp storage arrays, so I can make changes to how both operate to see if we can resolve the issue. I also have the ability to spin up a test instance of DCE as a VM, and test the operation of a new version (hopefully there will be one) with the NetApp array. I'll going to see if I can get some logs out of the NetApp today. Let me know if there is someone that I can send these to. Additionally - a great feature would be the ability to automatically purge backups older than a certain date, in the same manner the storage repository can be. Thanks in advance 🙂 Richard - if you've successfully managed to get an NFS share working, can you please ask your storage techs for the config and send it to me? I've been unable to get either CIFS or NFS working at all. cheers

(CID:96764150)

[DCIM_Support]

Hi Matt, To get the NFS share to work with DCE, the directory permissions for the share had to be set to 777 then we locked down the export to the specific hosts of the DCE appliances only. Like you I have also requested a way of managing old backup files as when you uses NFS share in this way it becomes difficult to manage adding and removing client host names all the time just to manage old backups. http://dcimsupport.apc.com/questions/96044992/dce-backup-management---feature-request Richard

(CID:96764151)

[DCIM_Support]

Hi Richard, To be clear, if you set the directory permissions on the NetApp box to 777, it does work? Steve

(CID:96764201)

[DCIM_Support]

Hi Steve Yes it does work in this way. We are currently backing up 5 DCE appliance to Netapp NFS shares configured in such a way. Richard

(CID:96764240)

[DCIM_Support]

Hello - We have been experiencing the same issues backing up to a CIFS (EMC Isilon) or a NFS mounted drive on RHEL. The backups (full and sync) fail almost 90% of the time but the configuration backup works fine. We had opened a ticket with APC but I'm getting no where with tech support. We are at DCE version 7.2.1 and don't mind upgrading to the latest version if APC can assure us that the upgrade will fix the issue. As a test, I spinned a windows vm with NTLMv1 and the backups work fine. But allowing NTLMv1 long term is not an option for us.

(CID:99156400)

[DCIM_Support]

Hi Amit, Your case with Cory has been brought to the attention those at higher levels and they are the ones that asked that the system be updated to current versions. I can not say that this will resolve your issue however engineering does like to make sure systems are up to date so we don't have to worry about aspects of the system that should already be updated. I'm also assuming you have also tried the suggestions Richard made above with setting permissions to 777 as I have seen issues with NetApp systems and this is one of the first resolutions I have seen in such cases. As for the NTLM version 1 vs version 2 aspect, this is a known issue and NTLM version 2 is not compatible with DCE at this time. Steve

(CID:99156408)

[DCIM_Support]

Hi Steve - We changed the permissions to 777 and unfortunately it didn't work.

(CID:99156411)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.