DCE Backup

[DCIM_Support]

Good day

Please assist.

DCE backups are failing when it reaches 98 - 99% completed.

According to DCE backup logs, there are many errors very similar to below;

 ERRR - Backup Log: LOG: Wrote /data/mount/backup/backup_"Server name"1_##########_7.4.1/backup.properties with values: Full 7.4.1 ################ true ##################

(com.netbotz.server.services.backup.impl.BackupJob)

ERRR - Backup Log: LOG: Backup finished 06-08-2017 02:24:22

(com.netbotz.server.services.backup.impl.BackupJob)

Please assist.

(CID:128758118)

[DCIM_Support]

Hello Juice,

Have you checked that there is enough space on the network share you are backing up to. What type of network share are you backup up to? If possible can you share the logs from the server.

Thanks & Regards

(CID:128758650)

[DCIM_Support]

Hi John

Thank you very much for your response.

Thus far we cleared up more space with no success. We actually store the backup on a local windows share on a workstation. The backup has been running very smoothly until recently. There we no major changes made.

Given security protocols and procedures im unable to forward logs, is there anything that i should look out for ?

Any response is greatly appreciated.

Yusuf

(CID:128758681)

[DCIM_Support]

Hi Yusuf,

On the DCE web client, go to the Logs screen, and check if there are any errors related to the backup in the nbc.xml logs for the last time you ran the backup. There could be corrupt data, or maybe there was a network interruption during the backup. But without being able to look at the logs myself, it is difficult to say for sure.

Regards

(CID:128759242)

[DCIM_Support]

Thank you very much John.

I have browsed through the logs and is alot of "red" and "ERRR" statements but nothing specific that i can say is the cause of the failure. 

below is a "safe" snippet of errors we are getting, i hope you can assist

2/20/18 13:05:40.681 ERRR - Backup Log: io timeout after 330 seconds -- exiting
(com.netbotz.server.services.backup.impl.BackupJob)

2/20/18 13:05:40.681 ERRR - Backup Log: rsync error: timeout in data send/receive (code 30) at io.c(200) [sender=3.0.6]
(com.netbotz.server.services.backup.impl.BackupJob)

2/20/18 13:05:40.681 ERRR - Backup Log: ERROR: Failed to sync the local repository 06-14-2017 23:57:49
(com.netbotz.server.services.backup.impl.BackupJob)

2/20/18 13:05:40.681 ERRR - Backup Log: LOG: Failed to backup the local repository to /data/mount/backup/backup_"Servername"20170614160000_7.4.1 06-14-2017 23:57:49
(com.netbotz.server.services.backup.impl.BackupJob)

2/20/18 13:05:40.694 ERRR - Backup Log: rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
(com.netbotz.server.services.backup.impl.BackupJob)

Many thanks

regards

(CID:128759405)

[DCIM_Support]

Hi Yusuf,

Sorry for my late reply. That looks like an input/output or network error and the connection between the DCE and backup target is getting interrupted. Is this a very large backup file and is is taking a long time? 
Can I ask you to download the capturelogs from the server using the instructions below.

1. For ISXC/DCE version 6.x or higher, type in the following line:  http:///capturelogs
2. The last capture server file generated will be listed.  You should generate a new capture server log by clicking on ""Create Server Log Archive"".  Let the new file generate.  Once complete, download the file.

When the file is download, unzip it and drill into the /data/logs folder and have a look at the nbcBackup.log file. Can you share the most recent part of that file.

Thanks & Regards

(CID:128760352)

[DCIM_Support]

Yes it is a large file. Its 428Gig.

Unfortunately i am unable to share the entire file but below is a snippet.

I hope this helps you to determine the issue.

"

18:44:24
LOG: Local ddfs sync finished 02-21-2018 18:44:25
LOG: Local repository sync started 02-21-2018 18:44:25
file has vanished: "/data/local/sensor2/B616354/B616354_nbModbusEnc73CD75F6/2018/sensor.052-journal"

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/data/mount/backup/backup_"Servername"20180221153000_7.4.1/data/local/sensor2/B616354/B616354_nbSNMPEnc884593C6/2014/.sensor.061.AQLRvN": Input/output error (5)
rsync: connection unexpectedly closed (572238 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
ERROR: Failed to sync the local repository 02-21-2018 19:04:59
LOG: Failed to backup the local repository to /data/mount/backup/backup_"Servername"_20180221153000_7.4.1 02-21-2018 19:04:59
LOG: Backup started 02-21-2018 19:05:04
LOG: Full backup 02-21-2018 19:05:04
LOG: Wrote /data/mount/backup/backup_"Servername"_20180221153000_7.4.1/backup.properties with values: Full 7.4.1 Servername false 02-21-2018 19:05:04
LOG: Local ddfs sync started 02-21-2018 19:05:04
LOG: Local ddfs sync finished 02-21-2018 19:05:05
LOG: Local repository sync started 02-21-2018 19:05:05

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/data/mount/backup/backup_"Servername"_20180221153000_7.4.1/data/local/sensor2/B616354/B616354_nbSNMPEnc884593C6/2014/.sensor.348.ZU2e2M": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(730) [receiver=3.0.6]
rsync: connection unexpectedly closed (250321 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
ERROR: Failed to sync the local repository 02-21-2018 19:10:50
LOG: Failed to backup the local repository to /data/mount/backup/backup_"Servername"20180221153000_7.4.1 02-21-2018 19:10:50
LOG: Backup started 02-21-2018 19:10:55
LOG: Full backup 02-21-2018 19:10:55
LOG: Wrote /data/mount/backup/backup_"Servername"_20180221153000_7.4.1/backup.properties with values: Full 7.4.1

Servername false 02-21-2018 19:10:55
LOG: Local ddfs sync started 02-21-2018 19:10:55
LOG: Local ddfs sync finished 02-21-2018 19:10:55
LOG: Local repository sync started 02-21-2018 19:10:55

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/data/mount/backup/backup_Servername_20180221153000_7.4.1/data/local/sensor2/B616354/B616354_nbSNMPEncA6A7E2C2/2017/.sensor.206.UAYzcw": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(730) [receiver=3.0.6]
rsync: connection unexpectedly closed (1017941 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
ERROR: Failed to sync the local repository 02-21-2018 19:34:40
LOG: Failed to backup the local repository to /data/mount/backup/backup_Servername_20180221153000_7.4.1 02-21-2018 19:34:40
LOG: Backup started 02-21-2018 19:34:40
LOG: Full backup 02-21-2018 19:34:40
LOG: Wrote /data/mount/backup/backup_Servername_20180221153000_7.4.1/backup.properties with values: Full 7.4.1 Servername false 02-21-2018 19:34:41
LOG: Local ddfs sync started 02-21-2018 19:34:41
LOG: Local ddfs sync finished 02-21-2018 19:34:41
LOG: Local repository sync started 02-21-2018 19:34:41

rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: close failed on "/data/mount/backup/backup_Servername_20180221153000_7.4.1/data/local/sensor2/B616354/B616354_nbSNMPEncB229E9B5/2015/.sensor.317.K4SvDG": Input/output error (5)
rsync error: error in file IO (code 11) at receiver.c(730) [receiver=3.0.6]
rsync: connection unexpectedly closed (610532 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
ERROR: Failed to sync the local repository 02-21-2018 19:49:50
LOG: Failed to backup the local repository to /data/mount/backup/backup_Servername_20180221153000_7.4.1 02-21-2018 19:49:50"

In the interim we will create another share to a different target workstation and re-run the backup and will feedback shortly.

Many thanks for all your effort = very much appreciated.

(CID:128760433)

[DCIM_Support]

Hi John

We have created another share to a separate workstation, however the results are not great. The backup has yet again failed.

We have verified enough space.

We have rebooted the DCE Server.

We have confirmed with the network team that the network-port is error free.

We are now thinking of reformatting the backup hard disk and re-partition to NTFS\FAT32 ? = any suggestions?

Question = Is there anyway to verify that the connection between client and server is 100% during the backup ?

regards

Once again - thank you for all your effort = very much appreciated.

(CID:128761426)

[DCIM_Support]

Hi Yusuf,

Sorry for the late reply. NTFS would be better to use than FAT32 as it is the newer of these filesystems. The only way to check if the client connection is lost during the backup is by looking at the Event Log in the web client. Check the Security Category logs for the time period that the backup was running on.
Another option might be to try doing the backup onto a Linux machine, if you suspect issues with the Windows machine you are using now. If you have not tried already, can you also try running a Synchronized backup to see if that completes successfully or not.

Regards

(CID:128762482)

[DCIM_Support]

Firstly, a big thank you for your reply John. It is much appreciated.

We had a look at the backup log and it seems to be a timeout setting. We not sure if its on the DCE server or the target server. We have escalated the issue to schneider technicians and is attending to the issue.

if you prefer, i will post the outcome and RCA.

Kind regards

Juice

(CID:129404929)

[DCIM_Support]

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.