EcoStruxure ITA Advisor - Won't Login

[Tafadzwa]

Hi All,

I have an EcoStruxure IT Advisor v 9.3.4 Server that I initiated a Backup as I usually do via the Web Client. The Backup did NOT complete successfully and now I am not able to Login into the Server via the Web client. I get the below Error:

When I try Logging in via the Desktop client I get the below Error:

I am able to Login via the Console and I can confirm that the Credentials and IP address are correct.

Please advise on how to go about resolving this issue?

Any feedback will be greatly appreciated!

Regards,

Tafadzwa.

[Jef · Accepted solution]

Hi Tafadzwa,

You mentioned that the backup did not complete successfully, so I guess the local disc, especially the root '/' partition (may hold /data/backup and /tmp folders), might be full or not enough free disc space left.
You might want to login to server Webmin interface (https://<ITA server IP>: 10000) and then go to:
 EcoStruxure IT Advisor > Status and see if the Web, Application and Database Status are okay/Running.

System Information > Disc Usage and see if there is enough free space in root partition.

You can also ssh (or console) login to the server and run the following command to verify the disc space:

df -k

If there is not enough free space left, then you might want to free some space.

Usually backup files (default location: /data/backup) and old upgrade iso file(s) (might be in /data folder) may occupy a considerable amount of the storage capacity. You might want to remove the old iso and backup files and then reboot the server and see if that helps.

Note: Before removing the files, you might want to save them somewhere safe (like a network drive or laptop for safekeep). And please also note that the latest backup file(s) might be incomplete and useless (check/compare the file size).

Furthermore, according to system requirements, it is recommended to assign at least 200 GB storage capacity for an ITA server. If your server should have been installed on a smaller disc, then I would suggest rebuilding the server with larger storage (and assign the most space to root partition).

Kind regards,

Jef

[Tafadzwa]

Hi @Jef 

Thank you for your feedback.

I managed to delete some old backups however the Root partition is still full and I still cannot run a Backup.

I cannot locate the Folder that has old iso files.

Perhaps I can look into mounting the Backups Remotely.

Regards, 

Tafadzwa.

[gsterling]

Hello Tafadzwa

Can you send a screenshot or copy of the output of the command "df -k" from your ITA server? It may be possible to clear out some other files so you have enough space for backups.

It is also possible to configure backups to be saved on a remote share. To do that you'll need to mount a windows or NFS share, then change backups settings in the "backup / restore" menu in the ITA web client UI (under the administration menu). To mount a SMB (windows) or NFS share, you may need to install the nfs-utils or smb-utils packages.. these installers are included on the ITA offline installer or if you have an online ITA server they could be installed directly.

Regards

Greg Sterling

[Tafadzwa]

Hi @gsterling ,

Thank you for your feedback. It is much appreciated!

Please see attached screenshot as per your request.

[gsterling]

100GB is large enough for many customers. How large is this ITA installation? i.e. how many racks and which licenses? 100GB may not be large enough for an install which is thousands of racks and keeping long capacity and audit history.

A few items/recommendations I can make immediately.

• Change backup settings to keep few backups (default is 30 days) ... perhaps reduce this value to 10 days? This change can be made from the backups settings view in the ITA web client UI under Administration->backup and restore.
• SSH to the ITA server, and as a root user delete files ending in .gz from /opt/jboss-as/standalone/log  (these are archived ITA log files)
• SSH to the ITA server, and as a root user delete older files ending in -YYYMMDD from /var/log (logs in this folder are archived with a datestamp), I'd keep the most recent file but delete the older ones.
• If you have had failed backups they may appear in /tmp in temporary folders with randomized names. For example the folder might be named something like this: /tmp/uQXd9PgDAJ ... you can delete these folders if any exist ... do not delete all folders in /tmp as some OS processes use temp as well. But in the case of temporary backup directories the above example can be removed with the command "rm -r -f /tmp/uQXd9PgDAJ"
• If your ITA server has capacity licensing, the server is storing capacity data each day for historical reporting. The customer can choose to purge this data as it ages from the ITA desktop client, tools menu, preferences, then the capacity history view. The default is to keep data forever, if they set it to a year (just an example) capacity history older than a year is purged. This can reduce the size of their ITA database and backup file.

Regards

Greg Sterling

[Tafadzwa]

Hi @gsterling ,

Thank you for your response. I will implement all your suggestions and try to run the Backup Locally again.

See below screenshots to answer your questions:

ITA Installation size:

Backup Schedule: Daily Backup Disabled. Backups are run every 3 months as changes to the Rooms are done once in a while.

[gsterling]

For a installation that size 100GB of disk space should be ok.

In your backups setting, you have told ITA to keep backups for 185 days. Have you also changed the actual backups schedule to run less often than daily? This can be done via the webmin UI. If not, then ITA is creating backups each day and keeping them 185 days before they are purged.

Regards

Greg Sterling

[Tafadzwa]

Hi @gsterling @Jef 

I wish to circle back to this issue.

I finally got a Remote Server for mounting Remote Backups however after following these steps (https://community.se.com/t5/ITA-server-settings/File-sharing-between-IT-Advisor-and-your-network/ta-...), I am getting an Error message that "No space is left on device". The Remote Server has way more than enough storage.

Please see screenshot below and advise on how to successfully store the Backup Remotely.

Regards,

Tafadzwa.

[Tafadzwa]

I forgot to upload one screenshot. Backup sizes were usually around 4.1GB but after deleting all Backups saved on the Server, I am still getting a Backup Error.

[Jef]

Hi Tafadzwa,

System (ITA) uses the /tmp directory to create a backup of the database before writing the final backup file to the specified backup location:

It appears there is not enough space left at the root portion (where the /tmp is located).

I would recommend to rebuild the setup by using a larger storage capacity, the minimum requirement is 200 GB. 

For now, I would try to free some space (from the root '/' partition):
- check the /data/backup folder. Backup files may be uploaded/copied to somewhere else (network drive or another computer/laptop) for safekeeping. And then clear/delete the old backup files. 

- check the /data folder, just in case if there should be any .iso files that might have been used for ITA upgrade. Remove the possible iso file(s).

- check the /opt/jboss-as/standalone/log folder, you might want to clear the old .gz files (which I would not have suggested under normal conditions). 
- check the /home/<username> (if mounted under the root partition) and see if there are any uploaded large file(s) that can be removed.

Kind regards,

Jef

[Jef]

Hi Tafadzwa,

To be able to run a backup job, you would need at least 9 GB ( 2x 4.1 GB) free space at root '/' portion.

4.1 GB for creating the backup file in /tmp directory (part of the backup job).

4.1 GB for copying the final backup file (from /tmp) to the specified location (/data/backup).

Please note that storage and /tmp folder is also used by the server (OS) for other jobs. So, 97.22 GB is too small for your setup.

Once you managed to run a latest backup file, it would be highly recommended to rebuild the server (using a larger storage capacity like 200+ GB) by a clean installation of ITA and then restoring a latest backup file.

Note: I would save the current backup files somewhere safe before doing any major changes.

Kind regards,

Jef

[Mr.T]

Hi @Jef @gsterling 

I wish to circle back to this issue.

I was away from this client for a while.

I managed to create a new VM with 200GB storage and restored an old backup however when I try to run a new Backup it still fails with the same error "No space left on device.'

I have implemented all the suggestions in all your replies and there is nothing else to delete.

Is there anything else I can look at?

Regards,

Tafadzwa.

[Jef]

Hi Tafadzwa,

It appears during the installation it was assigned about 77,4 GB to the root partition '/'  and currently you only have 7.35 GB free/available space there. This is not enough to run the ITA and/or restore a large backup file.

I guess during the installation the auto partitioning may have been selected (and possibly most of the disc space was assigned to /home)

You may need to rebuild the server by a clean installation of the ITA, but during the installation:

in INSTALLATION DESTINATION select 'I will configure partitioning' option.

and make sure that most of the storage capacity is assigned to the '/' partition.

installer may suggest a large /home partition, which can be deleted, and then include/add the freed space to the root '/' partition.
(https://community.se.com/t5/ITA-FAQ/ITA-on-premise-step-by-step-installation/ta-p/447248)

Kind regards,

Jef

[Mr.T]

Hi @Jef ,

Thank you for your feedback. It is much appreciated!

Regards,

Tafadzwa.