dco after upgrade password_change_required column missing in db

DCIM_Support · ‎2020-07-05

This question was originally posted on DCIM Support by Lukas Jegerlehner on 2019-07-30

Dear Team,

the upgrade from a DCO 8.2.12 --> to 8.3.2 did run through perfectly fine. All services are running fine. After the reboot I waited the required time and did not disrupt the process.

Problem:

on the web no user can login due to the missing column "password_change_required" in the public.users table.

I am suspecting that all the java which should be running after the reboot of the system, did not get startet and therefore the databases did not get updated.

this printscreen is from a server, which is running perfectly on 8.3.2

How can this issue be fixed?

The server has a DR node running. Before the upgrade i disconnected the DRs, by promoting the server.

A clean dco installation is not the solution for this customer.

Best Regards

Lukas

DCIM_Support · ‎2020-07-05

This comment was originally posted on DCIM Support by Paweł Kozioł on 2019-07-30

Hello Lukas,

Do you have specific error message or log regarding logging in issue? password_change_required column was added in version 8.0.0, so you should already have it for a long time.

I see on you screenshot that you have SchemaUpgradePowerMeasurementsData in your sma_database_schema table. Is it ture? If so, then you are right - database was not upgraded.

Would you be able to share log files from the DCO server? Specifically, from the /opt/jboss-as/standalone/log directory you should have a server.log, operations.log files, and you may have a dco-installer.log file in the same folder or in /var/log

Did the DCO server have access to the internet when the update was run?

Best Regards
Paweł

DCIM_Support · ‎2020-07-05

This comment was originally posted on DCIM Support by Lukas Jegerlehner on 2019-07-30

Hello Pawel,

thank you for your quick answer.

Yes it is true. But the print screen is from a successfully working 8.3.2 DCO. Not from the "problem" machine.

I will email you a link with the log files.

Yes the internet was accessible through the upgrade.

The upgrade went through well.

On the DCO web there is an error when trying to login. "password change required" not available.

DCIM_Support · ‎2020-07-05

This answer was originally posted on DCIM Support by Paweł Kozioł on 2019-07-31

Thanks for the logs. I looked into them and I can see in errors in server.log that saying that column force_single_sign_on does not exist in users table. We added it in 8.2.13, so it means that database wasn't upgraded.

In operations.log you can see that upgrade process didn't run them, because it thought that was running on DR node.

Jul 9 11:58:37 dcim-dcoperation01 operations: Not upgrading schema on a DR node.

To me it looks that server wasn't even tried to be promoted. There should be "Promoting this DR node to master" log, but I don't see it anywhere.

Could you check what is the value for ISXO_BACKUP_SITE in /etc/isx-operations/config?

Could you try restarting Operations service? It should run database schema upgrade if it's really master node.

DCIM_Support · ‎2020-07-05

This comment was originally posted on DCIM Support by Lukas Jegerlehner on 2019-08-05

please see my findings below:

here is a quick description how to provoke the case. Can you and your development team please look into fixing the issue for the next release. Thank you in advance.

Two clean dco installations (version 8.2.12)
Connect DR node.
For dco2 the ISXO_BACKUP_SITE field, within the /etc/isx-operations/config file, will become 1 (slave).
promote dco2 —> ISXO_BACKUP_SITE for dco2 will become 2 (master)
upgrade dco2 to the latest version 8.3.2. Schema will not update due to ISXO_BACKUP_SITE = 2 (wasn’t cleared).
ExecStartPre=/usr/libexec/dco-platform/schemaUpgrade.sh —> The IF within the schemaUpgrade Shell checks if the field ISXO_BACKUP_SITE is empty.
In the case of switching the DR back and forth between two servers (this could happen over time), both servers will end up with a field entry ISXO_BACKUP_SITE = 2, once disconnected again.
—> solution: check the IF statement within schemaUpgrade.sh, which was added after version 8.2.12. For the slave ISXO_BACKUP_SITE = 1 it makes sense to not run a schema update. For 2 and ‘ ‘ schema upgrades may should be performed ?

DCIM_Support · ‎2020-07-05

This answer was originally posted on DCIM Support by Lukas Jegerlehner on 2019-08-05

Hi Pawel,

thanks a lot for your feedback.

before the upgrade and in DR mode:

DCO1: ISXO_BACKUP_SITE = 2

DCO2: ISXO_BACKUP_SITE = 1

promoting the DCO2 results in: (the operations.log's shows that both DCOs are no longer in DR mode)

DCO1: ISXO_BACKUP_SITE = 2

DCO2: ISXO_BACKUP_SITE = 2

restarting the operations service . --> schema upgrade possible (operations.log)

upgrading from 8.2.12 --> 8.3.2

-upgrade runs through fine.

-restart of the node.

-now suddenly the operations.log shows again "Not upgrading schema on a DR node". This is very strange since 5min ago the schema upgrade was possible.

DCIM_Support · ‎2020-07-05

This question is closed for comments. You're welcome to start a new topic if you have further comments on this issue.