Piszki Lab | EN

My case study in the clouds…

Terrible upgrade vCenter 5.1U1 to version 5.1U2a.

| 0 comments

I recently appeared the service window on the production environment. Quite unexpectedly. Wide enough to use them to carry out long ago deposited vCenter 5.1 upgrade to the latest version. Move from 5.1U1 to  5.1U2a version seems a small step, eh? Nothing could be further from the truth! We had to deal with a two-day horror! In talks with representatives of VMware always praised sure that I do not know what they mean when they talk about the problems with the SSO in 5.1. Until now.

vm4

The first act, upgrade SSO. What could be simpler? Run installer, next, next, next, finish. The service did not get up. Symptoms are described in this KB (Server daemon died!). But reported walkthrough did not help. Next, Java copied from jre.zip file and SSO run without a problem, but does not work correctly. In the vCenter log (vpxd.log), appeared immediately post:

[01608 info ‘[SSO][SsoFactory_CreateFacade]’] Solution user set to: vCenterServer_2012.11.26_102929
[01608 info ‘[SSO][SsoFactory_CreateFacade]’] VC’s ServiceId in LookupService: {D3E3C386-3CCE-420E-AF3E-A72393D1A146}:9
[01608 info ‘[SSO][SsoFactory_CreateFacade]’] STS URI set to: https://PTSSO1.ptcloud.local:7444/ims/STSService
[01608 info ‘[SSO][SsoFactory_CreateFacade]’] Admin URI set to: https://PTSSO1.ptcloud.local:7444/sso-adminserver/sdk
[01608 info ‘[SSO][SsoFactory_CreateFacade]’] Groupcheck URI set to: https://PTSSO1.ptcloud.local:7444/sso-adminserver/sdk
[01608 info ‘[SSO][SsoFactory_CreateFacade]’] VC SSL certificate location: C:\ProgramData\VMware\VMware VirtualCenter\ssl\rui.crt
[01608 info ‘[SSO][CreateSsoFacade]’] [CreateUserDirectory] STS URI set to: https://PTSSO1.ptcloud.local:7444/ims/STSService
[01608 info ‘[SSO][CreateSsoFacade]’] [CreateUserDirectory] Admin URI set to: https://PTSSO1.ptcloud.local:7444/sso-adminserver/sdk
[01608 info ‘[SSO][CreateSsoFacade]’] [CreateUserDirectory] Groupcheck URI set to: https://PTSSO1.ptcloud.local:7444/sso-adminserver/sdk
[01608 error ‘Default’] Found dangling SSL error: [0] error:00000001:lib(0):func(0):reason(1)
[01608 error ‘Default’] Found dangling SSL error: [1] error:00000001:lib(0):func(0):reason(1)
[01608 error ‘[SSO][SsoFactory_CreateFacade]’] Unable to create SSO facade: vmodl.fault.SystemError.
[01608 error ‘vpxdvpxdMain’] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr)
–> Backtrace:
–> backtrace[00] rip 000000018018b86a
–> backtrace[01] rip 0000000180102ac8
–> backtrace[02] rip 0000000180103f9e
–> backtrace[03] rip 000000018008d22b
–> backtrace[04] rip 00000000003e5bdc
–> backtrace[05] rip 0000000000406652
–> backtrace[06] rip 00000001405cf001
–> backtrace[07] rip 00000001405c8e1c
–> backtrace[08] rip 00000001407ed8db
–> backtrace[09] rip 000007fefed7a82d
–> backtrace[10] rip 00000000776d59ed
–> backtrace[11] rip 000000007790c541
–>
[01608 warning ‘VpxProfiler’] ServerApp::Init [TotalTime] took 7784 ms
[01608 error ‘Default’] Failed to intialize VMware VirtualCenter. Shutting down…
[01608 info ‘vpxdvpxdSupportManager’] Wrote uptime information

Of course VMware has KB for everything, but this time described walkthrough did not work. Fortunately, I had the copy of the old jre directory from before the upgrade. We changed jre to old version and the SSO  surprisingly run properly. Also vCenter merged properly to the SSO.

Act Two, upgrade Inventory Service. The installer destroys jre directory, service does not get up. I used the procedure for substitution jre from an older (pre upgrade) version and everything is launched. Unfortunately, in the Inventory Service wrapper.log here is something like this:

INFO   | jvm 1    | 2014/10/29 08:09:35 | 2014-10-29 08:09:35 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor run
INFO   | jvm 1    | 2014/10/29 08:09:35 | SEVERE:
INFO   | jvm 1    | 2014/10/29 08:09:35 | java.lang.NoClassDefFoundError: sun/security/util/KeyUtil
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.crypto.provider.DHKeyAgreement.engineDoPhase(DashoA13*..)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at javax.crypto.KeyAgreement.doPhase(DashoA13*..)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.DHCrypt.getAgreedSecret(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.ServerHandshaker.clientKeyExchange(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.ServerHandshaker.processMessage(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.Handshaker$1.run(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at java.security.AccessController.doPrivileged(Native Method)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at com.sun.net.ssl.internal.ssl.Handshaker$DelegatedTask.run(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at org.apache.tomcat.util.net.SecureNioChannel.tasks(SecureNioChannel.java:285)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at org.apache.tomcat.util.net.SecureNioChannel.handshakeUnwrap(SecureNioChannel.java:343)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at org.apache.tomcat.util.net.SecureNioChannel.handshake(SecureNioChannel.java:193)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1642)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
INFO   | jvm 1    | 2014/10/29 08:09:35 |     at java.lang.Thread.run(Unknown Source)

Now We turn off the SSO, replaces the Java version from jre.zip file, run the Inventory Service. It works! So SSO works on the older versions of Java and the Inventory Service on the new (and the directory is shared). This problem could also be solved. JRE has been restored to old and SSO run. Alongside make new directory (jre_new) from file jre.zip and in file wrapper.conf from the “Program Files\VMware\Infrastructure\Inventory Service\conf” has been modified one line: “wrapper.java.command = D:/Program Files/VMware/Infrastructure/jre_new/bin/java”. After this modification, we have two services running on different versions of Java (that’s the type of installation where the SSO and the Inventory Service is installed on a separate host to vCenter). A very similar process took place in the vCenter, installer every time damaged the jre directory and had to be manually to swap its contents. In this case, all the Java services installed with vCenter work properly on the new and old versions of Java (We use new from jre.zip).

As for me, the strangest vCenter upgrade ever.

Rate this article:
[Total: 2    Average: 5/5]

Author: Piotr Pisz

Computer always, since I got a Commodore 64 at the end of primary school, through his beloved Amiga and Linux infinite number of consoles, until today, fully virtual day. Since 2001, Unix/Linux Systems Administrator, for seven years a faithful companion and protector of Solaris system, until his sad end. In the year 2011 came in the depths of virtualization, then smoothly ascended into the clouds and continues there today. Professionally working as Systems Architect in the Polish Security Printing Works.

Leave a Reply

Required fields are marked *.


.

Enjoyed the post? Support Piszki Lab | EN, click on the AD! :-)

.