VIM(Vcenter server for windows)的503问题一直深受诟病，包括我们也经常会因为主VCSA报错头疼一整天，这不，又一台VCSA出503这种错误了，一天的业务就此中断...

HeadPic: Maple

起因

今天下午收到了刘总的消息说vCenter访问报503了，如下所示：

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000fbb1883210] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

然后想起来之前自己重启VCSA的时候也遇到过好多次这样的问题，就回了一句等!,结果告诉我等和重启都试过了，遂叫他翻日志。

排查问题

上服务器翻日志的时候看到服务挂了一半：

PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --status
Running:
 EsxAgentManager VMWareAfdService VMWareCertificateService MWareDirectoryService VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vPostgres vmon vmonapi vmware-cis-config vmware-license vmware-psc-client vsphere-ui vspherewebclientsvc
Stopped:
 VMWareCAMService VMwareComponentManager VServiceManager content-library mbcs apiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health

不出所料，vpxd和vpxd-svcs都已经down掉了，手动敲命令开启失败。

PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --start vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2021-12-13T11:18:51.550Z   Service vpxd-svcs state STOPPED
Error executing start on service vpxd-svcs. Details {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd-svcs"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd-svcs'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Service-control failed. Error {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd-svcs"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd-svcs'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

解决方案

然后了解到服务器被上一位接手的老哥改过证书，然后想起一个在vmware社区看到的案例，更改证书的时候跑失败，然后vcenter回滚。
然后就让刘总看一下STS证书是不是有问题。

结果他翻遍了整个vim，根本找不着相关的日志

使用命令查看所有证书均在2021/12/12日过期：

PS C:\> $VCInstallHome = [System.Environment]::ExpandEnvironmentVariables("%VMWARE_CIS_HOME%");foreach ($STORE in & "$VCInstallHome\vmafdd\vecs-cli" store list){Write-host STORE: $STORE;& $VCInstallHome\vmafdd\vecs-cli" entry list --store $STORE --text | findstr /C:"Alias" /C:"Not After"}

STORE: MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Dec 12 05:36:33 2021 GMT
STORE: TRUSTED_ROOTS
Alias : d586b1b----------f72a3747----------603c2
            Not After : Dec  6 17:36:32 2029 GMT
STORE: TRUSTED_ROOT_CRLS
Alias : 03ad6cd----------f6e18e6f----------32f72
STORE: machine
Alias : machine
            Not After : Dec 11 17:27:58 2021 GMT
STORE: vsphere-webclient
Alias : vsphere-webclient
            Not After : Dec 11 17:28:00 2021 GMT
STORE: vpxd
Alias : vpxd
            Not After : Dec 11 17:28:01 2021 GMT
STORE: vpxd-extension
Alias : vpxd-extension
            Not After : Dec 11 17:28:03 2021 GMT
STORE: SMS
Alias : sms_self_signed
            Not After : Dec 12 17:43:12 2029 GMT
STORE: BACKUP_STORE
Alias : bkp___MACHINE_CERT
            Not After : Dec 12 05:36:33 2021 GMT
Alias : bkp_machine
            Not After : Dec 11 17:27:58 2021 GMT
Alias : bkp_vsphere-webclient
            Not After : Dec 11 17:28:00 2021 GMT
Alias : bkp_vpxd
            Not After : Dec 11 17:28:01 2021 GMT
Alias : bkp_vpxd-extension
            Not After : Dec 11 17:28:03 2021 GMT

然后想到了用证书管理器重置证书。（事后浏览了一下社区，很多人都是暴力敲8来重置证书过后就解决问题的）

原本想仅续签证书，但发现根ca也过期了

遂执行所有证书全部重新注册操作：

PS C:\Program Files\VMware\vCenter Server\vmcad> .\certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [[email protected]]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : y

Press Enter key to skip optional parameters or use Previous value.

Enter proper value for 'Country' [Previous value : US] :

Enter proper value for 'Name' [Previous value : CA] :

Enter proper value for 'Organization' [Previous value : VMware] :

Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] :

Enter proper value for 'State' [Previous value : California] :

Enter proper value for 'Locality' [Previous value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 10.-.-.---

Enter proper value for 'Email' [Previous value : [email protected]] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qual
ified Domain Name(FQDN), For Example : example.domain.com] : 10.-.-.---

Enter proper value for VMCA 'Name' :CA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Reset Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:9c6d---------------------------4cc76
Update service default-first-site:9c6d---------------------------4cc76; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ovwy7o
Get service default-first-site:5a5f---------------------------84beb
Update service default-first-site:5a5f---------------------------84beb; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ylk5yr
Get service default-first-site:083e---------------------------f3d01
Update service default-first-site:083e---------------------------f3d01; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_fsnv6_
Get service 76a5---------------------------c12b4
Update service 76a5---------------------------c12b4; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_yn5fml
Get service 68e8---------------------------83c0b
Update service 68e8---------------------------83c0b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_oysooz
Get service 6308---------------------------c6df1
Update service 6308---------------------------c6df1; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_quehgr
Get service d116---------------------------c5db8
Update service d116---------------------------c5db8; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vqxfvo
Get service 67f6---------------------------f72de
Update service 67f6---------------------------f72de; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_x0zbvu
Get service f648---------------------------3cec2
Update service f648---------------------------3cec2; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hspktf
Get service 8d3a---------------------------130e5
Update service 8d3a---------------------------130e5; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hzviao
Get service 5d75---------------------------5ec4b
Update service 5d75---------------------------5ec4b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_cqua3q
Get service 53ca---------------------------0dfb3
Update service 53ca---------------------------0dfb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9_c9pf
Get service 8891---------------------------c9003
Update service 8891---------------------------c9003; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_up4pcf
Get service c9ca---------------------------7d3dd
Update service c9ca---------------------------7d3dd; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9cyuzx
Get service d3ef---------------------------a1fdf
Update service d3ef---------------------------a1fdf; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hv6aa6
Get service 56e9---------------------------a9693
Update service 56e9---------------------------a9693; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ql6qpn
Get service a83b---------------------------09eea
Update service a83b---------------------------09eea; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_jjncqp
Get service 1975---------------------------65177
Update service 1975---------------------------65177; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ualmfy
Get service c835b---------------------------59bb3_kv
Update service c835b---------------------------59bb3_kv; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hwrqpu
Get service 12b7---------------------------68e3a
Update service 12b7---------------------------68e3a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ubyrj4
Get service 4ca6---------------------------fed53
Update service 4ca6---------------------------fed53; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vd5us4
Get service fba5---------------------------653e6
Update service fba5---------------------------653e6; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_m70tvt
Get service 8704---------------------------fa638
Update service 8704---------------------------fa638; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ygr5ee
Get service bd6f---------------------------abf9a
Update service bd6f---------------------------abf9a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec__6rsae
Get service c835b---------------------------9bb3_authz
Update service c835b---------------------------9bb3_authz; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_rw31ym
Get service c835b---------------------------9bb3
Update service c835b---------------------------9bb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_plshh5
Updated 26 service(s)
Status : 60% Completed [Reset vpxd-extension Cert...]
2021-12-13T11:46:20.423Z   Updating certificate for "com.vmware.vim.eam" extension


2021-12-13T11:46:23.173Z   Updating certificate for "com.vmware.rbd" extension

Reset status : 100% Completed [Reset completed successfully]

此时访问页面已经全部恢复正常。

问题解决

感情是这CA不会自动续签是吧。

马后炮

来自官方的重要信息：STS 证书过期时不会触发证书到期警报。本知识库文章中介绍了可以确定 STS 证书到期日期的唯一方法。VMware 建议您偶尔检查 STS 证书以确保其未过期。

参考文献

• 使用命令行验证和解决过期的 vCenter Server 证书 (82332)
• 将 vSphere 6.x/7.x 计算机 SSL 证书替换为证书颁发机构签名的自定义证书 (2112277)

文章版权

本文部分内容引用自刘明帅的博客的文章：记一次解vCenter 6.x WEB 503的故障
感谢他的付出

源文章链接：https://lmshuai.com/archives/499/

作者： 三月七

本文链接：https://blog.nanoka.moe/IT/vcenter-6-0-503.html

所有原创文章采用 知识共享署名-非商业性使用 4.0 国际许可协议 进行许可。
转载请注明出处，并附上本声明内容