Original Publishing Date: 2020-01-22 Symptoms One or several VE backups fail for a particular hardware node, the following pattern appears in the /var/log/pa/vps.log file on OACI IM node: ERROR ScheduledBackupTask [Shared executor thread #9 @1 @BACKGROUND] - Backup operation failed java.lang.reflect.UndeclaredThrowableException: null ... Caused by: com.parallels.c2u.jms.rpc.footprint.TimeoutException: timed backup_ve_cb out after 60000 MILLISECONDS at com.parallels.c2u.vm2vf.corba.Vm2VfApiOperationsClient.backup_ve_cb(Vm2VfApiOperationsClient.java:1418) ~[im.jar:na] ... 35 common frames omitted Caused by: java.util.concurrent.TimeoutException: null at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771) ~[na:1.8.0_112] at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) ~[na:1.8.0_112] at com.parallels.c2u.vm2vf.corba.Vm2VfApiOperationsClient.backup_ve_cb(Vm2VfApiOperationsClient.java:1410) ~[im.jar:na] ... 35 common frames omitted ERROR GenericBackupTask [Shared executor thread #9 @1 @BACKGROUND] - Backup of the VE [VeId[1012069.server-1031410-1]] aborted /var/log/pa/vps.vm2vf.log on the corresponding hardware node contains this pattern for the backup call: INFO BackupCallbackProxy [Thread-1129375] - INF calling: cbp 0x7f38e8002f68 job 0xcb416 [collect_markers_pp] index [3951](backup_action, backup.c, 492) @[common/generic_sdk_cb.c][706][reg_next_generic_sdk_cb][137750]) INFO BackupCallbackProxy [Thread-1129376] - INF creating: cbp 0x7f38e8002f68 job (nil) [collect_markers_pp] index [3951] (backup_action, backup.c, 492) @[common/generic_sdk_cb.c][714][reg_next_generic_sdk_cb][137750]) INFO BackupCallbackProxy [Thread-1129377] - INF pthread_cond_timedwait [110]: cbp 0x7f38e8002f68 job (nil) [__PrlSrv_GetCommonPrefs_cbp] index [3951] (backup_action, backup.c, 492) @[common/generic_sdk_cb_tools.c][151][cbps_timeout_sleep][755003]) ERROR BackupCallbackProxy [Thread-1129378] - ERR PrlJob_GetStatus: job (nil) failure -2147483645 @[common/generic_sdk_cb_tools.c][105][cbps_job_check_status_ex][755003]) This part is important: ERR PrlJob_GetStatus: job (nil) failure -2147483645 It is possible that several consecutive backups fail the same way and vm2vf service stops responding afterwards. All further calls to the node end up with a fast failure error message: fast failure for backup_ve_cb after 240000 MILLISECONDS Studying vps.vm2vf.log carefully for the actions preceding the error, one may find activity messages from 2 different backup calls running under the same thread uuid, for example: (f33d81ee-4f06-4ae9-b5ca-639f6b9a650e backup_ve_cb callback) INFO BackupCallbackMediator637 [Thread-59526] - XML: CALLBACK_3834f137-8299-4307-a2fa-af15aafc9a6b659// callback backup_ve_cb in_progress { "msg" : "PET_DSP_EVT_BACKUP_PROGRESS_CHANGED", "percent" : 0 } to destination queue://client (f33d81ee-4f06-4ae9-b5ca-639f6b9a650e backup_ve_cb callback) INFO BackupCallbackMediator637 [Thread-59520] - XML: CALLBACK_3834f137-8299-4307-a2fa-af15aafc9a6b659// callback backup_ve_cb in_progress { "msg" : "PET_DSP_EVT_BACKUP_PROGRESS_CHANGED", "percent" : 42 } to destination queue://client (c0f169ae-cc55-459b-992a-0fd9828dc58d backup_ve_cb callback) INFO BackupCallbackMediator637 [Thread-59523] - XML: CALLBACK_e831f0cc-d5a4-4cf5-9daa-14a8fd87a3b9659// callback backup_ve_cb in_progress { "msg" : "PET_DSP_EVT_BACKUP_PROGRESS_CHANGED", "percent" : 42 } to destination queue://client (c0f169ae-cc55-459b-992a-0fd9828dc58d backup_ve_cb callback) INFO BackupCallbackMediator637 [Thread-59529] - XML: CALLBACK_e831f0cc-d5a4-4cf5-9daa-14a8fd87a3b9659// callback backup_ve_cb in_progress { "msg" : "PET_DSP_EVT_BACKUP_PROGRESS_CHANGED", "percent" : 0 } to destination queue://client Under large pressure, vm2vf service may become completely unresponsive, not accepting any new requests. Cause The issue is treated in scope of CCU-18107: Deadlock on vm2vf under memory intensive operation (PRL_ERR_INVALID_ARG) Resolution To release vm2vf service from a hung state, restart it: # service PACI-vm2vf restart It is highly advised to restart Virtuozzo services as well, since the issue originates from Virtuozzo backend: For VZ 6.0: # service parallels-server restart For VZ 7.0: # systemctl restart prl-disp Internal content Link on internal Article

Original Publishing Date: 2020-01-22 Symptoms shaman stat command prints a warning: shaman: Failed to read XML model: syntax error What does it mean and how to fix this kind of error? Cause The error is not harmful and appears when there are "Unknown" resources: ~# shaman -c pcs stat Failed to read XML model: syntax error Cluster 'pcs' Nodes: 3 Resources: 5 NODE_IP STATUS RESOURCES * 10.1.1.10 Active 2 CT, 0 Unknown M 10.1.1.11 Active 1 CT, 0 Unknown 10.1.1.12 Inactive 1 CT, 0 Unknown CT ID PWRR STATUS OWNER_IP PRIORITY 101 on Active 10.1.1.11 0 102 on Active 10.1.1.13 0 103 on Active 10.1.1.12 0 104 on Active 10.1.1.11 0 Unknown STATUS OWNER_IP PRIORITY Broken Unknown 0 Resolution Clean up "Unknown" and "Broken" resources: # shaman cleanup-broken This command attempts to relocate all broken resources to a local node without starting them. After relocation these resources can be manually migrated to desired nodes. Files of broken resources, which cannot be relocated, are deleted. For other cases, to get rid of the error in shaman stat, the resources should be placed under appropriate node in shaman records. Get "Node ID” (or host_id) value for each PCS host in cluster by running "shaman stat" with verbose output ("-v" option"): # shaman -v -c pcs stat Failed to read XML model: syntax error Cluster 'pcs' Nodes: 3 Resources: 4 NODE_IP STATUS NODE_ID RESOURCES *M 10.1.1.37 Active 34705c9745f742a1 1 CT, 1 VM, 0 Unknown 10.1.1.38 Active da6fd9b1b4c34fb6 0 CT, 0 VM, 0 Unknown 10.1.1.39 Inactive 42064ead9c244d74 1 CT, 0 VM, 1 Unknown CT ID PWRR STATUS OWNER_IP OWNER_ID PRIORITY 101 on Active 10.1.1.39 42064ead9c244d74 0 999 on Active 10.1.1.37 34705c9745f742a1 0 VM NAME PWRR STATUS OWNER_IP OWNER_ID PRIORITY vm-test off Active 10.1.1.37 34705c9745f742a1 0 Unknown STATUS OWNER_IP OWNER_ID PRIORITY Active 10.1.1.39 42064ead9c244d74 0 The list of shaman resources is located under the /pstorage/$CLUSTER_NAME/.shaman/md.$NODE_ID/resources directory. Since the directory is under Pstorage mount, it can be accessed from any node of the cluster. Replace $CLUSTER_NAME and $NODE_ID in the example above with appropriate values. For existing virtual machines, their resource files should be moved under appropriate PCS node: ~# mv /pstorage/$CLUSTER_NAME/.shaman/md.$HOST_ID_1/resources/vm-NAME /pstorage/$CLUSTER_NAME/.shaman/md.$HOST_ID_2/resources/vm-NAME Where md.$HOST_ID_1 is the host improperly owning this VM and md.$HOST_ID_2 is the proper host running this particular VM. Note: Do not change the name for the resource file, it is always written in vm-NAME or ct-CTID format. Internal content Link on internal Article

Original Publishing Date: 2020-01-22 Symptoms A container state has been changed from RUNNING to MOUNTED as a result of either a direct shutdown from inside of the container or CT init process killed by OOM killer. However, the state is not reflected in CCP and subsequent VE stop/restart operations fail. In /var/log/IM/paci-snmp.log the following message can be obtained: 2016-02-20 15:30:11,382 (23f434f4-88f2-485f-bc62-55b2c031418c) TRACE SnmpManager [Shared executor thread #1 @1] - State discrepancy STARTED != STOPPED for VE 1000001.test has been ignored as 6653556764220510731 <= 6653556764221931990 Cause SNMP agent is down on the target Virtuozzo hardware node: [root@vz ~]# service snmpd status snmpd is stopped Backwards VE state monitoring is handled by paciagent SNMP module on the hardware node, installed with PACI-agent RPM. Resolution Make sure SNMP agent is up and running and configured for automatic start-up: # service snmpd start # chkconfig snmpd on Internal content

Original Publishing Date: 2020-01-22 Symptoms Task "Execute operation 'provisionWebsite' for Provisioning WebHosingPlesk subscription failed with error: There are no compatible Plesk nodes for provisioning: Nodes that do not meet the requirements In log /var/log/shm-dispatcher/dispatcher.log on Plesk Management Node can be found the corresponding message, for example: The node 192.0.2.20 does not meet the following hosting service requirements: Array [0] => Odin\ShmWebDispatcherBundle\Aps\Resource\StructureProperty Object ( [name] => panel [class] => setting [type] => enum [value] => 17.0 ) WebHosting Plesk Linux Service Template is configured to use particular version of Plesk panel, e.g. 17.0. Plesk nodes were updated recently and have Plesk panel version 17.5. Cause By design, system automatically picks Plesk version from node only in case the parameter Any Plesk version is set in PCP > Products > Service Templates > WebHosting Plesk Linux Template > Hosting Parameters > Web hosting panel version. In case if the particular version of Plesk panel is specified, WebHosting Plesk module relies on the value, specified in the Template, leading to error, since such version is not available on the node anymore. Feature request PFR-1535 for improving error reporting - such as adding subscription id to the task - has been submitted. Please contact your Technical Account Manager to check the status of the request. Resolution In PCP > Products > Service Templates > WebHosting Plesk Linux Template > Hosting Parameters change Web hosting panel version parameter to Any Plesk version or to new updated version; In PCP > Products > Service Templates > WebHosting Plesk Linux Template > Synchronize check and synchronize the required subscriptions; Restart failed task. Internal content

Original Publishing Date: 2020-01-22 Symptoms Customer owns a site located on NG Web-Cluster or standalone NG Web-server He uploaded file with size more than 2 Gb, i.e. 50 Gb, to the site via FTP But usage of resource Traffic increased only by ~2 Gb Cause This is a result of bug POA-114943. In result of the bug traffic log entries with size more than 2Gb are parsed incorrectly. Resolution Please contact your Technical Account Manager or pta@odin.com to clarify status of bug POA-114943 Internal content

Original Publishing Date: 2020-01-22 Symptoms Hotfix KB132334 was recently installed. Attempt to install SSL certificate to the webspace from the clipboard fails with: Error "Cannot install host certificate without private key" Cause The issue is acknowledged as POA-114957. Resolution A certificate and a key still can be uploaded via the file. https://download.automation.odin.com/oa/7.3/oapremium/portal/en/customer_ccpv1_guide/index.htm?fileName=62933.htm Contact Technical Account Manager to trace the status of the issue. Internal content

Original Publishing Date: 2020-01-22 Symptoms If Billing menu items are disabled according to the Operations Provider's Guide , the button Reassign User Licenses at CCPv2 > Office 365 still works, but the below error message is displayed: Resource limit is exceeded. Please contact provider. although there are enough free licenses. Cause This behavior was aknowleged as a software isse APSA-19627 and will be fixed in future relases of the Office 365 APS application. Resolution Please contact your Technical Accoutn Manage in order to clarify the current status of the issue. In order to work around the issue and hide the error message, please apply the below solution: Go to OA > System > Settings > CCP v2 Navigation Search for the view view-plugin@http://www.parallels.com/ccp-billing#bizApiPlugin Tick the checkbox next to the view and click Show Internal content

Original Publishing Date: 2020-01-22 Question: It is possible to use the Web-analytics Integration feature to build-in a customer JavaScript code into the Customer Control Panel. But how to run different code for CCP v1 and CCP v2? Answer: This can be done by analysing the CCP URL. For example, the following approach can be used: var getLocation = function(href) {var l = document.createElement("a"); l.href = href; return l;}; var u = getLocation(window.location.href;); i = u.pathname.split('/'); if (i[1] == 'ccp'){ } else { } The customer JS code is placed to OA > System > Settings > Brands > YourBrand > Web-Analytics Integration. Internal content

Original Publishing Date: 2020-01-22 Symptoms An Open-Xchange task fails: Task name: APS application 'Open-Xchange', id 109, instance 3525 -> service 'context', instance 18705: executing configuration script Output: Execution of configuration script for instance with id 18705 of service with id context of instance with id 3525 of application with id 109 failed - returned value: -1 output: '' errors: 'Creating OX context 5360 Cannot map '||' to the newly created context. This mapping is already in use. Failed to create OX context 5360: Cannot map '||' to the newly created context. This mapping is already in use. Cause This issue is caused by an issue with the ID APS-12159. Resolution Please contact your Account Manager to track the status of request APS-12159. As a workaround, you can: Change context login '' to '1' in the Parallels Operations Automation (POA) database. Please note that it is recommended you make backups before manual database modification: plesk=> begin; update aps_registry_object_setting set value= '`' where value= '' and name='admin_login' and registry_object_id in (select registry_object_id from aps_registry_object_setting where value=''); Use the pem.rescheduleTask.xml API method to reschedule pending tasks with the new parameter env_var_SETTINGS_admin_login = '1' (instead of ''): pem.tasks.rescheduleTask task_id params name env_var_SETTINGS_admin_login value 1 Process pending tasks by rescheduling with the new parameter env_var_SETTINGS_admin_login = '1' After the instance will be configured with the new context ID, revert your changes to the POA database back: plesk=> begin; update aps_registry_object_setting set value= '' where value= '1' and name='admin_login' and registry_object_id in (select registry_object_id from aps_registry_object_setting where value='1'); Check that it is now possible to create a Service User with Open-Xchange Webmail. Internal content

Original Publishing Date: 2020-01-22 Symptoms Backup tasks are stuck in task manager for 24 hours, not showing any progress, and fail with timeout as a result: 36596021 Backup 'MySQL Database "db1000010_content_hub_he"' into group 'more updates' Mar-23-2015 13:27:07 sc02databases.db010 1075196 Enabled Running 36596020 Backup 'Webspace "www.NG1075196-2445.ccccloud.com"' into group 'more updates' Mar-23-2015 13:27:07 sc03apache.ds21 1075196 Enabled Running Tasks get rescheduled with the output: Request has been timed out, details: system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0' TAO exception, minor code = 3e (timeout during recv; low 7 bits of errno: 62 Timer expired), completed = MAYBE Task error upon failure: Message:'Destination host 'winbackup.hosting.local' (#12), IP '10.10.10.14' : Operation execution is aborted by timeout. Please increase timeout for this operation and try again. Backups are done from Linux hosting nodes to a Windows backup node. The following command, run from the Management Node for the Windows backup node host_id, hangs indefinitely: # /usr/local/pem/bin/pleskd_ctl ping 12 Cause HCL operations get stuck when performing several simultaneous backup operations from Linux node(s) to a single Windows node. The behavior is recognized as a product issue with internal id POA-92544. Resolution There are 2 possible workarounds: regularly (once a day) restart "pem" service on the slave backup node: net stop pem && net start pem configure a Linux backup node for backing up Linux webspaces and DBs Note: the trigger for this issue is a high number of simultaneous backup tasks. So when they get failed in large amount, after pem service restart on the backup node, it is recommended to run the failed tasks one by one in order to avoid the same hang-up again. Internal content