Original Publishing Date: 2020-01-22 Symptoms Tasks related to VPS Hardware Node fails with authentication errors. Examples of tasks failures: Task ID 11957214 Task name Report raw traffic usage by dedicated VPSs on vzwin06 Task description Task gathers and reports raw traffic usage of dedicated VPSs hosted on Virtuozzo hardware node vzwin06(id = 682) Parent task ID 46 Queue status Failed Start not earlier than Jun-24-2014 10:21 Method name taskReportSingleHNTrafficUsage on SCREF:VPSTasks:0 Last execution output 404 : System errors : Authentication error: invalid credentials Task ID 1478501 Task name Update containers eid on HW node:vzlin001 Task description Task that updates all containers eid on hardware vzlin001 Queue status Failed Start not earlier than September 04, 2014 23:59 Method name taskUpdateEids on SCREF:VPSTasks:0 Last execution output 404 : System errors : Authentication error: invalid credentials Cause Administrator/root password for hardware node in OA is incorrect (or expired). Resolution There are two possible situations for this case: You have updated the root password on the VZ node and would like to update corresponding password on the OA side. In this case perform the following steps: Set the root/administrator password to the old one. Go to PCP > Services > Cloud Infrastructure > VPS Hardware Nodes Mark the problem node's checkbox Click the Change Administrator Password button Enter the new desired password and click Submit button Re-run the task You do not know the old password that was set on the server before and communication with this server from OA side with old password is not working. Contact Odin Technical Support to resolve the situation in this case. Internal content Link on internal Article

Original Publishing Date: 2020-01-22 Symptoms Provisioning "Panel_6_0" for APS application Parallels Plesk Panel task fails with the following error output: APS Application Error: 406 Not Acceptable [ApplicationUnknownError] Provisioning: resource 9f69d4dd-ae85-43f7-8723-ce8847545f94 of type 'panels' (http://www.parallels.com/aps/plesk/panel/1.2) for APS application 'Parallels Plesk Panel-1.2-0': P100001: Operation failed; details: ... [application/aps] Saving to: STDOUT 0K .......... .......... .......... .......... .......... 0% 41.7M 0s 50K .......... ... 0% 1.03M=0.01s Cannot write to - (Success).]. Cause Guest Tools are outdated inside the VM, where Plesk is getting provisioned. Recent versions of the tools have a new "prl_cat" binary included that is used during provisioning. Resolution Update Guest Tools inside the VM. In case VM templates have outdated tools version as well, it is advised to update them everywhere. Troubleshooting Check /var/log/PACI/paci.log on OA MN server and find the request that led to the error: 2016-02-17 04:47:12,280 -3d51- INFO LoggingFilter [apsc(3)] - 16448 * [PACI< 200 16448 < x-callback: true 16448 < Date: Wed, 17 Feb 2016 03:47:12 GMT 16448 < Content-Length: 27 16448 < x-paci-request-id: 6bd7f46d-d769-46d9-8564-f9c19ebfc94f 16448 < server: grizzly/2.2.21 16448 < Connection: close 16448 < Content-Type: text/plain 16448 < Put file has been initiated 2016-02-17 04:47:12,280 -3d51- DEBUG ConcurrentTimedMatcher [apsc(3)] - checkIn(/8303d18c-5f81-4590-8ed4-8c9a378fcdab, ...) 2016-02-17 04:47:22,432 -- DEBUG RESTCallbackListener [IM-Callback(2)] - Got IM callback, uri = [/8303d18c-5f81-4590-8ed4-8c9a378fcdab], rc = [-1] 2016-02-17 04:47:22,432 -- DEBUG ConcurrentTimedMatcher [IM-Callback(2)] - checkOut(/8303d18c-5f81-4590-8ed4-8c9a378fcdab, ...) 2016-02-17 04:47:22,433 -3d51- INFO LoggingFilter [apsc(3)] - 15697 * Server out-bound response 15697 < 406 15697 < Content-Type: text/plain 15697 < Cache-Control: max-age=0 15697 < P100001: Operation failed; details: ... HTTP request sent, awaiting response... 200 OK Length: 8471388 (8.1M) [application/aps] Saving to: STDOUT 0K .......... .......... .......... .......... .......... 0% 41.7M 0s 50K .......... ... 0% 1.03M=0.01s Cannot write to - (Success).] Copy the ID of the request from this line: 16448 < x-paci-request-id: 6bd7f46d-d769-46d9-8564-f9c19ebfc94f And search for this request ID in /var/log/IM/PACI-vm2vf.log on IM server to see what happens: 2016-02-17 04:47:12,299 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) INFO LoggingProxy [RequestProcessor-6276] - invoked: register_callback( 2016-02-17 04:47:12,300 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) INFO NativeVm2VfCode [RequestProcessor-6276] - [24541:3830814] INF ==== found mutex for 172.16.0.4 @[common/common_lib.c][575][establish_connection_ex][3880]) 2016-02-17 04:47:12,430 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) DEBUG NativeVm2VfCode [RequestProcessor-6276] - [24541:3830814] Callback invocation: set_handle(139812126297624) 2016-02-17 04:47:12,508 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) INFO NativeVm2VfCode [Thread-293158] - [24541:3830814] INF cmd: set -o pipefail ; wget --no-check-certificate --header 'Accept: application/aps' -O - 'https://172.16.0.3:6308//aps/2/packages/5089cc41-810f-433a-a958-bd75670f9aba' | prlctl exec {2d165850-f906-465b-8df3-487ef4abdbd7} prl_cat 'C:\Program Files\APS\9f69d4dd-ae85-43f7-8723-ce8847545f94\package.zip' @[exec.c][728][wget_thread][5130]) 2016-02-17 04:47:12,512 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) DEBUG NativeVm2VfCode [Thread-293159] - [24541:3830814] DBG fd=76 @[exec.c][258][read_string][5130]) 2016-02-17 04:47:22,413 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) DEBUG NativeVm2VfCode [Thread-293160] - [24541:3830814] DBG fd=76 closed len=2072 @[exec.c][281][read_string][5130]) 2016-02-17 04:47:22,413 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) DEBUG NativeVm2VfCode [Thread-293161] - [24541:3830814] DBG stderr = 2016-02-17 04:47:22,413 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) ERROR NativeVm2VfCode [Thread-293162] - [24541:3830814] ERR [{2d165850-f906-465b-8df3-487ef4abdbd7}] put_file failure: exit_code 2 2016-02-17 04:47:22,416 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) INFO NativeVm2VfCode [Thread-293163] - [24541:3830814] INF put cached connection: session uuid: "{70bbf576-a213-4414-b6eb-0f30029c09e9}", session handle 0xface, ref count 1 @[common/common_lib.c][729][drop_connection][5130]) 2016-02-17 04:47:22,416 (6bd7f46d-d769-46d9-8564-f9c19ebfc94f) DEBUG NativeVm2VfCode [Thread-293164] - [24541:3830814] Callback invocation: done_with_message(-2147483632, 5880159666763202560, Warning: Permanently added '172.16.0.3' (RSA) to the list of known hosts. Now the same operation can be tried on the target hardware node 172.16.0.4: [root@pcs01 ~]# wget --no-check-certificate --header 'Accept: application/aps' -O - 'https://172.16.0.3:6308//aps/2/packages/5089cc41-810f-433a-a958-bd75670f9aba' | prlctl exec {2d165850-f906-465b-8df3-487ef4abdbd7} prl_cat 'C:\Program Files\APS\9f69d4dd-ae85-43f7-8723-ce8847545f94\package.zip' --2016-02-17 04:22:13-- https://172.16.0.3:6308//aps/2/packages/5089cc41-810f-433a-a958-bd75670f9aba Connecting to 172.16.0.3:6308... connected. HTTP request sent, awaiting response... 200 OK Length: 8471388 (8.1M) [application/aps] Saving to: Б-°STDOUTБ-? 0% [> ] 65,303 --.-K/s in 0.001s Cannot write to - (Success). As a part of the command above, prlctl exec is launched with prl_cat command, trying to run it manually inside the VM shows that the command is not found: [root@pcs01 ~]# prlctl enter test_vm Microsoft Windows [Version 6.3.9600] (c) 2013 Microsoft Corporation. All rights reserved. C:\>prl_cat prl_cat C:\>'prl_cat' is not recognized as an internal or external command, operable program or batch file. Internal content

Original Publishing Date: 2020-01-22 Symptoms A domain in Odin Business Automation Standard (OBAS) suddenly stops resolving. How can I troubleshoot this issue? Resolution Verify the name servers used by the domain: ~# whois DOMAIN.TLD | grep NS Name Server: NS1.SERVER.TLD Name Server: NS2.SERVER.TLD ~# NS1.SERVER.TLD and NS2.SERVER.TLD are active name servers registered in OBAS: Top > Service Director > Domain Manager > Name Servers If the domain uses external name servers, reconfigure the domain's DNS zone at the authoritative name servers or change them to NS1.SERVER.TLD and NS2.SERVER.TLD in the registrar. If the NS records are missing, you should reconfigure the DNS zone. Add the OBAS managed name servers: DOMAIN.TLD. 600 NS NS1.SERVER.TLD. DOMAIN.TLD. 600 NS NS2.SERVER.TLD. A client can reconfigure a domain's DNS zone through the Control Panel. The records will appear in the DNS zone after synchronization. Verify synchronization status using DNS Records Suppliers: Top > Service Director > Domain Manager > Domains > DOMAIN.TLD > DNS Records Suppliers Verify the domain has DNS hosting. If the DNS zone is disabled, verify which orders are running. These orders are shown in the PBA-S Action Log (search by domain name) and in the domain's subscription: Top > Configuration Director > Logging and Errors > Action Log Top > Service Director > Domain Manager > Domains > DOMAIN.TLD > Subscription #NUMBER > Orders For Plesk Domain or Plesk Client subscriptions, verify the backup task is not running in Plesk. If the option Suspend domain until backup task is completed is enabled, the domain and its DNS zone are disabled during the backup process: Plesk > Subscriptions > DOMAIN.TLD > Open in Control Panel > Websites & Domains > Backup Manager > Scheduled Backup Settings If the DNS zone exists and it is active in OBAS, but the domain is still not resolved, verify that the domain is resolved at the OBAS-managed name servers: ~# dig DOMAIN.TLD @NS1.SERVER.TLD ~# dig DOMAIN.TLD @NS2.SERVER.TLD If an error appears when requesting the name servers (for example, a request timeout error), investigate the system log on the name servers and fix the problem. If you see that the domain's DNS zone is empty at the name servers, verify synchronization with OBAS-managed name servers according to the instructions from this KB: DNS zones are not synced to OBAS-managed nameservers Internal content

Original Publishing Date: 2020-01-22 Symptoms It is not possible to provision Cloud Infrastructure VE. Billing order fails with error: "The last error - "Service Creation Failed: Parallels Operations Automation error #error_code #0, extype_id #1, module_id #Common, Internal error: APSC: [IM] VE already exists.. Service Creation Failed: Parallels Operations Automation error #error_code #0, extype_id #1, module_id #Common, Internal error: APSC: [IM] VE already exists.."." In Cloud infrastructure Instance Manager in vps.log the following events could be found: 2018-03-06 08:52:01,880 (web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112) INFO CustomLoggingFilter [Grizzly-worker(2)] - 49428 * Server in-bound request 49428 > POST http://192.168.31.125:4465/paci/v1.0/of/1001214/ve 49428 > authorization: 49428 > x-paci-request-id: web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112 49428 > biz-desc: {"template":"Creating Server","attributes":{"name":"$VENAME"},"error":""} 49428 > initiator: 67493a43-275c-408b-848f-0b342135eda9 49428 > group-id: c70e5210-34b6-463c-be99-33cfe70610ad 49428 > x-callback-url: http://192.168.30.19:4477/5c475128-1c28-4da6-92a1-d39b39b3999c 49428 > accept: application/xml 49428 > content-type: application/xml 49428 > user-agent: Java/1.8.0_112 49428 > host: 192.168.31.125:4465 49428 > connection: keep-alive 49428 > content-length: 540 49428 > 0453143e-2fd4-4ec8-86c6-b3014836ddb71001214mailwaymailway1036216204810000011 2018-03-06 08:52:01,882 (web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112) INFO ClusterImpl [Grizzly-worker(2)] - CheckIn VeId[1001214.$VENAME] - com.parallels.c2u.im.resources.impl.VeResourceImpl.createVe_(VeResourceImpl.java:343) 2018-03-06 08:52:01,886 (web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112) INFO ClusterImpl [Grizzly-worker(2)] - CheckOut Ve VeId[1001214.$VENAME] 2018-03-06 08:52:01,886 (web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112) INFO CustomLoggingFilter [Grizzly-worker(2)] - 49428 * Server out-bound response 49428 < 409 49428 < Content-Type: text/plain 49428 < x-callback: false 49428 < x-paci-request-id: web_b87f774d-61e4-4b61-a698-e6c1c2d4990c_36112 49428 < VE already exists Cause Customer ordered second VE with the same hostname "$VENAME" while provisioning of the first subscription was not completed yet. Resolution Cancel failed order and create another with different hostname Internal content

Original Publishing Date: 2020-01-22 Ingram Micro Cloud is committed to providing high-quality and cost-effective solutions to our customers. As new technologies emerge, the demand for older product versions and components decreases. Accordingly, as we continue to address our customers' needs by introducing new products and services, we also periodically need to end maintenance and support for older software versions. End of Maintenance In keeping with our Product Lifecycle Policy, we will be ending maintenance for Hosted Lync modules (Hosted Lync 2010 APS 1.2, Hosted Lync 2013 APS 1.2, Hosted Lync 2013 APS 2.0) on December 31, 2018. After that date all Hosted Lync modules will discontinue any further development and/or maintenance. We suggest you to offer your existing Lync customers to switch to O365 Business services (bundled with Skype For Business), offered through direct partnership with Microsoft or resold through Ingram Micro (Referral program, Marketplace, OA Premium PaaS or dedicated option). Migration tools are not planned. Extended Support Period and End of Life Due to customer’s demand, an additional “Extended Support” period will be provided for Hosted Lync modules for Odin Automation 7.x branch which is the last one where these modules are supported. This period will last until June 30, 2019. During this time, support requests will be accepted, but will be limited to already known and available solutions/fixes. If you need any support during the Extended Support period, support incidents can be reported via the support portal at: https://www.cloudblue.com/support. Frequently Asked Questions Can we keep on using Lync service after the End of Life date? Yes, if you have a valid license. Can we upgrade our OA 7.x/8.0 installation to OA 8.2 without removing Hosted Lync module? Yes. Can we upgrade our OA 7.x/8.0/8.2 installation to Odin Automation 8.3 without removing Hosted Lync subscriptions and module? No. Upgrade to Odin Automation 8.3 is forbidden if there are existing Lync subscriptions. All Lync subscriptions must be destroyed before upgrading to Odin Automation 8.3. Please refer to Removing Hosted Lync Modules from Odin Automation. Internal content

Original Publishing Date: 2020-01-22 This hotfix is superseded by hotfix OA 7.3.0 HOTFIX 133169 PUI v10 Internal content

Original Publishing Date: 2020-01-22 Question How to setup OA in OBAS? Answer Please refer to the OBAS Provider's Guide Internal content

Original Publishing Date: 2020-01-22 Supported Odin Automation Versions The pre-sync mode is supported in the following Odin Automation versions: 6.0.9 HOTFIX 132337 Exchange WPE v5 7.4 and later New setting The new internal setting allows to suspend the migration process right before the completion and resume it any time later. Thus, you can schedule the mailboxes migration completion in a right time. (It does not affect moving between mailstores inside the same Exchange instance.) Activating the setting Contact your Odin support account manager to learn how to activate the setting. How migration behavior changed After the setting is activated, the following are the main steps to proceed with the migration: Schedule the migration (by any means: via switching service plan or changing service template, via OpenAPI, or using PCP). Mailboxes will be moved using cmdlet New-MoveRequest with the SuspendWhenReadyToComplete switch. After mailbox content is transferred to the target mailstore, the move request will be set to the AutoSuspended status. Wait till the move requests are in the AutoSuspended status. Get the list of the move requests in the AutoSuspended status using the Get-MoveRequest -MoveStatus AutoSuspended and resume the suspended move requests using cmdlet Resume-MoveRequest. Move requests will complete. Tasks in OA will track the move request completion and continue the work by finalizing the migration. Tracking migration status and troubleshooting Important Since the pre-sync feature suspends the mailbox move requests before its completion, it may cause performance issues due to many suspended move requests. OA tasks queue will grow. Tasks "Monitor move mailbox requests for service…" check statuses of all move requests and will process all suspended move requests. Pay attention and control the number of suspended move requests during the migration. Internal content Link on internal Article

Original Publishing Date: 2020-01-22 Symptoms An attempt to restart a stopped VE in CCP v2 results in the error message: FAILED TO REBOOT SERVER "SERVER-1000161-1" Screenshot: The same usecase does not lead to the error in CCP v1. Cause The behavior is planned to be improved in scope of CCU-15239. Resolution Presently there is no workaround to avoid the error message. Internal content

Original Publishing Date: 2020-01-22 This hotfix is superseded by hotfix BA 7.3.0 HOTFIX 132704 BM CORE v7 Internal content