Robert Bogart rhb2

smaug-ansible with vmware

Testing smaug-ansible changes can be done on a laptop. This is a guide to do so using vmware.

Prerequisites

The following must be installed on the host machine running the VMs:

Python 3.x
pip (Python package intaller)

From the headnode, login to the rebalancer zone

[rbogart@headnode (ap-northeast-1a) ~]$ manta-login rebalancer
0:   rebalancer        1 90ed0873-1c3a-4a25-84fe-53cb0a5e5443 10.92.72.42     
1:   rebalancer        1 d76fdeca-75ed-4451-8046-eda32382789c 10.92.64.42     
Choose a number: 0
[Connected to zone '90ed0873-1c3a-4a25-84fe-53cb0a5e5443' pts/8]
Last login: Tue Jan  5 19:52:23 on pts/6
 =  J O Y E N T  =

Component	File	Reviewer	Comments
admin	admin.go
admin	operation-mode.go
admin	operation-recovery.go
admin	operation-status.go
chunk	chunk.go
chunk	operation-delete.go
chunk	operation-delete_test.go
chunk	operation-get.go

sequenceDiagram
    participant User
    participant Gage
    participant Object API
    participant Chunk API
    participant FoundationDB
    User-->>Gage: S3 GET
    Gage->>Object API: SWebDAV GET
 Object API-&gt;&gt;FoundationDB: Get "key" = "value"

	32edcc12-778e-46e9-a864-e570f9d723f0=# select count(*) from evacuateobjects where skipped_reason is not null;
	count
	-------
	3855
	(1 row)

	3855 total failures


	Of those failures, there were sustained on the following agents:

	32edcc12-778e-46e9-a864-e570f9d723f0=# select distinct assignment_id,dest_shark,skipped_reason from (select assignment_id,dest_shark,skipped_reason from evacuateobjects where skipped_reason is NOT NULL) as foo order by dest_shark;
	assignment_id \| dest_shark \| skipped_reason
	--------------------------------------+----------------------------------+-------------------------
	d2c91df2-f6b1-4fe2-a007-5dc34cc9245d \| 1023.stor.eu-central.scloud.host \| destination_unreachable
	c5291fbf-5218-41e5-b8bf-991e538a52a6 \| 1032.stor.eu-central.scloud.host \| source_other_error
	db6e4ab4-9892-4823-8b48-085544083e9f \| 1118.stor.eu-central.scloud.host \| source_other_error
	eef43f7c-86d8-44db-8ac1-334948d37c99 \| 1136.stor.eu-central.scloud.host \| agent_fs_error
	20d15f1a-7b55-466a-845d-c62f7ade0537 \| 1138.stor.eu-central.scloud.host \| {http_status_code:404}
	82ae90d7-06cd-4890-874c-93bb324621a4 \| 1231.stor.eu-central.scloud.host \| agent_fs_error
	8de5f6cf-339e-482a-8c16-ea3

	dest_shark \| assignment_id \| source
	-------------------------------+--------------------------------------+------------------------------
	1364.stor.us-east.scloud.host \| efe01c99-7076-4ab1-b3b0-37677e5785c9 \| 55.stor.us-east.scloud.host
	1405.stor.us-east.scloud.host \| 8dd433bd-2d1a-4694-b00d-c241667e58dc \| 205.stor.us-east.scloud.host
	1580.stor.us-east.scloud.host \| 659a2036-9891-496a-b9f0-deafbfbb193e \| 145.stor.us-east.scloud.host
	1206.stor.us-east.scloud.host \| 03a33f61-afb7-4bfa-b78e-d76e7a0907e2 \| 91.stor.us-east.scloud.host
	1409.stor.us-east.scloud.host \| 340ac2d5-e297-4c24-9ded-901f90079bfd \| 74.stor.us-east.scloud.host
	1465.stor.us-east.scloud.host \| 97c6c792-a17d-4015-99fb-c38852149b8f \| 205.stor.us-east.scloud.host
	1440.stor.us-east.scloud.host \| 7f304a74-4fc3-44bd-8ee3-61dfecf2e286 \| 205.stor.us-east.scloud.host
	1002.stor.us-east.scloud.host \| 0f572419-26c3-48df-a138-f353d2153086 \| 205.stor.us-east.scloud.host

	[root@headnode (rbogart-dc) ~]# manta-hotpatch-rebalancer-agent deploy -a 4518269c-070a-454b-b597-8033c6798471
	This will do the following:
	- Import rebalancer-agent image 4518269c-070a-454b-b597-8033c6798471
	(master-20200616T183110Z-g82b8008) from updates.joyent.com.
	- Hotpatch rebalancer-agent image 4518269c-070a-454b-b597-8033c6798471
	(master-20200616T183110Z-g82b8008) on all 4 storage instances in this DC

	Would you like to hotpatch? [y/N] y
	Trace logging to "/var/tmp/manta-hotpatch-rebalancer-agent.20200617T223409Z.deploy.log"
	Importing image 4518269c-070a-454b-b597-8033c6798471 from updates.joyent.com

	#!/usr/sbin/dtrace -s
	#pragma D option quiet

	pid$target::malloc:entry
	{
	self->size = arg0;
	}

	pid$target::malloc:return
	/self->size/

	TASK [common_linux_packages : install yum-plugin-priorities] *****************************************************************

	fatal: [10.88.88.133]: FAILED! => {"ansible_facts": {"pkg_mgr": "yum"}, "changed": false, "msg": "https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: [Errno 12] Timeout on https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: (28, 'Connection timed out after 30001 milliseconds')\nTrying other mirror.\nhttps://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: [Errno 12] Timeout on https://joyent-gage:AKCp5fTjoH7g4jBZiJR5TwHGVS5XZVNDAYkVcFBn8fK1kpLxPWMtsnFieBNfGAfNe4Xvhyehu@repository-us.cerescloud.io/artifactory/gage-rpm/repodata/repomd.xml: (28, 'Connection time