Build/Deploy issues with GoCdAs you may have seen in a previous blog post we have a lot of pipelines, over 500 now and its set to grow by another 200 in the next month as we bring another 3 environments up. 700 pipelines is too much for a vanilla go instance to handle, the server can sometimes wait 3-4 minutes before detecting a check-in or to respond to a manual pipeline click.
Go server/agent setupWe have over 70 micro-services mainly written in .net but with a scattering of node and spa apps (detailed here), each is independently deployable and so we have decided to split them based on service boundary which we have roughly 6. So we have decided to create 6 go servers and split the workload across all six so that each server will only be handling 1-2 hundred pipelines each.
There are three consequences to splitting up the servers like this: One is that any templates are going to get duplicated, and you will have to keep track of failing builds on six servers instead of just one which means you will need to log into six servers and remember which services are on which go server instance. But by splitting the work by service boundary we will keep the value stream map intact where by we would loose it if we had split the servers by deployment region of which we now have three (each with a test, preprod and prod environment).
Installing multiple go agents on one VM, pointing at multiple different go serversSo we have six VMs each with a go server installed, and now by the use of powershell six agents on each VM. Each VM has an agent for each of the servers so that the workload can be spread evenly. The problem with this is that whilst a given VM can host several agents (read this) all the agents will point back to the same go server because the install uses an environment variable. The solution to this is to hack the config\wrapper-agent.conf file, change all the instances of '%GO_SERVER%' replacing it with 'go-serviceboundary1.mydomain.com' which is the DNS entry for the go server you are working for.
Given i was creating six agents on six machines i didn't want to do this by hand so i created a script to do it for me (found here) the more interesting bits are summarised below:
Download the latest go agent:
$goSetupExe = "go-agent-16.2.1-3027-setup.exe"
$client = new-object System.Net.WebClient
Command line install the first agent
.\go-agent-setup.exe /S /SERVERIP=go-serviceboundary1.mydomain.com /D=C:\go\agent1
Copy the install to a new instance
new-item "C:\go\agent$agentNumber" -ItemType Directory
Copy-Item "C:\go\agent1\*" -Destination "C:\go\agent$agentNumber" -Recurse
Rewrite the config
(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('go-serviceboundary1.mydomain.com', "go-$boundedContext.mydomain.com") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"
(Get-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf").replace('c:\go\agent1', "c:\go\agent$agentNumber") | Set-Content "C:\go\agent$agentNumber\config\wrapper-agent.conf"
Create and start the second service instance
New-Service -Name "Go Agent$agentNumber" -DisplayName "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -Description "Go Agent$agentNumber (go-$boundedContext.mydomain.com)" -StartupType Automatic -BinaryPathName "`"C:\go\agent$agentNumber\cruisewrapper.exe`" -s `"c:\go\agent$agentNumber\config\wrapper-agent.conf`""
Start-Service "Go Agent2"
It works really well for getting a new setup running quickly, and if i ever need to add a new go server and new agents this will be easy to extend to quickly get up and running.