Argo Workflow Controller CPU Usage Spikes

[Sushant20]

Team:
We are observing sudden spikes in CPU usage of workflow controller pods. Usually CPU usage is not more than 50 millicores however we now see max of 2 cores CPU usage suddenly with same no of workflow running. Would like to understand on what can lead to this behaviour and how to debug it?

[alexec]

Could you please add some information about this? E.g. what version are you running?

[Sushant20]

@alexec
Below are the details:
Argo workflow version : 2.11.4
Kubernetes version : 1.17
No customization to Argo workflow controller pod specs except enabling workflow archiving feature

I don't see errors and warnings in workflow controller logs

I did raise a similar observation sometime back with CPU spike: #4174 . Currently all our workflows are running fine but we have seen these CPU spikes as an early warning of issues in the past

Let me know, if you need more details

[alexec]

Do you know how often the spikes occur? Is there any output in the logs when it happens?

[Sushant20]

Hi @alexec
Once CPU spikes to 2 cores, they are pretty consistent at average of 1-1.5 cores. I don't see any strange output in the logs
Our normal average CPU usage of workflow controller pod is 10 millicores

[alexec]

So let me get this right. It's not really "spiking", instead it jumps up and remains up?

[Sushant20]

Yes you are correct @alexec , CPU jumps up and remains up and this seems very strange to me

[alexec]

There are known performance issues with v2.11.4. Can you please try v2.11.8?