Splunk: 'net stop splunk' only works *sometimes*

I'm trying to manage Splunk with Chef and ran across a problem when using Chef to programmatically start/stop/restart the Splunkforwarder service:

The request did not respond to the start or control request in a timely fashion. - ControlService: The service did not respond to the start or control request in a timely fashion.

Once the Chef run fails, I'm able to start the service with net start splunkforwarder.

I decided to try going through the restart (stop/start) process manually in Powershell, and now when I try to stop the service with net stop splunkfowarder I get an error:

The service is not responding to the control function.

Somtimes stopping the service works, but rarely. At this point I have no idea what's going on because I'm not used to working on Windows (or with Splunk), and am not sure if the error from Chef and the net stop splunkforwarder errors are related.

I've also found if I interact with splunk directly through the executable file in C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe that everything works fine. I can cd to that directory and run ./splunk restart without problems.

Anyone know what's going on or have advice on next-steps for troubleshooting?

Tags: