Did not quite get your question, but I don't think you can create a UFT API test outside of UFT.

My question is if we can create an API test using AOM. Also if we can invoke the API test execution via AoM or cmd line.

I'm afraid no, I don't think you can create an API test using AOM.

In regard to execution, officially we support running an API test in a docker container: Run UFT One tests in a Docker container (microfocus.com)

I have not tried it myself but by looking at the docker command, I think it is highly likely you can call HP.ST.Fwk.InternalExecuter.exe to run the API test directly without using docker.

Thank you for providing the reference on executing API tests in a docker. 

Will you be able to provide me more details on the file to be referred in an API test folder similar to what we have for GUI test - .mts file?

In an API test i see multiple files and subfolders. I see one main.cs file which contains information of API test to some extent.