Layout designer custom forms are unable to work with custom workflows. Only APEs can work with custom workflows. A tool named LayoutDesignerAPE.jar and its corresponding xml has been created to convert the custom form into a custom APE during run-time. However, installation of the tool is not known to the public.
Note : The rendering of the APE form after the conversion done by the LayoutDesignerAPE is different from the one seen in the layout designer itself. For instance, the controls in the APE will generate a label themselves. Therefore, the label controls in the layout designer form can be removed.
On top of that, one might notice that the dimension of the window looks larger in the layout designer compared to the APE form. The controls in the APE also looks clipped. This is because the layout designer shows everything in a smaller scale. This means that the controls usually need to be scale up before they appear on the window.
Other than dimension and label differences, controls like buttons and custom property lists do not render at all. The attachment control is also missing a Add Html.... button as well. This is because the HTML button is not supported in APEs and the rest of the missing controls are not implemented yet in the LayoutDesignerAPE.
Attached is a document to show how to install and use the tool.
To get the missing controls, one can update the source directly.