One comment prior to replying to the specific questions:

Reindex in Elastic is a service that moves data from one index to another one. In Octane we have a process (post-upgrade job) in which, per each existing index: a new index is created, the data is reindexed from the existing index to the new one and the existing one is removed. This is required as a preparation stage before upgrading elastic to a new major version since elastic supports indexes created in the major version of the elastic server you are working with or one version back. Also please note, that Octane uses such reindex cycles to add index enhancements that require reindex.
Also note that reindex can be a long duration task depending on index sizes. For example, in the Octane instance used by the Octane dev group, we have an index that takes 30+ hours to reindex. The good news is that in Octane 16.1.100, reindex was modified to be a near-zero downtime process. I.e., there are small periods during a reindex in which the index’s features are in downtime. For example, in those 30+ hours of reindex mentioned above, the related features were in a <10-minute downtime.

• Does the re-indexing need to be run in Octane 16.1.100 or can I do it after upgrading to 16.2.100

It could be done in 16.2.100, but since 16.1.200 there is an enforcement check that reindex has already been done. So non-reindexed spaces will not be accessible by users until reindex is completed on them. This means that users will not benefit from the near-zero-downtime nature of the Octane reindex process, and they will be blocked out of the space for the full duration of reindex. (we have quite a few customers where the whole process for their large spaces is <2 hours, but, as mentioned above, it could take much more, depending on the index sizes)

• Does the re-indexing need to be run while still using Elasticsearch 7.16.2 or can I upgrade it first to 8.6.0

Yes, it needs to run while elastic is in version 7. Note that if you happen to have indexes from version 6, you will not be able to upgrade elastic to version 8.

• Also the upgrade guide for 16.1.100 already mentions the re-indexing although the official support matrix for Octane 16.1.100 only mentions Elasticsearch 7.9.2  and 7.16.2 as supported versions. Is this a gap in the documentation?

As mentioned above, reindex is a preparation stage before upgrading elastic. So, in preparation for upgrading to elastic 8 we execute reindex (while still using elastic 7). And not a documentation gap .

Thank you for the detailed response.