I'm not sure about ESTER but for SOFR, the FED has stated that the way to get forwards is derive them from SOFR futures.

https://www.federalreserve.gov/econres/notes/feds-notes/indicative-forward-looking-sofr-term-rates-20190419.htm

See section A.2 in this paper:

https://www.federalreserve.gov/econres/feds/files/2019014pap.pdf

The answer looks to be similar for ESTER as well, the hope is by the decommissioning of LIBOR, swap and or futures markets will be liquid enough.