Optical transmission systems, high-order quadrature amplitude modulation, 5G+ networks,
coherent optical communication, high-capacity broadband, digital signal processing, forward error correction, spectral efficiency, fiber-optic networks, dual-polarization techniques, have the great potential to transform communication systems. The capacity to handle bandwidths efficiently with low energy consumption addresses requirements of future networks leading to stronger, more scalable and higher capacity optical infrastructure. Solutions with a high data rate are required to accommodate the expanding global mobile traffic. For networks with brief to medium range, such as automated industrial or residential systems, this will necessitate gigabit-capacity channels. In addition to supporting novel services such as Internet of Things (IoT), 5G technology satisfies this demand [1]. Gigabit service per user, minimal latency, and great spectral efficiency are some of the primary goals of the 5G network, which aims to handle traffic one hundred times faster than 4G networks [2-4]. The optical system must be adapted to handle high-speed and capacity data in order to meet the needs of fifth-generation optical fibers. All areas of digital communication now rely on complex higher-order modulation formats based on quadrature amplitude modulation (QAM) to increase the capacity and efficiency of the spectrum and, by extension, transmission bit rates across existing infrastructures [5]. In comparison to lower-order QAM, higher-order QAM is able to transmit more data [6]. Coherent systems employ spectrally efficient advanced modulation forms to maximize the channel bandwidth. Formats such as 16-32-64-128-QAM and quadrature phase-shift keying are a few examples. Smaller constellations up to 64-QAM may soon be used in practical systems, ones up to 256-QAM becoming feasible in five to ten years, as a result of laser phase noise and OSNR limitations [7]. Improving the capacity, cost-per-bit, and bit rate of coherent optical transmission systems COTS is a top research priority for long-distance communications. The effective technique known as wavelength division multiplexing WDM can solve a broad range of problems. In particular, high-capacity data transmission guarantees coverage across large areas, improved flexibility, and long-distance communication. When building transport domains between the baseband unit BBU and the remote radio heads RRH, WDM-PON is usually the major framework used by current 5G mobile access networks [8]. Figure 1 shows 5G bandwidth and latency needs [9].