The rapid surge in internet-driven smart devices and bandwidth-hungry multimedia applications demands high-capacity internet services and low latencies during connectivity. Cloud radio access networks (CRANs) are considered the prominent solution to meet the stringent requirements of fifth generation (5G) and beyond networks by deploying the fronthaul transport links between baseband units (BBUs) and remote radio units (RRHS). High-capacity optical links could be conventional mainstream technology for deploying the fronthaul in CRANs. But densification of optical links significantly increases the cost and imposes several design challenges on fronthaul architecture which makes them impractical. Contrary, Ethernet-based fronthaul links can be lucrative solutions for connecting the BBUs and RRHs but are inadequate to meet the rigorous end-to-end delays, jitter, and bandwidth requirements of fronthaul networks. This is because of the inefficient resource allocation and congestion control schemes for the capacity constraint Ethernet-based fronthaul links. In this research, a novel reinforcement learning-based optimal resource allocation scheme has been proposed which eradicates the congestion and improves the latencies to make the capacity-constraints low-cost Ethernet a suitable solution for the fronthaul networks. The experiment results verified a notable 50% improvement in reducing delay and jitter as compared to the existing schemes. Furthermore, the proposed scheme demonstrated a significant enhancement of up to 70% in addressing conflicting time slots and minimizing packet loss ratios. Hence, the proposed scheme outperforms the existing state-of-the-art resource allocation techniques to satisfy the stringent performance demands of fronthaul networks.