Perhaps the charger has seen better days or maybe the cable is not all that great. Start with a different charger and see if that helps. Process of elimination is what you are going to have to go through to solve your problem.
The output wattage of your charger is too low (especially if you use a computer/laptop usb port). Find one that does 2 Amps or more. My Samsung Charger that came with my Galaxy charges at 5.4 V at 2 Amps (10.8 W). It took 2 Hrs & 30 minutes to go from 10% -> 100%. The majority of the charge is done under 2 hours, as the last few minutes the current drops significantly to top off the batteries. That is the battery internal circuits doing this.
I also have a USB PD charger (my laptop power), and it charges a little over 1 Hr and 30 minutes.
The only way to charge faster is to bypass the usb and charge directly to the batteries main port. You can charge in 60-70 minutes this way with products from Amazon.