I figured it out after I called the ISP. They made a change to their network, and something was not right and it was dropping traffic. The external port checker really helped here, and they were very helpful (switched me back to the old network it seems).