I want to study physics and related maths, but I am confused….

here is what i wish to say.

basically i am a 23 year old guy who has wasted a lot of time not finishing any course i started and also occasionally trying to be a musician but also did not fully get into that.
There were times since completing high school that I found myself very intersted in reading elementary physics and maths like calclus,classical Newtonian physics and also I dreamt of someday understanding higher and higher levels of physics and maths.

I am now settling on doing a graduate level sort of physics maths chemistry course from a distant education sort of university.
and i am reading texts relevant to it currently.
it is not exactly a typical specific degree.it would contain a lot of core topics that one would see in a bachelors degree of maths,physics or chemistry.
it is basically a general science bachelors degree.after which you can pursue a master.if you succeed.really emphasise the if succeed part.

now the problems i am facing is…

I get scared by the length of higher level physics maths texts and also how easily what you study in them is forgotten.also had this scary thought that it would take me a lifetime just reading those big maths physics books.Maybe not that exactly but definitely a really long time.5 or more years really.

also a lot of times i would just have to finish the study material of the university first which since i studied it a bit of 1st semester
i think is taken from typical physics maths and chemistry texts only.
so i get bogged down by the thought that maybe it will be hard to read both that study material and typical books at same time.

and also very intimated by how hard physics and maths texts get at higher levels like relativity quantum mechanics or something like abstract algebra topology etc.
and at the same time i do have this desire to do all this since,i personally think it would become a regret in my life if I didn’t study physics maths and got into some other field.
Since i was and am always very curious and interested in what deep tuths lie in these two subjects.

and well not like some other field would be a piece of cake,and also there i might have this regret sort of thought going in my mind that i really wished i had not given up on the physics maths desire.

Can say I do wish to climb the mountain but get scared I might not be able to and might not be able to make a career and just be some really big failure.
Any advise for someone like me?
Open to all sorts of them currently.

Thank you for your time.