I feel the same as you with the flying lack of confidence saw a person die on my first and last time at skydiving. I have been stopping and starting to get my licence since 1999!!
And I am about to have another attempt at it this week at Adelaide biplanes. Some people instructors have said to me if you are not enjoying it you should just burn your money and not waste your time flying.
I will not go to my grave saying I wish I finished my license I don’t care how much it costs.
All the best
Shafs