Tell your doctor exactly what you are saying here. There are two general strains of ADHD meds, and even amongst those two the variants tend to have certain side effects.
For me, I had very similar side effects from vyvanse. It did make me very conscious and able to live totally in the moment, which was wonderful. However, it also gave me bad anxiety. For some time my Psych tried to counter that with anti-anxiety meds, but it did not work, so I was switched to Concerta. Concerta and Ritalin are from the same strain, and if Vyvanse is not agreeing with you, it may be a great reason to switch.
This is what my prescriber recommends with stimulants. They are very helpful for adhd, but hopefully not required for all functionality. From what I understand, if that seems to be the case, a stimulant will be paired with and anti-anxiety or anti-depressant for daily use to, hopefully, decrease the need for stimulants.
For myself, taking a day or two off seems to help the efficacy of the stimulants. I’m not able to take vyvanse, as it ended up giving me terrible anxiety, but that will happen for some. I’m doing well on concerta instead, but the same principle applies.