Kompozinaut
Sylph Sword
Heavenly Maiden said:I do not have the manga volume right here in front of me so I might have quoted the wrong pages but there is a scene where Casca turns Charlotte down for joining them where Guts seems to get offended by Casca's manner and snatches his arm away when she tries to touch him. Casca looks upset as well.
To me Guts's annoyance comes from him realizing that both he and Casca are still wrapped up in Griffith's story. (Paraphrased: "The one who is still bound to Griffith...is me") This is revealed to him by Casca empathizing with Charlotte in the previous pages.
Casca quickly realizes she might be acting insensitive towards Guts and apologizes. Guts then appears to silently accept her apology and comes to the revelation I quoted above. The whole situation is dropped from that moment on.