I'm 18 and he's 24? what do you think?

I say go for it. If it's obvious he likes you and from the way you're talking about him you obviously like him. You don't have to ask him out, just give a bit more attention to let him know your interested. Once you get past the 18 year old mark, age doesn't matter as much as personality and maturity. It's not like high school when it's a big thing to be dating someone out of your year level. If he's showing interest in you he obviously isn't concerned about the age gap, why should you be?