A study of college students and how much time they report studying and preparing for their classes shows that engineering and physics students need to spend the most time. This suggests that these two majors are among the most, if not the most, challenging. This ultimately is a subjective sort of topic which leads to good debates among students and graduates, but it is likely to be about right. Speaking from my own experience, engineering and physics students spent more time in their studies when I was in school, with often long problem sets, technically challenging topics that simply require a large amount of time to mentally digest, for engineers spending what may seem like endless hours trying to get projects working, and for physicists spending many hours in labs and research. This would also be true for just about any science major.
Most majors in college are going to be challenging and require hard work, but a student will know if it is the right major for them if the work is difficult and yet fun, because if it is something one truly enjoys and wants to do as a career, it will be quite rewarding as time goes on.