You should definitely have gotten a permit. It's worth it. Though, it can take a few days with the whole "lottery" system. I lucked-out, and happened to arrive just minutes before the lottery started. There were about 45 people in the ranger station, but luckily enough, I was one of the names they called. On any given day, there will be 40-80 people trying for permits; this sounds discouraging, I know, but if you aren't selected, the rangers will add an additional number in the lottery tumbler for your party. Thus, the odds of being selected become pretty good after a few days.
Conversely, you can sign up for this online, apparently, but I think it's booked ~ a year in advance.