If you put the towel in the pouch you won't be able to close it!

Maybe rip it up and put some of the towel in there? I think it's likely to be the smell they like, if that's the first thing you wrapped them in then it's going to have a familiar scent which the pouches won't have. Maybe you should wear a strip of the towel next to your skin for a day and put that in too so that way they will become familiar with your scent too.