My wife and I just came back from 3 months backpacking in South America. My wife and I are both fairly into photography (not crazy talented, but like taking pictures).
We brought an SLR (old Nikon D60), a more compact Point and Shoot (high end Pentax), and both our phones.
We found that we only took the SLR out on hikes or when we were specifically going somewhere cool where we wanted pictures (Machu Picchu, Torres del Paine, etc) and that when you're wandering around cities, its too obtrusive, so we made do with the P&S and the phones. Even though the D60 was old, the colours and white balance were much better than the P&S, so we tried taking it when we could, but that wasn't often. Most often, it sat inside our room.
If you're looking for a smaller camera, I'd highly recommend the Sony A6000. Its much smaller than a full SLR and takes really great pictures, great autofocus, and low-light performance as well.
+1 for not carrying around an extra bag for your camera. If nothing else, grab one of those neoprene inserts, if you're worried about a camera getting damaged, and store it in your day-pack.
I found that even with the camera on our phones being pretty good (Galaxy S4), the difference in quality and adjustment settings, not to mention ability to zoom (optically), meant that it was worth it for me to have the other cameras.
We came back with more than 5000 pictures from our trip... still sorting through them.
The Ex: 2002 Subaru 2.5RSTi w/ JDM V7 STI Swap, JDM 6-Speed Trans, Dom 1.5XTR, ST Coilovers
The Ride: 2014 Ducati Multistrada S, Tuneboy w/ CC
Daily: Boring Volvo.