This is probably your biggest problem.. i know it was mine!
In the end i ditched all that old stuff that i have carried around and never used such as a million different sized hooks that came with tackle boxes etc etc.
Now i have a much more streamlined system where i only take the stuff i need for the fish i target 99% of the time. (snapper/gummy, flathead/whiting & squid). And even then i only take what i will need for the day eg: 5 or 10 of each type of hook that i want rather than a packet of 50 of each. Everything else gets left at home in a larger tackle box. This way i am only carrying a couple of smaller tackle trays and a small bag, which all slots neatly into the side pockets or besides the drivers seat.
As you say, you can always count on your mates the bring a bunch of crap that only gets in the way so it really does help if you can start with an uncluttered boat.