There are some things that you can do but it does get very tricky! You may have to create your own custom web part to achieve the display you are looking for. Some ideas though:
You can always right click on the web part in Sharepoint Designer, select convert to XSLT. Once you have done that, show the code in split view. You can then select and modify the way the list is displaying. You can set conditional formatting on a row or item based on the values in the row or you can remove the intial category label and just display "Contacts", "Miscellaneous", and "References" . Much more than general formatting is very hard to do without creating your own web part.
That is another discussion all together.
Can you describe in more detail what you are visually trying to achieve?