I'm wondering if it is a plaster wall at all, from the description it may be an early answer to today's backer board.
Backer board or cement board is specifically designed to take tile. Made of cement it can withstand moisture better than drywall or plaster which falls apart when wet.
Plaster is white, chalky and crumbles easily. Cement is grey, harder and does not crumble easily. You will need to know which you are dealing with to know with what to repair it. If in doubt take a sample of the chipped off wall to any hardware store/ home store so they can tell you what you are dealing with.
Can you chip away the glue without removing too much of the wall? If yes, then proceed, then after you remove the glue you can texture coat the surface of the walls with plaster (AKA Skim Coat) - possibly even skim coat it a bit to get a smoother texture to it. There are adhesive removers on the market as well.
Smoothing and texturing plaster is not really a huge deal; it does take a little skill but is quickly learned. I would suggest using pre-mixed plaster coating and ask the guys and gals at your hardware/home store what tools you need and a quick lesson on how to use those tools effectively. I would also take a sample of the original wall so they know what type of plaster you need to finish coat your walls.
http://www.doityourself.com/wall/repairdamagedplasterwalls.htm Has a lot of useful information along those lines.
I would suggest texturing the wall with the finish coat, there are many ways to do that, from bouncing a sponge to leave behind peaks which you knock down with a trowel, to purposefully leaving behind trowel lines.
A good decorative and more popular wall treatment is Venetian Plaster:
http://spatespainting.com/VenetianPlaster/instructional.htmlhttp://www.texston.com/Venetian Plaster may very well cover up and hide any and all flaws that your wall will have from being repaired. The last link has a decent gallery of types of plaster jobs. The technique specifically aims for that old world (Mediterranean) aged/semi-deteriorated look in several of the photos.