Removing mysterious material holding bricks from drywall.

The previous owners adhered some bricks to a wall and I'm trying to remove them and restore the drywall. I'm not sure if they used plaster, concrete or mortar to adhere it to the drywall. Removing the brick and grout is easy, but removing the material used to adhere it to the wall has been quite a challenging process.

I'm not sure how to identify the material, how to easily remove it or what kind of service one usually hires for this.