Building on my father's legacy & family/neighbors are upset.

From 1997 to 2015, my father flew and instructed others in flying Experimental ultralight aircraft. This was done on our family farm. His father was a WW2 pilot that also flew aircraft on the farm. I am a 3rd generation pilot. I am starting an light craft instruction school on the Farm for ultralight and Paramotors. Using the exisiting airplane hanger and grass strip, just like my father did. Since 2015 we have some new neighbors, they are threatening to band together and cause me legal trouble, on top of complaints every day for less than 30 mins of engine noise. (Keep in mind, my airstrip is in the MIDDLE of the 120 acre farm), I guess they are complaining about with I have students in the air.

Should I be worried?