Yes I would definitely walk the fence line in. Like the others have said, it will be very hard to continuously see deer if you stomp through the area they feel safe in. They may not move far, but they could become completely nocturnal quick.
Glad to see you are 2 new hunters trying your...