While a crack or hole in the flue is possible it is not the likely reason for smoke entering your property. Most modern houses and modernised older houses are fairly airtight, This in conjunction with mechanical kitchen and bathroom extractor vents results in negative air pressure with some properties.
This results in outside air trying to enter properties to replace the air being extracted by kitchen and bathroom vents. In poorly insulated houses the air that is being extracted through the kitchen or bathroom vent is usually replaced by air entering though poorly sealed older windows, gaps around doors or wall vents. However in newer or modernised houses there are less areas or gaps for air to enter the property, so air is taken in through the chimney flue. This normally causes no problems as fresh air enters into the house through the open chimney.
However, when your neighbour decides to light a fire in his house, the carbon monoxide fumes escaping out his chimney can get sucked down your adjoining flue due to negative air pressure in your property.
There are a number of simple and various solutions to this problem, these solutions will depend on a number of factors but mostly revolve abut removing the negative air pressure in your property.