Deep learning applied to image processing is revolutionizing the field of computer vision. Many successful applications have emerged, such as automatic online image and video indexing, scene interpretation in autonomous driving and classification of hard to describe objects. This last point is of particular interest in visual inspection: classical machine vision algorithms with heuristic approaches work well when the object appearance can be described through simple, quantifiable rules, however they become intractable with objects of very complex appearances. This is especially true for natural materials such as wood, because it can grow and get damaged in unpredictable ways.Luxscan has an experience of 20 years in manufacturing scanners for inspecting wood boards, and has accumulated a large library of image processing algorithms based on heuristics for wood defect detection and classification. A major challenge is that these algorithms need to be readjusted by a skilled technician whenever the wood characteristics change, which is a costly process and still remains prone to incorrect predictions.Deep learning approaches have proven themselves to be capable of learning robust image features for distinguishing very complex objects. We therefore want to apply such techniques for wood defect detection, classification and pixelwise labeling (semantic segmentation). Luxscan already has a first experience in dataset collection, defect classification and semantic segmentations through past internships, with promising results. However, we have seen that this is a complex problem, including challenges such as additional data collection, finding convolutional or recurrent neural network architectures, transfer learning, multimodality, distribution shift and imbalanced labels, as well as constraints on computational resources and real-time processing: all this justifies a long-term research investment through a PhD thesis.