The PhD project aims to investigate self-learning algorithms design and implementation for real-time bidding (RTB) system. The PhD candidate is supervised by Dr. Radu State and Prof. Björn Ottersten from the University of Luxembourg and Prof. Mats Brorsson from the Royal Institute of Technology (KTH). The project cooperates with a partnering company OLAmobile, which provides tailor solutions for ad campaigns in order to maximize campaign’s targeted performance. There is a rich literature ranging from various topics regarding RTB system, such as prediction algorithms design and real-time bidding strategies design. However, less is known in the mobile advertising market. Besides, the research carried out on RTB systems have been limited in industry since it is sensitive to release their data to the public. Working with OLAmobile, the anonymized user data can be used for offline analysis while their RTB platform is available for online evaluation and testing. The goal of the PhD project is to automatically match right customers to advertisers, especially in the mobile ad market. In this way, advertisers achieve a more efficient spending and users receive customized ad display which leads to higher potential of click or conversion. Consequently, publishers obtain benefits in terms of total revenue.The major problems are automatic feature selection, cold-start problem, and algorithm integration with real-time streaming architecture. The project proposes to apply deep learning and reinforcement learning to facilitate feature engineering and to adaptively learn the best bidding strategies for each ad campaign. RTB system requires bid responses generated within rather limited time on the scale of 100ms. Thus, integrating the developed algorithms into real-time streaming architecture is also a major challenge. Further improvements and optimization for mobile ad network will also be provided. The developed algorithms for real-time prediction can be further applied to other large-scale problem areas.