![Orbis Terrarum Nova et Accuratissima Tabula, by [Nicolaes Visscher](https://en.wikipedia.org/wiki/Nicolaes_Visscher_I), 1658. Source:[ Wikipedia](https://en.wikipedia.org/wiki/File:Orbis_Terrarum_Nova_et_Accuratissima_Tabula_by_Nicolaes_Visscher,_1658.jpg)](img/map2.jpg) Dmap is multi-application crawler that *reduces the complexity* in both executing Internet measurements and analyzing results. Dmap uses a list domain names (scalable to many millions), and: 1. *Crawls* domains for DNS, HTTP, TLS, SMTP and a creates a screenshot of a webpage (if it's available) 1. *Converts* the raw measurement data and stores this into a 166-features unified SQL data model, so users can easily analyze the results using SQL. We have been using Dmap since 2017 to analyze the .nl DNS zone. Just run it and query it. More details on our [research paper](paper.pdf). **IMPORTANT: data model changes (2018-08-20)** * We keep improving Dmap continuously. As such, there are some changes from the version described on our [research paper](paper.pdf). * More importantly, the [data model](datamodel) page in this website *refers to the model used in the [research paper](paper.pdf).* * The data model used in the the current version can always be found updated on our [Github repository][2] (don't have access? Fill in the [form][5]) **NEWS:** * **2018-08-20**: We presented during the [IETF 102](https://www.ietf.org/how/meetings/102/) in Montreal, Canada. * Full presentation at the [IEPG meeting](http://iepg.org/), on July 15,2018. Slides [available here](http://iepg.org/2018-07-15-ietf102/maura-iepg-102.pdf) * Short presetation at the [Measurement and Analysis Reserach Work group (IRTF MAPRG)](https://datatracker.ietf.org/rg/maprg/about/). Slides [available here.](https://datatracker.ietf.org/meeting/102/materials/slides-102-maprg-dmap-automating-domain-name-ecosystem-measurements-and-applications-giovane-c-m-moura-00.pdf) * **2018-06-06**: We will present our [Dmap paper](paper.pdf) at the [TMA 2018 conference](http://tma.ifip.org/2018/) on June 27th, 2018, in Vienna, Austria. * **2018-06-05**: We will start sharing Dmap code on 2018-06-25. * **2018-06-05**: Website online # How to get it We have developed [Dmap](http://dmap.sidnlabs.nl) as a research project at [SIDN Labs](https://sidnlabs.nl), the research arm of [SIDN](https://sidn.nl), the .nl top-level domain registry. We are more than happy to make Dmap available under an academic license, both code and binaries, for academic use. This license does not allow Dmap to be used for commercial puposes. Therefore, **if you are a researcher**, just: * Fill in the [form][5] and you will receive an e-mail from us with instructions. * *Notice*: we will start sharing the code on Github on June 25th, 2018. # [Demo dataset ](demo.html) Very often, researchers spend an awful lot of time planning and executing measurements. The complexity of different data formats and issues that may emerge drain energy that could be otherwise be spent on research questions. [Dmap](http://dmap.sidnlabs.nl) is intended to be an enabling platform and is designed to free researchers from the complexity of measurements and parsing complex data formats; rather, it automates the measurements and stores the data in an easy to use SQL database. We have a Dmap database dump for the Alexa 1M domains list, which we cover in our [research paper](paper.pdf). Want to give at try? Check the [demo dataset](demo.html) and the [data model](datamodel.html) for a description of the data model. # [Research paper](paper.pdf) For a full description of Dmap, please download our [research paper](paper.pdf) ```verbatim Wullink, M., Moura, G.C. M., Hesselman, C.: Dmap:Automating Domain Name Ecosystem Measurements and Applications. In: IFIP/IEEE Network Traffic Measurement and Analysis Conference (TMA 2018). Vienna, Austria, 26-29 June 2018. ``` Bibtex: ```verbatim @inproceedings{wullink2018-dmap, author = {Wullink, Maarten and Moura, Giovane C. M. and Hesselman, Cristian}, title = {{Dmap: Automating Domain Name Ecosystem Measurements and Applications}}, booktitle = {IFIP/IEEE Network Traffic Measurement and Analysis Conference (TMA 2018)}, year = {2018}, month=jun, address = {Vienna,Austria} } ``` # Contact * E-mail: `sidnlabs at sidn dot nl` * For code issues, please use [Github's][2] issue tracking * Last update: Tue Aug 21 09:05:44 UTC 2018 [5]: https://goo.gl/forms/RM09ZRPSyl2iblT22 [2]: https://github.com/SIDN/dmap Copyright (C) 2018 SIDN Labs