creating-custom-mmdb-files
Prompt: How can I create dummy MaxMind GeoLite2 .mmdb files that will respond to ALL IP lookups with the same information? For example these these would be .mmdb files that will have entries for 0.0.0.0/0 and ::/0 that will cover all possible IP lookups?
Gemini 2.0 Flash will actually provide a moderately useful response, albeit one based on the legacy MaxMind perl based code available from MaxMind-DB-Writer-perl. It also correctly recommends using the mmdbinspect tool to test the .mmdb files that have been produced.
ChatGPT (free not Plus) also provides a fairly useful response, albeit one based on python code. It doesn't seem to know about mmdbinspect and does produced .mmdb file testing using python code instead.
Claude.ai provides something similar to ChatGPT's solution.
The Better Solution
However, unsurprisingly, while all of these solutions appear valid, they're based on outdated information and are incomplete if you want detailed GeoIP lookup data to be returned. To really get the solution you want, even if you were to stick with their perl/python code (assuming it actually works out of the paste...), will require a deep dive on what fields can/should exist.
MaxMind also replaced their Perl code with Go quite a few years ago, completely deprecating the Perl code at least 2 years ago. The replacement is mmdbwriter.
However this new Go package is simply something that can work with .mmdb files, and while there are some example commands included they're not immediately usable.
A more useful repo exists in the form of MaxMind-DB. This repo includes a heap of raw test data and test cases so it has everything we need to work out how to create some raw GeoLite2 source data as .json files, and a tool to convert those .json files to .mmdb files.
The included write-test-data command can, with some minor modifications, be used to accept 0.0.0.0/0 and ::/0 to cover all possible IPv4 and IPv6 lookups.
Modifications
Having cloned MaxMind-DB I've only had to make one modification to the current content.
I've modified pkg/writer/geoip2.go and inserted IncludeReservedNetworks: true as an argument within mmdbwriter.Options as that's provided to the mmdbwriter constructor mmdbwriter.New().
I found this was necessary to allow acceptance of 0.0.0.0/0 and ::/0.
Without this additional option the produced files would contain no entries and the fact that these entries were dropped will not be reported at all. You'll just have .mmdb files that won't provide anything for any IP.
This appears to be because the code in mmdbwriter will filter any IP or IP network that is in or overlaps within various reserved blocks, e.g. refer to mmdbwriter/blob/main/reserved.go for what this includes.
user@box MaxMind-DB % git diff pkg/writer/geoip2.go
diff --git a/pkg/writer/geoip2.go b/pkg/writer/geoip2.go
index a95c0af..e8e1b8d 100644
--- a/pkg/writer/geoip2.go
+++ b/pkg/writer/geoip2.go
@@ -50,6 +50,7 @@ func (w *Writer) WriteGeoIP2TestDB() error {
DatabaseType: dbType,
Description: description,
DisableIPv4Aliasing: false,
+ IncludeReservedNetworks: true,
IPVersion: 6,
Languages: languages,
RecordSize: 28,
user@box %
You'll need to build the write-test-data command similar to the below, and understand how to use the arguments it accepts.
user@box MaxMind-DB % cd cmd/write-test-data
user@box write-test-data % ls -alF
total 8
drwxr-xr-x 3 user staff 96 15 Feb 11:48 ./
drwxr-xr-x 3 user staff 96 15 Feb 09:17 ../
-rw-r--r-- 1 user staff 1569 15 Feb 11:46 main.go
user@box write-test-data % go build
user@box write-test-data % ls -alF
total 7568
drwxr-xr-x 4 user staff 128 15 Feb 11:48 ./
drwxr-xr-x 3 user staff 96 15 Feb 09:17 ../
-rw-r--r-- 1 user staff 1569 15 Feb 11:46 main.go
-rwxr-xr-x 1 user staff 3870210 15 Feb 11:48 write-test-data*
user@box write-test-data % ./write-test-data -h
Usage of ./write-test-data:
-source string
Source data directory
-target string
Destination directory for the generated mmdb files
user@box write-test-data %
You can now clone the default source-data files in order to modify them. You could of course also modify these in place if you'd prefer.
user@box MaxMind-DB % cp -Rpv source-data source-data-test
source-data -> source-data-test
source-data/GeoIP2-Precision-Enterprise-Test.json -> source-data-test/GeoIP2-Precision-Enterprise-Test.json
source-data/GeoIP2-Enterprise-Test.json -> source-data-test/GeoIP2-Enterprise-Test.json
source-data/GeoIP2-Precision-Enterprise-Sandbox-Test.json -> source-data-test/GeoIP2-Precision-Enterprise-Sandbox-Test.json
source-data/GeoIP2-Country-Test.json -> source-data-test/GeoIP2-Country-Test.json
source-data/GeoIP2-Domain-Test.json -> source-data-test/GeoIP2-Domain-Test.json
source-data/GeoLite2-City-Test.json -> source-data-test/GeoLite2-City-Test.json
source-data/GeoIP2-Anonymous-IP-Test.json -> source-data-test/GeoIP2-Anonymous-IP-Test.json
source-data/GeoIP2-Static-IP-Score-Test.json -> source-data-test/GeoIP2-Static-IP-Score-Test.json
source-data/GeoIP2-Connection-Type-Test.json -> source-data-test/GeoIP2-Connection-Type-Test.json
source-data/GeoIP2-IP-Risk-Test.json -> source-data-test/GeoIP2-IP-Risk-Test.json
source-data/GeoLite2-Country-Test.json -> source-data-test/GeoLite2-Country-Test.json
source-data/GeoIP2-City-Test.json -> source-data-test/GeoIP2-City-Test.json
source-data/GeoIP2-ISP-Test.json -> source-data-test/GeoIP2-ISP-Test.json
source-data/GeoIP2-DensityIncome-Test.json -> source-data-test/GeoIP2-DensityIncome-Test.json
source-data/GeoLite2-ASN-Test.json -> source-data-test/GeoLite2-ASN-Test.json
source-data/GeoIP2-User-Count-Test.json -> source-data-test/GeoIP2-User-Count-Test.json
user@box MaxMind-DB %
If you don't also copy the GeoIP2-*-Test.json files you'll get errors when running the command, you can make further code modifications to pkg/writer/geoip2.go if you don't want to have these generated at all.
Now modify the files as needed.
In my case I just want to modify the GeoLite2 .json files, and I'm going to ensure all ASN's returned are 64496 which is a documentation specific ASN defined by RFC-5398 and that all location data is for Greenwich UK and more specifically the location coordinates are for the Royal Greenwich Observatory. I've picked Greenwich as I couldn't think of a more appropriate real world location. You can, almost certainly, inject your own descriptions and even using 0/0 for lat/long though that would place the IP as in the ocean off the coast of Ghana.
GeoLite2-ASN-Test.json
[
{
"0.0.0.0/0" : {
"autonomous_system_number" : 64496,
"autonomous_system_organization" : "Documentation ASN"
}
},
{
"0000:0000:0000:0000:0000:0000:0000:0000/0" : {
"autonomous_system_number" : 64496,
"autonomous_system_organization" : "Documentation ASN"
}
}
]
GeoLite2-City-Test.json
[
{
"0.0.0.0/0" : {
"city" : {
"geoname_id" : 2647937,
"names" : {
"en" : "Greenwich"
}
},
"continent" : {
"code" : "EU",
"geoname_id" : 6255148,
"names" : {
"en" : "Europe"
}
},
"country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"location" : {
"accuracy_radius" : 0,
"latitude" : 51.47687,
"longitude" : -0.00041,
"time_zone" : "Europe/London"
},
"registered_country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"subdivisions" : [
{
"geoname_id" : 6269131,
"iso_code" : "ENG",
"names" : {
"en" : "England"
}
}
]
}
},
{
"0000:0000:0000:0000:0000:0000:0000:0000/0" : {
"city" : {
"geoname_id" : 2647937,
"names" : {
"en" : "Greenwich"
}
},
"continent" : {
"code" : "EU",
"geoname_id" : 6255148,
"names" : {
"en" : "Europe"
}
},
"country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"location" : {
"accuracy_radius" : 0,
"latitude" : 51.47687,
"longitude" : -0.00041,
"time_zone" : "Europe/London"
},
"registered_country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"subdivisions" : [
{
"geoname_id" : 6269131,
"iso_code" : "ENG",
"names" : {
"en" : "England"
}
}
]
}
}
]
GeoLite2-Country-Test.json
[
{
"0.0.0.0/0" : {
"continent" : {
"code" : "EU",
"geoname_id" : 6255148,
"names" : {
"en" : "Europe"
}
},
"country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"registered_country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
}
}
},
{
"0000:0000:0000:0000:0000:0000:0000:0000/0" : {
"continent" : {
"code" : "EU",
"geoname_id" : 6255148,
"names" : {
"en" : "Europe"
}
},
"country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
},
"registered_country" : {
"geoname_id" : 2635167,
"iso_code" : "GB",
"names" : {
"en" : "United Kingdom"
}
}
}
}
]
Now run write-test-data to build the .mmdb files. Note that the target directory does not need to exist already, write-test-data will create it if it doesn't exist.
user@box MaxMind-DB % ./cmd/write-test-data/write-test-data -source source-data-test -target test-data-test
user@box MaxMind-DB % ls -ld test-data-test/*
-rw-r--r-- 1 user staff 3289 15 Feb 12:07 test-data-test/GeoIP2-Anonymous-IP-Test.mmdb
-rw-r--r-- 1 user staff 21096 15 Feb 12:07 test-data-test/GeoIP2-City-Test.mmdb
-rw-r--r-- 1 user staff 3228 15 Feb 12:07 test-data-test/GeoIP2-Connection-Type-Test.mmdb
-rw-r--r-- 1 user staff 18176 15 Feb 12:07 test-data-test/GeoIP2-Country-Test.mmdb
-rw-r--r-- 1 user staff 1649 15 Feb 12:07 test-data-test/GeoIP2-DensityIncome-Test.mmdb
-rw-r--r-- 1 user staff 5077 15 Feb 12:07 test-data-test/GeoIP2-Domain-Test.mmdb
-rw-r--r-- 1 user staff 8571 15 Feb 12:07 test-data-test/GeoIP2-Enterprise-Test.mmdb
-rw-r--r-- 1 user staff 1651 15 Feb 12:07 test-data-test/GeoIP2-IP-Risk-Test.mmdb
-rw-r--r-- 1 user staff 74019 15 Feb 12:07 test-data-test/GeoIP2-ISP-Test.mmdb
-rw-r--r-- 1 user staff 18675 15 Feb 12:07 test-data-test/GeoIP2-Precision-Enterprise-Test.mmdb
-rw-r--r-- 1 user staff 9938 15 Feb 12:07 test-data-test/GeoIP2-Static-IP-Score-Test.mmdb
-rw-r--r-- 1 user staff 5006 15 Feb 12:07 test-data-test/GeoIP2-User-Count-Test.mmdb
-rw-r--r-- 1 user staff 1352 15 Feb 12:07 test-data-test/GeoLite2-ASN-Test.mmdb
-rw-r--r-- 1 user staff 1551 15 Feb 12:07 test-data-test/GeoLite2-City-Test.mmdb
-rw-r--r-- 1 user staff 1401 15 Feb 12:07 test-data-test/GeoLite2-Country-Test.mmdb
-rw-r--r-- 1 user staff 616 15 Feb 12:07 test-data-test/MaxMind-DB-no-ipv4-search-tree.mmdb
-rw-r--r-- 1 user staff 1334 15 Feb 12:07 test-data-test/MaxMind-DB-string-value-entries.mmdb
-rw-r--r-- 1 user staff 3188 15 Feb 12:07 test-data-test/MaxMind-DB-test-decoder.mmdb
-rw-r--r-- 1 user staff 1285 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv4-24.mmdb
-rw-r--r-- 1 user staff 1448 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv4-28.mmdb
-rw-r--r-- 1 user staff 1611 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv4-32.mmdb
-rw-r--r-- 1 user staff 2794 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv6-24.mmdb
-rw-r--r-- 1 user staff 3209 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv6-28.mmdb
-rw-r--r-- 1 user staff 3624 15 Feb 12:07 test-data-test/MaxMind-DB-test-ipv6-32.mmdb
-rw-r--r-- 1 user staff 2249 15 Feb 12:07 test-data-test/MaxMind-DB-test-metadata-pointers.mmdb
-rw-r--r-- 1 user staff 3048 15 Feb 12:07 test-data-test/MaxMind-DB-test-mixed-24.mmdb
-rw-r--r-- 1 user staff 3492 15 Feb 12:07 test-data-test/MaxMind-DB-test-mixed-28.mmdb
-rw-r--r-- 1 user staff 3936 15 Feb 12:07 test-data-test/MaxMind-DB-test-mixed-32.mmdb
-rw-r--r-- 1 user staff 2589 15 Feb 12:07 test-data-test/MaxMind-DB-test-nested.mmdb
user@box MaxMind-DB %
Test the produced .mmdb files,
GeoLite2-ASN.mmdb
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-ASN-Test.mmdb 203.0.113.0
[
{
"Database": "./test-data-test/GeoLite2-ASN-Test.mmdb",
"Records": [
{
"Network": "203.0.113.0/1",
"Record": {
"autonomous_system_number": 64496,
"autonomous_system_organization": "Documentation ASN"
}
}
],
"Lookup": "203.0.113.0"
}
]
user@box MaxMind-DB %
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-ASN-Test.mmdb 8.8.8.8
[
{
"Database": "./test-data-test/GeoLite2-ASN-Test.mmdb",
"Records": [
{
"Network": "8.8.8.8/1",
"Record": {
"autonomous_system_number": 64496,
"autonomous_system_organization": "Documentation ASN"
}
}
],
"Lookup": "8.8.8.8"
}
]
user@box MaxMind-DB %
GeoLite2-City.mmdb
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-City-Test.mmdb 203.0.113.0
[
{
"Database": "./test-data-test/GeoLite2-City-Test.mmdb",
"Records": [
{
"Network": "203.0.113.0/1",
"Record": {
"city": {
"geoname_id": 2647937,
"names": {
"en": "Greenwich"
}
},
"continent": {
"code": "EU",
"geoname_id": 6255148,
"names": {
"en": "Europe"
}
},
"country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"location": {
"accuracy_radius": 0,
"latitude": 51.47687,
"longitude": -0.00041,
"time_zone": "Europe/London"
},
"registered_country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"subdivisions": [
{
"geoname_id": 6269131,
"iso_code": "ENG",
"names": {
"en": "England"
}
}
]
}
}
],
"Lookup": "203.0.113.0"
}
]
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-City-Test.mmdb 8.8.8.8
[
{
"Database": "./test-data-test/GeoLite2-City-Test.mmdb",
"Records": [
{
"Network": "8.8.8.8/1",
"Record": {
"city": {
"geoname_id": 2647937,
"names": {
"en": "Greenwich"
}
},
"continent": {
"code": "EU",
"geoname_id": 6255148,
"names": {
"en": "Europe"
}
},
"country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"location": {
"accuracy_radius": 0,
"latitude": 51.47687,
"longitude": -0.00041,
"time_zone": "Europe/London"
},
"registered_country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"subdivisions": [
{
"geoname_id": 6269131,
"iso_code": "ENG",
"names": {
"en": "England"
}
}
]
}
}
],
"Lookup": "8.8.8.8"
}
]
user@box MaxMind-DB %
GeoLite2-Country.mmdb
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-Country-Test.mmdb 203.0.113.0
[
{
"Database": "./test-data-test/GeoLite2-Country-Test.mmdb",
"Records": [
{
"Network": "203.0.113.0/1",
"Record": {
"continent": {
"code": "EU",
"geoname_id": 6255148,
"names": {
"en": "Europe"
}
},
"country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"registered_country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
}
}
}
],
"Lookup": "203.0.113.0"
}
]
user@box MaxMind-DB % mmdbinspect -db ./test-data-test/GeoLite2-Country-Test.mmdb 8.8.8.8
[
{
"Database": "./test-data-test/GeoLite2-Country-Test.mmdb",
"Records": [
{
"Network": "8.8.8.8/1",
"Record": {
"continent": {
"code": "EU",
"geoname_id": 6255148,
"names": {
"en": "Europe"
}
},
"country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
},
"registered_country": {
"geoname_id": 2635167,
"iso_code": "GB",
"names": {
"en": "United Kingdom"
}
}
}
}
],
"Lookup": "8.8.8.8"
}
]
user@box MaxMind-DB %
Congratulations you now have GeoLite2 database files that will respond to any query against them.
To use these within your elastic-package managed Elastic stack, simply overwrite the files in the appropriate profile under ~/.elastic-package.
user@box MaxMind-DB % find ~/.elastic-package -name \*.mmdb
/Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-City.mmdb
/Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-Country.mmdb
/Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-ASN.mmdb
user@box MaxMind-DB %
user@box MaxMind-DB % for i in ASN City Country ; do cp -v test-data-test/GeoLite2-${i}-Test.mmdb ~/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-${i}.mmdb ; done
test-data-test/GeoLite2-ASN-Test.mmdb -> /Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-ASN.mmdb
test-data-test/GeoLite2-City-Test.mmdb -> /Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-City.mmdb
test-data-test/GeoLite2-Country-Test.mmdb -> /Users/user/.elastic-package/profiles/default/stack/ingest-geoip/GeoLite2-Country.mmdb
user@box MaxMind-DB %
After doing this, and a restart of the elastic-package managed stack isn't really necessary, you should see your dummy GeoIP data appearing on test events, e.g.
user@box beelzebub % elastic-package test pipeline
2025/02/15 11:02:30 INFO New version is available - v0.109.1. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.109.1
Run pipeline tests for the package
--- Test results for package: beelzebub - START ---
FAILURE DETAILS:
beelzebub/logs test-beelzebub-logs-ndjson.log:
--- want
+++ got
@@ -206,6 +206,24 @@
]
},
"source": {
+ "as": {
+ "number": 64496,
+ "organization": {
+ "name": "Documentation ASN"
+ }
+ },
+ "geo": {
+ "city_name": "Greenwich",
+ "continent_name": "Europe",
+ "country_iso_code": "GB",
+ "country_name": "United Kingdom",
+ "location": {
+ "lat": 51.47687,
+ "lon": -4.1E-4
+ },
+ "region_iso_code": "GB-ENG",
+ "region_name": "England"
+ },
"ip": "203.0.113.133",
"port": 60748
},
@@ -257,6 +275,24 @@
%{BREVITY}%
@@ -54798,6 +66498,24 @@
]
},
"source": {
+ "as": {
+ "number": 64496,
+ "organization": {
+ "name": "Documentation ASN"
+ }
+ },
+ "geo": {
+ "city_name": "Greenwich",
+ "continent_name": "Europe",
+ "country_iso_code": "GB",
+ "country_name": "United Kingdom",
+ "location": {
+ "lat": 51.47687,
+ "lon": -4.1E-4
+ },
+ "region_iso_code": "GB-ENG",
+ "region_name": "England"
+ },
"ip": "203.0.113.53",
"port": 40742
},
╭───────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────┼──────────────┤
│ beelzebub │ logs │ pipeline │ (ingest pipeline warnings test-beelzebub-logs-ndjson.log) │ PASS │ 359.849459ms │
│ beelzebub │ logs │ pipeline │ test-beelzebub-logs-ndjson.log │ FAIL: test case failed: Expected results are different from actual ones │ 42.7460275s │
╰───────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────┴──────────────╯
--- Test results for package: beelzebub - END ---
Done
Error: one or more test cases failed
user@box beelzebub %