Skip to main content

elastic-package-replaceips-sh

· 3 min read

When you're building packages for Elastic Fleet/Agent, if you're providing sample logs to test pipeline and system processes with it's not uncommon to have elastic-package whinge about non-approved IP's being there similar to the below.

Elastic's rationale is, presumably, that sensitive IP information may be leaked if it's not replaced with IP's from an approved set. To a lesser extent I believe they're trying to force the use of real IP's that will have actual GeoIP information associated with them so that if a geoip processor is used in an ingest pipeline it will actually return geoip data that will be inserted into documents.

Frustratingly this also means that elastic-package will not accept RFC defined documentation network IP's, e.g. 192.0.2.0/24, 203.0.113.0/24, 2001:db8::/something etc.

user@box beelzebub % elastic-package test pipeline --generate
Run pipeline tests for the package
--- Test results for package: beelzebub - START ---
FAILURE DETAILS:
beelzebub/logs test-beelzebub-logs-ndjson.log:
[0] parsing field value failed: the IP "103.100.225.133" is not one of the allowed test IPs (see: https://github.com/elastic/elastic-package/blob/main/internal/fields/_static/allowed_geo_ips.txt)
%{BREVITY}%
[44] parsing field value failed: the IP "92.255.57.58" is not one of the allowed test IPs (see: https://github.com/elastic/elastic-package/blob/main/internal/fields/_static/allowed_geo_ips.txt)


╭───────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                 │ RESULT                                                                      │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────┼──────────────┤
│ beelzebub │ logs        │ pipeline  │ (ingest pipeline warnings test-beelzebub-logs-ndjson.log) │ PASS                                                                        │ 321.152166ms │
│ beelzebub │ logs        │ pipeline  │ test-beelzebub-logs-ndjson.log                            │ FAIL: test case failed: one or more problems with fields found in documents │ 1.868443625s │
╰───────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────┴──────────────╯
--- Test results for package: beelzebub - END   ---
Done
Error: one or more test cases failed
user@box beelzebub % cd ../..

I wound up creating the following script, which I keep in the top of my integrations fork, which makes quick work of replacing IP's with those that will be accepted. I've now published this on GitHub as a gist for future modification tracking.

#!/bin/bash

function usage() {
  echo "Usage: ${0} package_name"
  exit 1
}

PACKAGE=${1}

IPv4_LEAD="1.128.0."
IPv6_LEAD="2a02:cf40:"
#IPv4_LEAD="203.0.113." # RFC 5737 - TEST-NET-3
#IPv6_LEAD="2001:db8:" # RFC 3849

test -z "${1}" && echo "ERROR: package name not provided" && usage
test ! -d "./packages/${PACKAGE}" && echo "ERROR: folder does not exist at ./packages/${PACKAGE}" && usage

for FILE in ./packages/${PACKAGE}/data_stream/*/_dev/test/pipeline/test-*.log ./packages/${PACKAGE}/data_stream/*/_dev/test/pipeline/test-*.json ; do
  echo "### Fixing IP's in ${FILE}"
  sed -r -i.backup "s/\"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.([0-9]{1,3})/\"${IPv4_LEAD}\1/g" "${FILE}" && rm -f "${FILE}.backup"
  sed -r -i.backup "s/\"(([A-F0-9]{1,4}:){2,2})((:|:[A-F0-9]{1,4}){1,5}|([A-F0-9]{1,4}:){1,5}:|([A-F0-9]{1,4}:){1,4}:([A-F0-9]{1,4})|([A-F0-9]{1,4}:){5,6}([A-F0-9]{1,4}))/\"${IPv6_LEAD}\3/gi" "${FILE}" && rm -f "${FILE}.backup"
  sleep 1
done 

# EOF

Example,

user@box integrations % ./replaceips.sh beelzebub
### Fixing IP's in ./packages/beelzebub/data_stream/logs/_dev/test/pipeline/test-beelzebub-logs-ndjson.log
### Fixing IP's in ./packages/beelzebub/data_stream/logs/_dev/test/pipeline/test-beelzebub-logs-ndjson.log-expected.json
user@box integrations % cd packages/beelzebub 
user@box beelzebub % ls
_dev            changelog.yml   data_stream     docs            img             manifest.yml
user@box beelzebub % elastic-package test pipeline --generate
Run pipeline tests for the package
--- Test results for package: beelzebub - START ---
╭───────────┬─────────────┬───────────┬───────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE   │ DATA STREAM │ TEST TYPE │ TEST NAME                                                 │ RESULT │ TIME ELAPSED │
├───────────┼─────────────┼───────────┼───────────────────────────────────────────────────────────┼────────┼──────────────┤
│ beelzebub │ logs        │ pipeline  │ (ingest pipeline warnings test-beelzebub-logs-ndjson.log) │ PASS   │ 341.923375ms │
│ beelzebub │ logs        │ pipeline  │ test-beelzebub-logs-ndjson.log                            │ PASS   │ 2.992818916s │
╰───────────┴─────────────┴───────────┴───────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: beelzebub - END   ---
Done
user@box beelzebub %